Improved GAIN method and device for repairing random missing of multi-sensor data

Abstract

The present disclosure provides a method and device for improving GAIN to repair random missing of multi-sensor data. The improved GAIN includes a generator and a discriminator. The generator and the discriminator both include an encoder-decoder architecture composed of a convolutional neural network, and a self-attention mechanism is integrated in the generator to make the network focus more on important features and enhance the ability of the network to extract global features. Both the generator and the discriminator are provided with jump connections, which can alleviate the problem of gradient disappearance and improve the utilization rate of features. The method trains the improved GAIN by using an incomplete data matrix, and the data corresponding to each sensor in the incomplete data matrix are randomly partially missing. Thus, the sensor data actually collected is repaired according to the trained generator, which can ensure the repair effect of the missing data of all sensors.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of data processing, and in particular relates to a method and a device for improving GAIN to repair random missing of multi-sensor data. Background Art

[0002] The large amount of sensor data accumulated in the structural health monitoring (SHM) system provides a solid data support for in-depth exploration of basic scientific issues of engineering structures, such as structural condition assessment. Therefore, the accuracy and effectiveness of structural condition assessment depend largely on the quality of sensor data. However, due to the harsh environment in which sensors are applied and their improper maintenance during operation, the random lack of sensor data is a common problem in the SHM field.

[0003] In the current technical solutions, the missing data repair method based on the generative adversarial interpolation network is often used to repair the missing data. However, this method still requires some complete sensor data, while all sensor data may be missing in actual engineering, and it is difficult to obtain complete sensor data. In particular, if all sensor data is missing at the same time, it will undoubtedly further increase the difficulty of data repair. Therefore, how to ensure the training effect of the network when all sensor data is incomplete, so as to improve the repair effect of the trained network on the missing data of all sensors is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a method and device for improving GAIN to repair random missing of multi-sensor data, which can simultaneously repair the missing data of all sensors when all sensor data have a certain degree of random missing.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a method for improving GAIN to repair random missing of multi-sensor data is provided, which is used to repair the missing data of all sensors when there are random missing of multi-sensor data. The improved GAIN includes a generator and a discriminator. The generator and the discriminator are both constructed by a convolutional neural network with an encoder-decoder architecture to enhance the learning and expression capabilities of the network; a self-attention mechanism is embedded between the encoder and the decoder of the generator to extract important features, and a jump connection technology is introduced in both the generator and the discriminator to alleviate the problem of gradient disappearance and improve the utilization rate of features;

[0007] The method comprises:

[0008] An incomplete data matrix in which all sensor data are randomly missing is constructed, and a corresponding mask matrix and a random noise matrix are constructed based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix fills the missing positions of the original data with values ​​uniformly distributed from -0.01 to 0.01; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network;

[0009] The generator is constructed according to a convolutional neural network with an encoder-decoder architecture, and the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and the decoder of the generator to enhance the ability of the network to capture global features; a jump connection technology is introduced between the encoder and the decoder of the generator to pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance;

[0010] The discriminator is constructed according to a convolutional neural network with an encoder-decoder architecture, wherein the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and alleviate the problem of gradient vanishing;

[0011] The incomplete data matrix, the mask matrix and the random noise matrix are used as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain the data prediction matrix output by the generator;

[0012] The data prediction matrix and the prompt matrix are input into the discriminator to be trained, so that the discriminator predicts the probability of true or false data of each position according to the data prediction matrix; in the process of the discriminator judging the true or false data, the prompt matrix can provide the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution;

[0013] Through adversarial training, the deception of the data prediction matrix generated by the generator to the discriminator is improved, and the accuracy of the discriminator in distinguishing the authenticity of data in the data prediction matrix is ​​improved;

[0014] The generator that has completed training is used to repair the incomplete data that are randomly missing from all sensors actually collected.

[0015] According to one aspect of an embodiment of the present application, there is provided an improved GAIN device for repairing random missing of multi-sensor data, which is used to repair the missing data of all sensors when there are random missing of multi-sensor data. The improved GAIN includes a generator and a discriminator, and the generator and the discriminator are both constructed by a convolutional neural network with an encoder-decoder architecture to enhance the learning and expression capabilities of the network; a self-attention mechanism is embedded between the encoder and the decoder of the generator to extract important features, and a skip connection technology is introduced in both the generator and the discriminator to alleviate the problem of gradient disappearance and improve the utilization rate of features;

[0016] The device comprises:

[0017] An acquisition module is used to construct an incomplete data matrix in which all sensor data are randomly missing, and to construct a corresponding mask matrix and a random noise matrix based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix is ​​filled with values ​​uniformly distributed from -0.01 to 0.01 at the original data missing positions; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network;

[0018] The first building module is used to build the generator according to a convolutional neural network with an encoder-decoder architecture, wherein the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and the decoder of the generator to enhance the ability of the network to capture global features; a skip connection technology is introduced between the encoder and the decoder of the generator to pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance;

[0019] The second building module is used to build the discriminator according to a convolutional neural network with an encoder-decoder architecture, wherein the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and alleviate the problem of gradient disappearance;

[0020] A first input module, used to use the incomplete data matrix, the mask matrix and the random noise matrix as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain a data prediction matrix output by the generator;

[0021] A second input module is used to input the data prediction matrix and the prompt matrix into the discriminator to be trained, so that the discriminator predicts the probability of true or false data of each position according to the data prediction matrix; in the process of the discriminator judging the true or false data, the prompt matrix can provide the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution;

[0022] An optimization module, used to improve the deception of the data prediction matrix generated by the generator to the discriminator and improve the accuracy of the discriminator in distinguishing the authenticity of data in the data prediction matrix through adversarial training;

[0023] The processing module is used to perform data repair on the incomplete data with random missing data actually collected by all sensors according to the trained generator.

[0024] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, an improved GAIN method for repairing random missing of multi-sensor data as described in the above embodiment is implemented.

[0025] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement an improved GAIN method for repairing random missing of multi-sensor data as described in the above embodiment.

[0026] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes an improved GAIN method for repairing random missing of multi-sensor data provided in the above embodiment.

[0027] In the technical solutions provided in some embodiments of the present application, the corresponding mask matrix, random noise matrix and prompt matrix are constructed through an incomplete data matrix, the value of the position corresponding to the data missing position in the incomplete data matrix in the mask matrix is ​​0, and the other positions are 1; the position corresponding to the data missing position in the incomplete data matrix in the random noise matrix is ​​a value that obeys a uniform distribution of -0.01 to 0.01; in the prompt matrix, 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network. The incomplete data matrix, the mask matrix and the random noise matrix are used as the input of the generator to be trained, so that the generator numerically repairs the data missing position in the incomplete data matrix according to the input, and then the data of the non-missing part in the original data and the data predicted by the generator at the missing position of the original data are combined to obtain the data prediction matrix output by the generator. The data prediction matrix and the prompt matrix are input into the discriminator so that the discriminator predicts the mask matrix according to the input, and then the generator is optimized according to the prediction result of the discriminator, so as to improve the deception of the data prediction matrix generated by the generator to the discriminator, that is, to improve the authenticity of the data filled by the generator. After the training of the generator is completed, the data can be repaired according to the incomplete data with random missing of all sensors actually collected. In this way, even if all sensor data are missing, the training effect of the network can be guaranteed, thereby improving the repair effect of the trained network on the missing data of all sensors.

[0028] Furthermore, the improved generative adversarial interpolation network includes a generator and a discriminator, both of which include a convolutional neural network with an encoder-decoder architecture, and a self-attention mechanism is embedded between the encoder and the decoder of the generator, thereby enhancing the network's ability to capture global features and improving the accuracy of its output results. In addition, a skip connection technology is introduced between the generator and the discriminator to alleviate the problem of gradient vanishing and improve feature utilization.

[0029] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0031] Figure 1 A schematic flow chart of a method for improving GAIN to repair random missing of multi-sensor data according to an embodiment of the present application is shown;

[0032] Figure 2 A network architecture diagram of a generator according to an embodiment of the present application is shown;

[0033] Figure 3 A network architecture diagram of a discriminator according to an embodiment of the present application is shown;

[0034] Figure 4 A schematic diagram showing a time domain repair result of the data of the sensor 1 when the missing rate is 30% according to an embodiment of the present application is shown;

[0035] Figure 5 Shows a schematic diagram of the time domain repair result when the missing rate of sensor 2 data is 30% according to one embodiment of the present application;

[0036] Figure 6 A schematic diagram showing a frequency domain repair result of the sensor 1 data when the missing rate is 30% according to an embodiment of the present application;

[0037] Figure 7 A schematic diagram showing a frequency domain repair result of the data of the sensor 2 when the missing rate is 30% according to an embodiment of the present application is shown;

[0038] Figure 8 A block diagram of an improved GAIN repair device for random missing of multi-sensor data according to an embodiment of the present application is shown;

[0039] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0041] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0044] Figure 1 A flow chart of a method for improving GAIN to repair random missing of multi-sensor data according to an embodiment of the present application is shown.

[0045] It should be noted that the improved GAIN method for repairing random missing of multi-sensor data can be applied to terminal devices or servers. The terminal device may include but is not limited to one or more of a smart phone, a tablet computer, a portable computer, and a desktop computer; the server may be a physical server or a cloud server.

[0046] like Figure 2 and Figure 3As shown, the improved GAIN (Generative Adversarial Imputation Network) used in the embodiment of the present application includes a generator and a discriminator, both of which include a convolutional neural network (CNN) with an encoder-decoder architecture. The convolutional neural network reduces the number of model parameters through parameter sharing and local connections, improves training efficiency, and enhances the ability to capture local features. Compared with the original generative adversarial interpolation network composed of fully connected layers, the convolutional layer can better capture the spatial correlation of the input data, which is crucial for multi-sensor time series data with a clear spatial structure. In the encoder-decoder architecture, the convolutional layer forms a more advanced feature representation through layer-by-layer abstraction, while allowing the network to learn the features of the data at multiple scales, which is very helpful for generating more refined and realistic data. In addition, this architecture can also improve the quality and authenticity of generated data in adversarial training, the generator can more accurately simulate the distribution of real data, and the discriminator can more effectively distinguish between true and false data. In the multi-sensor data inpainting task, the encoder-decoder architecture can effectively handle incomplete data, use the intact part to learn the overall structure and characteristics of the data, and then generate a reasonable missing data estimate in the decoder. Therefore, the encoder-decoder architecture composed of CNN has significant advantages in the field of spatial and temporal data processing of multi-sensor data missing.

[0047] Specifically, both the generator and the discriminator include a convolutional layer (Conv Layer), a batch normalization layer (BN Layer), a pooling layer (Pooling Layer), and an upsampling layer (Upsampling Layer). The encoder (i.e., the downsampling stage) is used to extract and compress the high-level features of the input, and the decoder (i.e., the upsampling stage) decompresses the high-level features compressed by the encoder and restores them to an output with the same dimension as the input. In addition, CNN can extract local optimal features from data in both the time domain and the frequency domain, and has a strong feature learning ability for large-scale data sets. In this way, the learning and expression capabilities of the network can be greatly improved.

[0048] During the data repair process, the encoder part gradually extracts the features of the data through convolutional layers and pooling layers, and compresses the spatial dimensions to form feature representations. The decoder part gradually enlarges the size of the feature map through upsampling and convolutional layers, reconstructs the data structure, and restores the dimensions of the original data. Finally, through adversarial training between the encoder and the decoder, the generator learns to generate realistic data to "fool" the discriminator, while the discriminator learns to distinguish between true and false data. This adversarial process prompts the generator to continuously improve the quality of generated data.

[0049] It should be understood that in order to extract features between distant samples, CNN requires a sufficient number of layers or a sufficiently large convolution kernel, but this can easily lead to too many network parameters, thereby reducing the efficiency of network training. Figure 2 As shown in , a self-attention mechanism is embedded between the encoder and decoder of the generator to globally capture the long-distance dependencies between input data and highlight important features. In addition, unlike the convolution kernel whose receptive field is limited by the size of the convolution kernel, the self-attention mechanism provides the network with a global receptive field, thereby making up for the limitations of CNN, allowing the network to focus on more important parts of the global features, enhancing the network's ability to capture global features, and thus improving the overall performance of the network.

[0050] During the data patching process, the self-attention layer first calculates the attention weights of each element in the input data, which reveal the relationship between elements and their importance. Then, the network performs weighted summation of the features according to the calculated weights through weighted feature fusion to highlight important features. Using these weighted features, the model is able to strengthen the features of the non-missing parts that contribute to the missing data patching and generate reasonable data estimates at the missing locations to patch the missing data. In addition, the self-attention mechanism is combined with the encoder-decoder architecture to extract and encode high-level features of the data in the encoder part, and the decoder then uses these features to generate the patching of the missing data. This combination enables the data patching model to flexibly capture long-distance dependencies in sequence data and effectively use the information in the non-missing data to predict and patch the missing parts, improving the accuracy of random missing patching of multi-sensor data and the overall quality of the data.

[0051] In some embodiments of the present application, Figure 2 and Figure 3 As shown, skip connections are also set between the encoder and decoder of the generator and discriminator. It should be understood that problems such as gradient disappearance may occur during network training, resulting in network training failure. In view of this, by setting up skip connections, it can provide a direct path to bypass one or more layers, realize the direct propagation of gradients from the shallow layer to the deep layer of the network, and can directly pass the low-level features of the encoder stage to the decoder stage, thereby bypassing the downsampling and upsampling process, and transferring the underlying details lost in the convolution process to the deeper layers, which is conducive to retaining the underlying details.

[0052] Thus, the jump connection realizes the direct transmission of features between the encoder and the decoder, allowing the decoder to directly access and utilize the feature information in the encoder, thereby improving the efficiency of information utilization. In addition, the jump connection also promotes the fusion of features, by combining the features from the encoder with the features in the decoder, that is, combining low-level detailed features with high-level abstract features, enhancing the network's ability to express features, which helps to improve the model in processing complex tasks, that is, data repair under random missing of multi-sensor data in this technology.

[0053] In addition, jump connections also speed up the network training process, improve the stability and accuracy of the patching effect, and at the same time, by retaining more original information, help prevent network overfitting and enhance the generalization ability of the model.

[0054] Therefore, the generator includes convolutional layers, batch normalization layers, pooling layers, upsampling layers, and self-attention layers; the discriminator includes convolutional layers, batch normalization layers, pooling layers, and upsampling layers. The specific connection method of the jump connection between the encoder and the decoder is to directly connect the output of a certain layer of the encoder to the input of the corresponding layer of the decoder, and concatenate the feature maps of the two in the channel dimension.

[0055] Please continue to refer to Figure 1 The improved GAIN method for repairing random missing of multi-sensor data includes at least steps S110 to S170, which are described in detail as follows (hereinafter, the method is described by taking the application of the method to a terminal device as an example, hereinafter referred to as "terminal"):

[0056] In step S110, an incomplete data matrix in which all sensor data are randomly missing is constructed, and a corresponding mask matrix and random noise matrix are constructed based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix fills the missing positions of the original data with values ​​that are uniformly distributed from -0.01 to 0.01; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents missing data, and 0.5 represents data that needs to be judged by the network.

[0057] In this embodiment, the incomplete data matrix includes sample values ​​collected by multiple sensors at multiple times, and all sensor data in the incomplete data matrix are randomly missing. It should be noted that the present application considers that the missing rate of each sensor data is the same. When the total missing rate remains unchanged, the situation where all sensors have the same missing rate is more difficult to repair. Therefore, the most challenging situation is studied and applied in this technology.

[0058] Therefore, the incomplete data matrix can be used to simulate the data missing situation that may exist in actual engineering. Subsequent network training based on the incomplete data matrix can ensure the network training effect when all sensor data are missing, thereby improving the filling effect of the missing data by the trained network.

[0059] The mask matrix can be used to represent the information of the data missing positions in the incomplete data matrix. The values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the remaining positions are 1; the positions corresponding to the data missing positions in the incomplete data matrix in the random noise matrix are values ​​that obey a uniform distribution of -0.01 to 0.01; the prompt matrix is ​​generated based on the mask matrix, which can prompt the discriminator with some information about the real data. In the prompt matrix, 1 represents the observed original data, 0 represents missing data, and 0.5 represents data that needs to be judged by the network.

[0060] In some embodiments of the present application, when constructing an incomplete data matrix and its corresponding mask matrix, random noise matrix and prompt matrix, the method includes:

[0061] Constructing an incomplete data matrix collected by multiple sensors, wherein the incomplete data matrix includes sampling values ​​of each sensor at different times, and each sensor data has random missing;

[0062] According to the incomplete data matrix, a corresponding mask matrix, a random noise matrix and a prompt matrix are generated.

[0063] Please continue to refer to Figure 1 In step S120, the generator is constructed according to a convolutional neural network with an encoder-decoder architecture, and the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and decoder of the generator to enhance the ability of the network to capture global features; a jump connection technology is introduced between the encoder and decoder of the generator to pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance.

[0064] In this embodiment, the generator adopts the main structure of the encoder-decoder architecture, which can better capture the complex input-to-output mapping relationship; the convolution layer can extract high-level local features of the data; the pooling layer can reduce the dimension of the data features and reduce the number of parameters, suppressing possible overfitting problems; the upsampling layer is used to restore the feature size and channel dimensions in the decoder stage; the batch normalization layer can make the network training more stable and improve the generalization ability of the model. The self-attention layer can enhance the network's ability to capture global features; the jump connection technology can pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance.

[0065] The generator can attempt to generate data similar to real samples when all sensor data are randomly missing, so as to confuse the discriminator's judgment on the authenticity of the data;

[0066] In step S130, the discriminator is constructed according to a convolutional neural network with an encoder-decoder architecture, and the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and effectively alleviate the problem of gradient disappearance.

[0067] In this embodiment, the discriminator can judge the authenticity of the data generated by the generator as accurately as possible when all sensor data are randomly missing.

[0068] In step S140, the incomplete data matrix, the mask matrix and the random noise matrix are used as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain the data prediction matrix output by the generator.

[0069] In this embodiment, the input of the generator consists of an incomplete data matrix , the mask matrix and the random noise matrix The random noise matrix Z The dimensions and mask matrix M Similarly, when inputting the generator, it only adds random noise to the data missing positions of the incomplete data matrix. The generator can numerically patch the data missing positions in the incomplete data matrix according to the input, thereby obtaining a data prediction matrix.

[0070] In one embodiment, the generator outputs a prediction value for each position of the incomplete data matrix based on the random noise matrix to obtain an initial data prediction matrix; the target data prediction matrix output by the generator consists of two parts: data of non-missing parts in the original data and predicted data of missing positions of the original data in the initial data prediction matrix.

[0071] In this example, the initial output of the generator may look like this:

[0072]

[0073] in, Represents the data matrix after generator interpolation (i.e., the initial data prediction matrix); Represents the operation of the generator; represents the mask matrix; represents the random noise matrix; represents the Hadamard product, which is the product of corresponding elements in the matrix; Represents an incomplete data matrix.

[0074] It is worth noting that although the non-missing parts of the incomplete data matrix are true values, the generator will output a predicted value for each position, and there is still a certain error between the predicted value and the true value. Therefore, the target data prediction matrix obtained by the generator is , should consist of two parts: the data that are not missing in the original data and the predicted values ​​generated by the generator at the data missing position of the original data, that is:

[0075] .

[0076] Please continue to refer to Figure 1 In step S150, the data prediction matrix and the prompt matrix are input into the discriminator to be trained, so that the discriminator predicts the probability of true or false data at each position according to the data prediction matrix; in the process of the discriminator judging the true or false data, the prompt matrix can provide the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution.

[0077] In this embodiment, when determining the data prediction matrix Afterwards, the terminal can predict the data matrix And the prompt matrix The prompt matrix can be used as auxiliary information to help the discriminator more accurately predict the authenticity of the data. The output of the discriminator can be expressed as:

[0078]

[0079] in, is the output of the discriminator, i.e., the mask prediction matrix, D Represents the operation of the discriminator.

[0080] In step S160, adversarial training is used to improve the deceptiveness of the data prediction matrix generated by the generator to the discriminator and to improve the accuracy of the discriminator in distinguishing the authenticity of data in the data prediction matrix.

[0081] In this embodiment, the generator is optimized based on the prediction results of the discriminator to improve the deceptiveness of the data prediction matrix generated by the generator to the discriminator, and the discriminator learns the robust features of real data from more and more "real" generated data and can accurately judge the authenticity of the data, thereby improving the performance of the generator and the discriminator through the process of adversarial training.

[0082] Specifically, after obtaining the mask prediction matrix output by the discriminator After that, the terminal can use the mask matrix The difference between the two is used to optimize the generator and the discriminator, improve the deception of the data prediction matrix generated by the generator to the discriminator, and improve the accuracy of the mask prediction matrix predicted by the discriminator. The terminal can perform multiple rounds of training on the generator and the discriminator until a certain number of training times or a certain accuracy requirement is reached.

[0083] In step S170, data repair is performed on the incomplete data with random missing data actually collected by all sensors according to the trained generator.

[0084] In this embodiment, when it is necessary to perform data repair on the sensor data actually collected, the terminal can call the trained generator and input the incomplete data of all sensors actually collected into the generator, so as to perform data repair on the data missing locations in all sensor data through the generator.

[0085] Therefore, based on Figure 1In the embodiment shown, by obtaining the mask matrix, random noise matrix and prompt matrix corresponding to the data matrix of all incomplete sensor data, the value of the position corresponding to the data missing position in the incomplete data matrix in the mask matrix is ​​0, and the other positions are 1; the position corresponding to the data missing position in the incomplete data matrix in the random noise matrix is ​​a value that obeys a uniform distribution of -0.01 to 0.01; in the prompt matrix, 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network. The incomplete data matrix, the mask matrix and the random noise matrix are used as the input of the generator to be trained, so that the generator numerically repairs the data missing position in the incomplete data matrix according to the input, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the original data missing position, to obtain the data prediction matrix output by the generator. The data prediction matrix and the prompt matrix are input into the discriminator so that the discriminator predicts the mask matrix according to the input, and then the generator is optimized according to the prediction result of the discriminator, so as to improve the deception of the data prediction matrix generated by the generator to the discriminator, that is, to improve the authenticity of the data filled by the generator. After the training of the generator is completed, the data can be repaired according to the incomplete data with random missing of all sensors actually collected. In this way, even if all sensor data are randomly missing, the training effect of the network can be guaranteed, thereby improving the repair effect of the trained network on the missing data of all sensors.

[0086] In some embodiments of the present application, the loss function of the generator includes a binary cross entropy loss function and a mean square error loss function. The binary cross entropy loss function is used to calculate the discriminant loss of the discriminator, and the mean square error loss function is used to make the output of the generator in the non-missing part of the data close to the observed data.

[0087] In this embodiment, the loss function of the generator includes a binary cross entropy loss function and a mean square error loss function. The binary cross entropy loss can be used to calculate the discriminant loss of the discriminator and optimize the discriminator and the generator. The mean square error loss function is to ensure that the output of the generator in the non-missing part is close to the observed data and better generate real data.

[0088] In one example, the loss function of the generator can be determined according to the following formula:

[0089]

[0090] in, G represents the operation of the generator, is the network parameter corresponding to the generator, represents the loss function that minimizes the generator, It means to find the mathematical expectation. M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, D represents the operation of the discriminator, is the weight, is the initial data prediction matrix, H is the prompt matrix.

[0091] The discriminator is trained to output values ​​close to 1 for real data and close to 0 for fake data (data filled by the generator). Therefore, the loss function of the discriminator can be determined according to the following formula:

[0092]

[0093] in, D represents the operation of the discriminator, is the network parameter corresponding to the discriminator, represents the loss function that maximizes the discriminator, It means to find the mathematical expectation. M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, H is the prompt matrix.

[0094] Based on the technical solution of the above embodiment, a specific application scenario of the embodiment of the present application is introduced below:

[0095] Taking a pedestrian bridge on the campus of Tufts University in Medford, Massachusetts, USA as an example, a continuous wireless SHM system was installed on the bridge to continuously monitor the vibration response and environmental conditions of the bridge.

[0096] After acquiring the monitoring data, the collected data is preprocessed. First, the data of multiple sensors are processed by a low-pass filter with a cutoff frequency of 16 Hz. Then, in order to speed up the training speed of the network and improve the accuracy, the minimum-maximum normalization is used to normalize the data of all sensors to the range of 0 to 1.

[0097] After preprocessing, each sensor randomly and independently discards a certain proportion of sampling points to simulate data missing in actual projects. Considering that the missing rate of data in actual projects is relatively low, high missing rate is relatively rare. Therefore, three random missing conditions are considered, with missing rates of 10%, 20%, and 30%, respectively. In addition, the missing rate of each sensor is set to be the same. However, when the total missing rate remains unchanged, the situation where all sensors have the same missing rate is more difficult to fix.

[0098] Finally, to enhance the data set, the training data can be segmented using a sliding window. The window size is 128, the overlap rate is 50%, and the sliding step length is 64. It is worth noting that this technology only uses the incomplete data set after the missing to train the network, and the true value of the missing position is only used to calculate the error of the data repair result in the end.

[0099] In order to measure the effect of this application on the missing data repair, the L1 norm relative error between the true value and the repaired value can be selected and L2 norm relative error As the evaluation criteria, the specific formula is as follows:

[0100] ;

[0101] in, represents the real observed data at the missing location, Represents the patched data, Represents the number of missing points.

[0102] Among them, the L1 norm relative error Can be used as an evaluation indicator to measure the overall performance, L2 norm relative error It can be used as a more sensitive performance indicator and is more sensitive to outliers.

[0103] In order to show the repair results of the data more clearly, only fragments of sampling points with certain lengths are selected to show the comparison between the repair results and the real data. Figure 4 and Figure 5 The time domain patching results of 800 randomly selected sample points for the data of two sensors (Sensor 1 and Sensor 2 in the aforementioned multiple sensors) are compared when the missing rate is 30%. The blue points represent the positions of the missing sample points, which is consistent with the random missing pattern.

[0104] from Figure 4 and Figure 5 It can be seen that the patched value of the missing position is very close to the real data, which shows that the data filling method provided by the present application can well patch the missing data even when all sensors have the same missing rate and only use incomplete data sets for training.

[0105] In order to further demonstrate the data repair effect, the real data and the repaired data can be compared in the frequency domain. Figure 6 and Figure 7 The frequency domain diagram of the corresponding segments of the real data and the patched data of sensor 1 and sensor 2 when the missing rate is 30% is shown. Figure 6 and Figure 7It can be seen that in terms of frequency domain performance, although all sensor data are missing, the frequency domain diagram of the repaired signal is basically consistent with the real signal. Therefore, whether in the time domain or the frequency domain, the data repair method provided by this application shows good performance.

[0106] Similarly, a data set with a random missing rate of 10% and 20% for all sensor data is used to test the effectiveness of the proposed data repair technology. The data repair method provided in this application can also show excellent repair performance.

[0107] The following describes an apparatus embodiment of the present application, which can be used to execute an improved GAIN method for repairing random missing of multi-sensor data in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of an improved GAIN method for repairing random missing of multi-sensor data.

[0108] Figure 8 A block diagram of an improved GAIN repair device for random missing of multi-sensor data according to an embodiment of the present application is shown.

[0109] Reference Figure 8 As shown, according to an embodiment of the present application, an improved GAIN device for repairing random missing multi-sensor data is used to repair the missing data of all sensors when there are random missing multi-sensor data. The improved GAIN includes a generator and a discriminator. The generator and the discriminator are both constructed by a convolutional neural network with an encoder-decoder architecture to enhance the learning and expression capabilities of the network; a self-attention mechanism is embedded between the encoder and the decoder of the generator to extract important features, and the skip connection technology is introduced in the generator and the discriminator to alleviate the problem of gradient disappearance and improve the utilization rate of features;

[0110] The device includes:

[0111] An acquisition module is used to construct an incomplete data matrix in which all sensor data are randomly missing, and to construct a corresponding mask matrix and a random noise matrix based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix is ​​filled with values ​​uniformly distributed from -0.01 to 0.01 at the original data missing positions; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network;

[0112] The first building module is used to build the generator according to a convolutional neural network with an encoder-decoder architecture, wherein the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and the decoder of the generator to enhance the ability of the network to capture global features; a skip connection technology is introduced between the encoder and the decoder of the generator to pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance;

[0113] The second construction module is used to construct the discriminator according to a convolutional neural network with an encoder-decoder architecture, wherein the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and effectively alleviate the problem of gradient disappearance;

[0114] A first input module, used to use the incomplete data matrix, the mask matrix and the random noise matrix as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain a data prediction matrix output by the generator;

[0115] A second input module is used to input the data prediction matrix and the prompt matrix into the discriminator to be trained, so that the discriminator predicts the probability of true or false data of each position according to the data prediction matrix; in the process of the discriminator judging the true or false data, the prompt matrix can provide the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution;

[0116] An optimization module, used to improve the deception of the data prediction matrix generated by the generator to the discriminator and improve the accuracy of the discriminator in distinguishing the authenticity of data in the data prediction matrix through adversarial training;

[0117] The processing module is used to perform data repair on the incomplete data with random missing data actually collected by all sensors according to the trained generator.

[0118] In some embodiments of the present application, the generator outputs a prediction value for each position of the incomplete data matrix based on the random noise matrix to obtain an initial data prediction matrix; the target data prediction matrix output by the generator consists of two parts: data of non-missing parts in the original data and predicted data of missing positions of the original data in the initial data prediction matrix.

[0119] In some embodiments of the present application, the loss function of the generator includes a binary cross entropy loss function and a mean square error loss function. The binary cross entropy loss function is used to calculate the discriminant loss of the discriminator, and the mean square error loss function is used to make the output of the generator in the non-missing part of the data close to the observed data.

[0120] In some embodiments of the present application, the loss function of the generator is determined according to the following formula:

[0121]

[0122] in, G represents the operation of the generator, is the network parameter corresponding to the generator, represents the loss function that minimizes the generator, It means to find the mathematical expectation. M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, D represents the operation of the discriminator, is the weight, is the initial data prediction matrix, H is the prompt matrix.

[0123] In some embodiments of the present application, the loss function of the discriminator is determined according to the following formula:

[0124]

[0125] in, D represents the operation of the discriminator, is the network parameter corresponding to the discriminator, represents the loss function that maximizes the discriminator, It means to find the mathematical expectation. M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, H is the prompt matrix.

[0126] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0127] It should be noted that Fig. 9 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] like Fig. 9 As shown, the computer system includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method described in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, the ROM 702 and the RAM 703 are connected to each other through the bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0129] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0130] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0131] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0132] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0134] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0135] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0136] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0137] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0138] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An improved GAIN method for repairing random missing of multi-sensor data, used to repair missing data of all sensors when multi-sensor data are randomly missing, characterized in that: The improved GAIN includes a generator and a discriminator, and both the generator and the discriminator are constructed by a convolutional neural network with an encoder-decoder architecture to enhance the learning and expression capabilities of the network; a self-attention mechanism is embedded between the encoder and the decoder of the generator to extract important features, and skip connection technology is introduced in both the generator and the discriminator to alleviate the problem of gradient disappearance and improve the utilization rate of features; The method comprises: An incomplete data matrix in which all sensor data are randomly missing is constructed, and a corresponding mask matrix and a random noise matrix are constructed based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix fills the missing positions of the original data with values ​​uniformly distributed from -0.01 to 0.01; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network; The generator is constructed according to a convolutional neural network with an encoder-decoder architecture, and the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and the decoder of the generator to enhance the ability of the network to capture global features; a jump connection technology is introduced between the encoder and the decoder of the generator, the output of a certain layer of the encoder is directly connected to the input of the corresponding layer of the decoder, and the feature maps of the two are spliced ​​in the channel dimension to pass the underlying features to the decoder stage to alleviate the problem of gradient disappearance; The discriminator is constructed according to a convolutional neural network with an encoder-decoder architecture, wherein the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and alleviate the problem of gradient vanishing; The incomplete data matrix, the mask matrix and the random noise matrix are used as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain the target data prediction matrix output by the generator; The target data prediction matrix and the prompt matrix are input into the discriminator to be trained, so that the discriminator predicts the probability of each position data being true or false according to the target data prediction matrix; in the process of the discriminator judging the true or false of the data, the prompt matrix provides the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution; Through adversarial training, the deception of the target data prediction matrix generated by the generator to the discriminator is improved, and the accuracy of the discriminator in distinguishing the authenticity of data in the target data prediction matrix is ​​improved; The generator that has completed training is used to repair the incomplete data that are randomly missing from all sensors actually collected.

2. The method according to claim 1, characterized in that When constructing an incomplete data matrix and its corresponding mask matrix, random noise matrix and prompt matrix, the method includes: Constructing a missing data matrix collected by multiple sensors, wherein the incomplete data matrix includes sampling values ​​of each sensor at different times, and each sensor data has random missing values; According to the incomplete data matrix, a corresponding mask matrix, a random noise matrix and a prompt matrix are generated.

3. The method according to claim 1, characterized in that The generator outputs a prediction value for each position of the incomplete data matrix according to the random noise matrix to obtain an initial data prediction matrix; the target data prediction matrix output by the generator consists of two parts: data of the non-missing part of the original data and predicted data of the missing position of the original data in the initial data prediction matrix.

4. The method according to claim 3, characterized in that The loss function of the generator includes a binary cross entropy loss function and a mean square error loss function. The binary cross entropy loss function is used to calculate the discriminant loss of the discriminator, and the mean square error loss function is used to make the output of the generator in the non-missing part of the data close to the observed data.

5. The method according to claim 4, characterized in that The loss function of the generator is determined according to the following formula: in, M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, D represents the operation of the discriminator, is the weight, is the initial data prediction matrix, H is the prompt matrix.

6. The method according to claim 4, characterized in that The loss function of the discriminator is determined according to the following formula: Where M is the mask matrix, represents the Hadamard product, is the target data prediction matrix output by the generator, H is the prompt matrix, D Represents the operation of the discriminator.

7. An improved GAIN device for repairing random missing of multi-sensor data, used to repair missing data of all sensors when multi-sensor data are randomly missing, characterized in that: The improved GAIN includes a generator and a discriminator, and both the generator and the discriminator are constructed by a convolutional neural network with an encoder-decoder architecture to enhance the learning and expression capabilities of the network; a self-attention mechanism is embedded between the encoder and the decoder of the generator to extract important features, and skip connection technology is introduced in both the generator and the discriminator to alleviate the problem of gradient disappearance and improve the utilization rate of features; The device comprises: An acquisition module is used to construct an incomplete data matrix in which all sensor data are randomly missing, and to construct a corresponding mask matrix and a random noise matrix based on the incomplete data matrix, wherein the values ​​of the positions corresponding to the data missing positions in the incomplete data matrix in the mask matrix are 0, and the values ​​of the remaining positions are 1; the random noise matrix is ​​filled with values ​​uniformly distributed from -0.01 to 0.01 at the original data missing positions; a prompt matrix is ​​constructed based on the mask matrix, in which 1 represents the observed original data, 0 represents the missing data, and 0.5 represents the data that needs to be judged by the network; The first construction module is used to construct the generator according to a convolutional neural network with an encoder-decoder architecture, wherein the generator includes a convolution layer, a pooling layer, a self-attention layer, a batch normalization layer and an upsampling layer; the convolution layer is used to extract high-level local features of the data; the pooling layer is used to reduce the dimension of the data features and reduce the number of parameters to suppress the overfitting problem; the upsampling layer is used to restore the feature size and channel dimension in the decoder stage; the batch normalization layer is used to make the network training more stable and improve the generalization ability of the model; and a self-attention layer is embedded between the encoder and the decoder of the generator to enhance the ability of the network to capture global features; a jump connection technology is introduced between the encoder and the decoder of the generator, the output of a certain layer of the encoder is directly connected to the input of the corresponding layer of the decoder, and the feature maps of the two are spliced ​​in the channel dimension to pass the bottom layer features to the decoder stage to alleviate the problem of gradient disappearance; The second building module is used to build the discriminator according to a convolutional neural network with an encoder-decoder architecture, wherein the discriminator includes a convolutional layer, a pooling layer, a batch normalization layer, and an upsampling layer; a skip connection is introduced between the encoder and the decoder of the discriminator to facilitate the preservation of underlying detail features and alleviate the problem of gradient disappearance; A first input module is used to use the incomplete data matrix, the mask matrix and the random noise matrix as inputs of the generator to be trained, so that the generator numerically repairs the data missing positions in the incomplete data matrix according to the inputs, and then combines the data of the non-missing part of the original data with the data predicted by the generator at the missing position of the original data to obtain a target data prediction matrix output by the generator; A second input module is used to input the target data prediction matrix and the prompt matrix into the discriminator to be trained, so that the discriminator predicts the probability of each position data being true or false according to the target data prediction matrix; in the process of the discriminator judging the true or false of the data, the prompt matrix provides the discriminator with some information about the real data, so that the discriminator forces the generator to learn the expected data distribution; An optimization module, used to improve the deception of the target data prediction matrix generated by the generator to the discriminator and improve the accuracy of the discriminator in distinguishing the authenticity of data in the target data prediction matrix through adversarial training; The processing module is used to perform data repair on the incomplete data with random missing data actually collected by all sensors according to the trained generator.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the improved GAIN method for repairing multi-sensor data random missing is implemented as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement an improved GAIN method for repairing random missing of multi-sensor data as described in any one of claims 1 to 6.

Images