Satellite Internet of Things semantic communication method for fire monitoring

Through deep learning models and semantic communication technology, the problem of high data transmission delay and cost of satellite IoT in fire monitoring is solved, accurate detection and rapid response of fires are achieved, and transmission load and communication costs are reduced.

CN119048857AActive Publication Date: 2024-11-29NANJING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411020692.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-29
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

Existing satellite IoT technology has problems of data transmission delay, limited bandwidth and high cost in fire monitoring, making it difficult to meet the needs of large-scale data transmission.

Method used

Deep learning model is used for fire image detection and semantic extraction. By creating a shared codebook, the input continuous coded features are mapped into a base vector, and only the discrete index of the base vector is transmitted. The amount of data transmitted is reduced by using similarity comparison, and the fire is judged by fire. The model evaluates the severity of the fire.

Benefits of technology

Accurate detection and rapid response of fires are achieved, transmission load and communication costs are reduced, and the system's response speed and data transmission efficiency are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048857A_ABST
    Figure CN119048857A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of satellite communication, and discloses a satellite Internet of Things semantic communication method for fire monitoring, which comprises four steps of model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic recovery. The model construction and training module constructs and trains a fire detection model, a semantic extraction and recovery model and a fire behavior judgment model, and deploys the trained models to the satellite Internet of Things; the fire image preprocessing module judges whether a fire exists in the image and transmits the original image directly through a channel at regular intervals; the fire image semantic extraction and transmission module performs semantic extraction on the image and transmits the image through a channel; and the fire image semantic recovery module recovers an original image, judges the size of a fire behavior, and outputs an auxiliary decision according to a fire behavior result. According to the invention, the fire hazard is accurately detected, the transmission load can be reduced, the transmission efficiency can be improved, and monitoring personnel can know the size of the fire behavior in time and make an auxiliary decision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of satellite communications, and in particular relates to a satellite Internet of Things semantic communication method for fire monitoring. Background Art

[0002] The traditional method of monitoring forest fires is mainly based on manual observation and the establishment of video surveillance systems. At present, the Internet of Things technology and wireless communication technology can be used to accurately monitor the collected images, conduct data analysis after obtaining specific data, and dispatch resources in time to eliminate hidden dangers in the bud, thereby reducing the occurrence of forest fires.

[0003] However, there are some significant defects in the practical application of ground IoT: the monitoring range of ground IoT devices is usually limited by the terrain and the number of devices, making it difficult to cover vast forest areas; ground devices are easily affected by weather, geographical conditions and natural disasters, resulting in equipment damage and monitoring interruption.

[0004] Satellite IoT technology can effectively overcome the above defects. Satellite IoT uses satellites for wide-area monitoring and data transmission, which can monitor fires and transmit data more efficiently and in real time. The traditional communication methods of satellite IoT mainly include direct satellite communication, store-and-forward communication, and relay satellite communication. Direct satellite communication is the most basic communication method, in which the ground terminal communicates directly with the ground station via satellite. This method is mainly used for real-time data transmission, but it has high requirements for bandwidth and coverage. E. Del Re et al. discussed the application of satellite communication in emergency situations in their research. Store-and-forward communication means that after receiving data from the ground terminal, the satellite temporarily stores the data and then forwards the data to the ground station at the appropriate time. This method is suitable for large-scale data transmission, but there is a certain delay. J. Pelton discussed the application of store-and-forward communication in remote sensing data transmission in his research. Relay satellite communication uses one or more relay satellites to forward signals to cover a wider geographical area. This method has a wide coverage and high reliability, but the system is complex and the cost is high. Liu et al. introduced the relay satellite system in their research.

[0005] Although these traditional communication methods have been widely used in satellite IoT, they also have some limitations: there is a certain delay in data transmission; the bandwidth of satellite communication is limited and it is difficult to meet the needs of large-scale data transmission; the cost is high, and the initial construction and maintenance costs of relay satellite systems and direct satellite communications are high. Summary of the invention

[0006] In order to overcome the deficiencies in the prior art, the present invention provides a satellite Internet of Things semantic communication method for fire monitoring, which effectively detects fire images through a deep learning model, and transmits through semantic communication to avoid a large amount of redundant transmission from the source. This method creates a shared codebook in the encoder-decoder, maps the input continuous coding features into basis vectors in the codebook, and the transmitter only needs to transmit the discrete index of the basis vector. This process converts three-dimensional data into two-dimensional data, and uses similarity comparison for the index. The two-dimensional data can also be converted into one-dimensional data, which greatly reduces the transmission overhead. Finally, the fire judgment model is used to judge the fire of the restored image, evaluate the severity or size of the fire, and output the final decision. In this way, accurate detection of fire is achieved, transmission load is reduced, and the response speed of the system is improved, so that operators can understand the severity of the fire in a timely manner and take timely and effective measures to deal with fire incidents.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0008] The present invention is a satellite Internet of Things semantic communication method for fire monitoring, which is characterized in that the satellite Internet of Things semantic communication method includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows:

[0009] Step 1: Model construction and training: Use deep learning to build and train fire detection models, semantic extraction and recovery models, and fire intensity judgment models. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining.

[0010] Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission cycle of the original image has been reached. If the transmission cycle has been reached, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction.

[0011] Step 3, semantic extraction and transmission of fire images: Use the semantic extraction model in the semantic extraction and recovery model to semantically encode the monitored fire images to obtain the encoded feature vectors, use the nearest neighbor search for the encoded feature vectors, map the feature vectors into basis vectors in the codebook, and thus obtain the corresponding basis vector indexes. The channel only needs to transmit the obtained basis vector indexes; at the same time, store the previous group of indexes of the basis vector indexes, and compare the similarity between the indexes currently to be transmitted and the previous group of indexes. If a similar one is found, only the image index needs to be transmitted. If it is not similar, the complete codebook index needs to be transmitted. At the same time, the channel will transmit the time, longitude and latitude and device number of the current IoT sensor to collect the image;

[0012] Step 4, semantic restoration of fire images: After receiving the complete codebook index, the visible satellite restores the feature vector according to the shared codebook and sends the feature vector to the semantic restoration network to restore the complete image; if the received image index is a picture index, the image is directly restored from the most recently transmitted picture without going through the semantic restoration model; after the complete image is restored, the image is input into the fire intensity judgment model to determine the size of the fire, and the final decision is output based on the fire intensity result; the visible satellite will also receive the original image transmitted directly through the channel, and compare the structural similarity index with the image restored by the model. If the image restored by the model is not good, retraining needs to be started. The visible satellite notifies the fusion center to retrain the semantic extraction and semantic restoration model; at the same time, the visible satellite will also receive the time, longitude and latitude of the current image acquisition and the device number.

[0013] A further improvement of the present invention is that the fire intensity judgment model construction and training in step 1 specifically includes the following steps:

[0014] Step 1.1.1. The fire intensity judgment model is to separate the fire area separately, classify each pixel of the image as having fire or not, and complete the image segmentation task. The fire intensity judgment model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer contains an H-block encoder and an H-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder contains multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire intensity judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The structure on the left side of the model is spliced ​​with the corresponding structure on the right side.

[0015] Step 1.1.2: After the fire intensity judgment model is constructed, the fire intensity judgment model is trained, and the loss function used is the binary cross entropy:

[0016]

[0017] For the dataset of fire image segmentation, it is necessary to manually annotate the fire area and generate a true label mask. The fire judgment model is allowed to predict each pixel of the image and generate a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to achieve the segmentation of the fire area and the judgment of the fire intensity. The proportions of training data and validation data in the fire judgment model dataset are t% and (1-t)% respectively. T refers to the proportion of training data in the entire dataset. Before importing the model for training, the data is shuffled. The maximum number of iterations is E. However, when the performance has not changed substantially within P iterations, it will stop early. The batch size of training is B.

[0018] Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and the parameters of the fire intensity judgment model to the visible satellites.

[0019] A further improvement of the present invention is that in step 1, the fire detection model construction and training specifically includes the following steps:

[0020] Step 1.2.1, the fire detection model classifies the entire image as having fire or not, completing the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus converted into an image segmentation task. The fire detection model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C, and each image is designed to have a unified input size and three RGB channels. The hidden layer includes an L-block encoder and an L-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes multiple CNNs and multiple residual blocks, and the residual block is composed of multiple CNNs. The decoder includes multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire detection model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0021] Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy:

[0022]

[0023] The fire detection model and the fire intensity judgment model use the same data set. They also need to manually annotate the fire area and generate a true label mask. The fire detection model predicts each pixel of the image and generates a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to achieve the segmentation of the fire area and the detection of the fire. The proportion of training data and verification data in the fire detection model data set is t% and (1-t)% respectively. T refers to the proportion of training data in the entire data set. Before importing the model for training, the data is shuffled. The maximum number of iterations is E. However, when the performance has not changed substantially within P iterations, it will stop early. The batch size of training is B.

[0024] Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

[0025] A further improvement of the present invention is that the semantic extraction and restoration model construction in step 1 specifically includes the following steps:

[0026] The semantic extraction and restoration model includes an input layer, a hidden layer and an output layer. The size of the input layer is H 1 ×W 1 ×C, the input layer preprocesses the image, each image is designed to have a unified input size and three RGB channels, the hidden layer contains a K-block encoder and a K-block decoder, the encoder is used to extract semantic information, the decoder is used to restore semantic information, the encoder contains multiple CNNs, multiple residual blocks, 1 deconvolution network and 1 embedding layer, the residual block is composed of multiple CNNs, the decoder contains multiple CNNs, multiple residual blocks and multiple deconvolution neural networks, the residual block is also composed of multiple CNNs, the deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image, the output layer is a two-dimensional deconvolution layer, and finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, the encoder is on the left side of the model, and the decoder is on the right side of the model.

[0027] The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vector in the shared discrete codebook has the same dimension D as the feature vector encoded by the encoder. The encoder maps the input continuous encoding features into the basis vector in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly restores the original basis vector based on the received discrete index.

[0028] A further improvement of the present invention is that in step 1, the semantic extraction and recovery model training specifically includes the following steps:

[0029] Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composed of, D for each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), and maps it to the basis vector z by nearest neighbor search b (s), calculated as follows:

[0030]

[0031] Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s);

[0032] Step 1.3.2: Substitute the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradients are approximated using a straight-through estimator and the gradients are fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e (s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and during the back propagation process, the gradient Passed to the encoder, the encoder output and the decoder input have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ;

[0033] The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as:

[0034]

[0035] Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is:

[0036]

[0037] The loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j || 2 In order to make the basis vectors and their corresponding encoder outputs as close as possible, ||z e (s)-sg[e j ]|| 2 It is used to constrain the output of the encoder and prevent it from going too far away from the basis vector. γ controls the relative learning speed of the encoder. The decoder only To optimize, the encoder passes and ||z e (s)-sg[e j ]|| 2 For optimization, the basis vectors are obtained by ||sg[z e (s)]-e j || 2 Optimize

[0038] Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss It is expressed as:

[0039]

[0040] in, m Represents the number of pixels in the image.

[0041] Step 1.3.3: After the semantic extraction and recovery model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and recovery model to the ground terminal by region, and the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and recovery model to the visible satellites by region.

[0042] A further improvement of the present invention is that the inter-satellite update and model retraining in step 1 specifically include the following steps:

[0043] Step 1.4.1: During the monitoring mission, if the satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update.

[0044] Step 1.4.2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain; after the retraining is completed, since the models in different areas are the same but the parameters are different, only part of the semantic extraction and recovery model parameters are distributed.

[0045] A further improvement of the present invention is that in step 3, the semantic extraction and transmission process of the fire image is specifically as follows:

[0046] Step 3.1: Use the semantic extraction model in the trained semantic extraction and recovery model to semantically encode the monitored fire image. The semantic extraction model has two encoders, which are encoded twice to obtain the encoded feature vector (B 1 , M, M, D);

[0047] Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B 1 ,M,M,D) for the first quantization and obtain the codebook index (B) through the nearest neighbor search 1 ,N,N);

[0048] Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The size of the array is (B 1 ,N×N), the index N that needs to be transmitted i ×N i ,i=1,2,...,B 1 Compare similarity with the index in the array. If similar, only the array index b needs to be transferred. i ,i=1,2,...,B 1 If they are not similar, transmit the complete codebook index N i ×N i ,i=1,2,...,B 1 ;

[0049] Step 3.4: At the same time, the encoder will send the time, longitude and latitude and device number of the current sensor to collect the image.

[0050] A further improvement of the present invention is that in step 4, the semantic restoration of the fire image specifically includes the following steps:

[0051] Step 4.1: If the data received from the visible satellite is a complete codebook index N i ×N i ,i=1,2,...,B 1 , first perform inverse quantization, and the decoder at the receiving end converts the received codebook index N according to the shared codebook ε i ×N i , i = 1, 2, ..., B 1 Recover the codebook basis vector ( N i ,N i ,D ) ,i=1,2,...,B 1, then, the basis vector (N i , N i , D), i = 1, 2, ..., B 1 Input semantic recovery model in semantic extraction and recovery model. The semantic recovery model has two decoders. After the first decoding, the decoded feature vector is quantized for the second time. The size of the quantized codebook index is the same as the first one, that is, N i ×N i ,i=1,2,...,B 1 , and then dequantize to get the eigenvector (M i ,M i ,D),i=1,2,...,B 1 The feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H 1 ×W 1 ×C;

[0052] Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i ,i=1,2,...,B 1 , the receiving end decoder creates an array (B) that can store the previous set of received pictures during the monitoring task execution. 1 , H 1 ×W 1 ×C), directly according to the array index b i ,i=1,2,...,B 1 , from the picture array (B 1 , H 1 ×W 1 ×C) to take out the restored image without going through the semantic restoration model in the semantic extraction and restoration model;

[0053] Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining needs to be started. The visible satellite will notify the fusion center to retrain the semantic extraction and semantic restoration model.

[0054] Step 4.4, after the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire judgment model. The fire is divided into four levels: large fire, medium fire, small fire and no fire. Then, auxiliary decision-making is output based on the fire results. Finally, the time, longitude and latitude of the fire, equipment number, fire size and auxiliary decision-making are output in the form of frame format.

[0055] A further improvement of the present invention is that: in step 3, the index currently to be transmitted is compared with the previous group of indexes for similarity, specifically: during the execution of the monitoring task, the encoder creates an array to store the indexes of the sent pictures, and the array size is (B 1 ,N×N), the decoder creates an array to store the received pictures, the size of the array is (B 1 ,H 1 ×W 1 ×C); the index N currently to be transmitted i ×N i ,i=1,2,...,B 1 The array (B 1 ,N×N) to compare the similarity. If they are similar, only the array index b needs to be transmitted. i ,i=1,2,...,B 1 , the decoder receives the array index b i , i = 1, 2, ..., B 1 From the decoder array (B 1 , H 1 ×W 1 ×C) Take out the corresponding picture; if they are not similar, the complete codebook index N needs to be transmitted i ×N i ,i=1,2,...,B 1 .

[0056] The beneficial effects of the present invention are as follows: the present invention can accurately identify key information in fire images through a deep learning model, thereby improving the accuracy and reliability of fire detection;

[0057] The present invention avoids a large amount of redundant transmission from the source through semantic communication transmission, creates a shared code book that can map the input continuous coding features into the basis vectors in the code book, and the encoder only needs to transmit the discrete index of the basis vector. The semantic communication does not emphasize the accurate restoration of the transmitted signal at the receiving end, and takes the communication of meaning as the goal. The transmitting end directly avoids a large amount of redundant transmission from the source, thereby reducing the amount of transmitted data, saving bandwidth, improving the real-time performance of data transmission, and reducing communication costs.

[0058] The present invention can evaluate the fire intensity of the restored image through the fire intensity judgment model, accurately evaluate the severity or size of the fire, help understand the development of the fire in real time, and take timely and effective rescue measures. This achieves accurate detection of fires, reduces transmission load and improves transmission efficiency, and timely understands the severity of the fire, which helps to quickly take rescue measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an implementation flow chart of the present invention.

[0060] Figure 2 It is a structural schematic diagram of the fire detection model of the present invention.

[0061] Figure 3 It is a schematic diagram of the structure of the fire intensity judgment model of the present invention.

[0062] Figure 4 It is a schematic diagram of the structure of the semantic extraction and recovery model of the present invention.

[0063] Figure 5 The following is a performance comparison chart of training loss, recall rate, and precision rate under different model structures when the model input layer size is 512.

[0064] Figure 6 Performance comparison of the fire semantic extraction and semantic restoration model training loss and structural similarity index under different learning rates and optimizers.

[0065] Figure 7 The performance evaluation results of the proposed fire semantic extraction and semantic recovery model and the structural similarity index of the JPEG method under different modulation modes and signal-to-noise ratios.

[0066] Figure 8 The transmission data volume evaluation results of JPEG, the proposed fire semantic extraction and semantic restoration model, and the proposed fire semantic extraction and semantic restoration model with fusion similarity comparison.

[0067] Fig. 9 This is a performance comparison diagram of the structural similarity index of the proposed fire extraction and semantic recovery model considering parameter update and parameter non-update. DETAILED DESCRIPTION

[0068] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation described is only a part of the implementation of the present invention, not all of the implementations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The present invention is a satellite Internet of Things semantic communication method for fire monitoring. The monitoring image is firstly detected by a fire detection model to detect whether there is a fire, and the image is transmitted in real time when the fire is detected; then, the monitoring image is compressed and transmitted through a semantic extraction and recovery model based on a CNN, and at the same time, a shared code book is created at a receiving end and a transmitting end. The channel does not need to transmit a complete coding vector, but only needs to transmit a code book index, and the receiving end can recover the coding vector according to the shared code book; finally, the fire intensity is judged on the recovered image through a fire intensity judgment model, and the severity of the fire is evaluated, so as to output a final decision.

[0070] like Figure 1 As shown, a satellite Internet of Things semantic communication method for fire monitoring of the present invention includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows:

[0071] Step 1: Model construction and training: Use deep learning to build and train fire detection models, semantic extraction and recovery models, and fire intensity judgment models. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining.

[0072] The fire intensity judgment model construction and training in step 1 specifically include the following steps:

[0073] The implemented fire detection and fire intensity judgment model is based on a convolutional neural network (CNN) and is named the Fire Detection Fire Judgement Model (FDFJM).

[0074] Table 2 shows the performance evaluation results of the fire judgment task under different model structures.

[0075] Table 2

[0076]

[0077] As can be seen from Table 2, considering the results of the five indicators of accuracy, precision, recall, AUC, F1-score and Specificity, the fire intensity judgment model has the best effect when the input layer size is 512 and there are 3 encoders and 3 decoders. Therefore, FDFJM_3En_3De(512) is used as the fire intensity judgment model. Figure 3 Schematic diagram of the FDFJM_3En_3De(512) model structure.

[0078] Step 1.1.1, the fire judgment model is to separate the fire area and classify each pixel of the image as having fire or not, to complete the image segmentation task. The fire judgment model includes an input layer, a hidden layer and an output layer. The image size of the dataset is 254×254×3. Before inputting the model, the bilinear interpolation method is used to convert the image into a uniform size of H×W×C. Two values ​​are set for the model, H=W=256 or 512. The size of the input layer is H×W×C. Each image is designed to have a uniform input size and three RGB channels, such as Figure 3As shown, the hidden layer includes O=3 encoders and O=3 decoders. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes 7 CNNs and 6 residual blocks. The residual block is composed of 2 CNNs. The decoder includes 4 CNNs, 6 residual blocks and 3 deconvolution neural networks. The residual block is also composed of 2 CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolution layer with a Sigmoid activation function. The fire judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The structure on the left side of the model is spliced ​​with the corresponding structure on the right side.

[0079] Step 1.1.2: After the fire intensity judgment model is constructed, the fire intensity judgment model is trained, and the loss function used is the binary cross entropy:

[0080]

[0081] For the dataset of fire image segmentation, it is necessary to manually annotate the fire area and generate a true label mask. The fire judgment model is allowed to predict each pixel of the image and generate a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to achieve the segmentation of the fire area and the judgment of the fire intensity. The proportions of training data and validation data in the fire judgment model dataset are t% and (1-t)% respectively, where t refers to the proportion of training data in the entire dataset. Specifically, t% = 85% and (1-t)% = 15%. Before importing the model for training, the data is shuffled. The maximum number of iterations is E = 50, but when the performance has no substantial change within P = 5 iterations, it will stop early. The batch size of the training is B = 16.

[0082] Figure 5 The performance comparison of training loss, recall rate and precision rate under different model structures (model input layer size is 512) is shown in Figure 2. Figure 5 It can be seen that the training loss has gradually decreased and tended to be stable, and the recall rate and precision rate have gradually increased and tended to be stable, which means that the fire judgment model FDFJM_3En_3De(512) has been trained.

[0083] Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and the parameters of the fire intensity judgment model to the visible satellites.

[0084] The fire detection model construction and training in step 1 specifically include the following steps:

[0085] Table 1 shows the performance evaluation results of fire detection tasks under different model structures.

[0086] Table 1

[0087]

[0088]

[0089] As can be seen from Table 1, the fire detection model has the highest accuracy and the best fire detection task effect when the input layer size is 512 and there are 2 encoders and 2 decoders. Therefore, FDFJM_2En_2De(512) is used as the fire detection model. Figure 2 Schematic diagram of the FDFJM_2En_2De(512) model structure.

[0090] Step 1.2.1, the fire detection model is to classify the entire image as having fire or not, completing the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus transformed into an image segmentation task, such as Figure 2 As shown, the fire detection model includes an input layer, a hidden layer and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer includes L=2 block encoders and L=2 block decoders. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes 7 CNNs and 4 residual blocks. The residual block is composed of 2 CNNs. The decoder includes 5 CNNs, 4 residual blocks and 3 deconvolution neural networks. The residual block is also composed of 2 CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolution layer with a Sigmoid activation function. The fire detection model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0091] Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy:

[0092]

[0093] The fire detection model and the fire intensity judgment model use the same data set. They also need to manually annotate the fire area and generate a true label mask. The fire detection model predicts each pixel of the image and generates a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to achieve the segmentation of the fire area and the detection of the fire. The proportions of training data and validation data in the fire detection model data set are t% and (1-t)% respectively. T refers to the proportion of training data in the entire data set. Specifically, t% = 85% and (1-t)% = 15%. Before importing the model for training, the data is shuffled. The maximum number of iterations is E = 50. However, when the performance has no substantial change within P = 5 iterations, it will stop early. The batch size of the training is B = 16.

[0094] Figure 5 The performance comparison of training loss, recall rate and precision rate under different model structures (model input layer size is 512) is shown in Figure 2. Figure 5 It can be seen that the training loss has gradually decreased and stabilized, and the recall rate and precision rate have gradually increased and stabilized, indicating that the fire detection model FDFJM_2En_2De(512) has been trained.

[0095] Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

[0096] The semantic extraction and recovery model construction in step 1 specifically includes the following steps:

[0097] The semantic extraction and recovery model implemented is based on CNN and is named Fire Disaster oriented Semantic Extraction Semantic Recovery Model (FDSESRM). Figure 4 Schematic diagram of the FDSESRM model structure.

[0098] like Figure 4 As shown, the semantic extraction and restoration model includes an input layer, a hidden layer and an output layer. The size of the input layer is H 1 ×W 1 ×C, specific H 1 =W 1=256, the input layer preprocesses the image, each image is designed to have a unified input size and three RGB channels, the hidden layer contains K=2 block encoders and K=2 block decoders, the encoder is used to extract semantic information, the decoder is used to restore semantic information, the encoder contains 7 CNNs, 4 residual blocks, 1 deconvolution network and 1 embedding layer, the residual block is composed of 2 CNNs, the decoder contains 2 CNNs, 4 residual blocks and 3 deconvolution neural networks, the residual block is also composed of 2 CNNs, the deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image, the output layer is a two-dimensional deconvolution layer, and finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, the encoder is on the left side of the model, and the decoder is on the right side of the model.

[0099] The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vector in the shared discrete codebook has the same dimension D=64 as the feature vector encoded by the encoder. The encoder maps the input continuous encoding features into the basis vector in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly restores the original basis vector based on the received discrete index.

[0100] The training of the semantic extraction and restoration model in step 1 specifically includes the following steps:

[0101] Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composed of, D for each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), and maps it to the basis vector z by nearest neighbor search b (s), calculated as follows:

[0102]

[0103] Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s);

[0104] Step 1.3.2: Substitute the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradients are approximated using a straight-through estimator and the gradients are fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e(s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and during the back propagation process, the gradient Passed to the encoder, the encoder output and the decoder input have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ;

[0105] The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as:

[0106]

[0107] Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is:

[0108]

[0109] The loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j || 2 In order to make the basis vectors and their corresponding encoder outputs as close as possible, ||z e (s)-sg[e j ]|| 2 It is used to constrain the output of the encoder and prevent it from going too far away from the basis vector. γ controls the relative learning speed of the encoder. The decoder only To optimize, the encoder passes and ||z e (s)-sg[e j ]|| 2 For optimization, the basis vectors are obtained by ||sg[z e (s)]-e j || 2 Optimize

[0110] Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss It is expressed as:

[0111]

[0112] Here, m represents the number of pixels in the image.

[0113] Figure 6 The performance comparison of the fire semantic extraction and semantic restoration model training loss and structural similarity index under different learning rates and optimizers is shown in Figure 2. Figure 6 It can be seen that the training loss has gradually decreased and tended to be stable, and the structural similarity index has gradually increased and tended to be stable, indicating that the semantic extraction and semantic recovery model has been trained. The structural similarity index is used to evaluate image quality. The higher the value, the better the restored image. Therefore, it can be seen that when lr = 0.0003 and optimizer = Adam, the model works best. Therefore, the model trained under the condition of lr = 0.0003 and optimizer = Adam is used in the simulation.

[0114] Step 1.3.3: After the semantic extraction and recovery model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and recovery model to the ground terminal by region, and the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and recovery model to the visible satellites by region.

[0115] Figure 7 The performance evaluation results of the fire semantic extraction and semantic restoration model of the present invention and the structural similarity index of the JPEG method under different modulation modes and signal-to-noise ratios are shown in FIG. Figure 7 It can be seen that under the same modulation method, the structural similarity index of FDSESRM is always better than that of JPEG.

[0116] Figure 8 The transmission data volume evaluation result diagram of JPEG, the fire semantic extraction and semantic restoration model of the present invention and the fire semantic extraction and semantic restoration model of the present invention with fusion similarity comparison is shown in FIG. Figure 8 It can be seen that the amount of data of FDSESRM is significantly less than that of JPEG. Taking the transmission time of 30s as an example, the amount of data of FDSESRM is reduced by 74.55% compared with that of JPEG, and the amount of data of FDSESRM combined with similarity comparison is reduced by 81.05% compared with that of JPEG.

[0117] The inter-satellite update and model retraining in step 1 specifically include the following steps:

[0118] Step 1.4.1: During the monitoring mission, if the satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update.

[0119] Step 1.4.2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain; after the retraining is completed, since the models in different areas are the same but the parameters are different, only part of the semantic extraction and recovery model parameters are distributed.

[0120] Fig. 9 This is a performance comparison diagram of the structural similarity index of the fire extraction and semantic recovery model of the present invention when the parameters are updated and when the parameters are not updated. In the figure, the first-level update represents the inter-satellite update, and the second-level update represents the model retraining update. Fig. 9 It can be seen that the structural similarity index of both the first-level update and the second-level update has been improved, and the image restoration quality has improved.

[0121] Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission cycle of the original image has been reached. If the transmission cycle has been reached, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction.

[0122] Step 3: Fire image semantic extraction and transmission:

[0123] Step 3.1: Use the semantic extraction model in the trained semantic extraction and recovery model to semantically encode the monitored fire image. The semantic extraction model has two encoders, which are encoded twice to obtain the encoded feature vector (B 1 ,M,M,D);

[0124] Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B 1 ,M,M,D) for the first quantization and obtain the codebook index (B) through the nearest neighbor search 1 ,N,N);

[0125] Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The size of the array is (B 1 ,N×N), the index N that needs to be transmitted i ×N i ,i=1,2,...,B 1 Compare similarity with the index in the array. If similar, only the array index b needs to be transferred. i ,i=1,2,...,B 1 If they are not similar, transmit the complete codebook index N i ×N i,i=1,2,...,B 1 ;

[0126] Step 3.4: At the same time, the encoder will send the time, longitude and latitude and device number of the current sensor to collect the image.

[0127] Step 4: Fire image semantic restoration:

[0128] The semantic extraction model and the semantic recovery model are trained together. The fire detection model and the fire intensity judgment model are trained in the same way. The data received by the satellite can be divided into three categories: the first category is the codebook index of the complete image, the second category is the array index, and the third category is the original image directly transmitted through the channel. The following steps are included according to the data type:

[0129] Step 4.1: If the data received from the visible satellite is a complete codebook index N i ×N i ,i=1,2,...,B 1 , first perform inverse quantization, and the receiving decoder converts the received codebook index N according to the shared codebook ε i ×N i , i = 1, 2, ..., B 1 Recover the codebook basis vector (N i , N i , D), i = 1, 2, ..., B 1 , then, the basis vector (N i , N i , D), i = 1, 2, ..., B 1 Input semantic recovery model in semantic extraction and recovery model. The semantic recovery model has two decoders. After the first decoding, the decoded feature vector is quantized for the second time. The size of the quantized codebook index is the same as the first one, that is, N i ×N i ,i=1,2,...,B 1 , and then dequantize to get the eigenvector (M i ,M i ,D),i=1,2,...,B 1 The feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H 1 ×W 1 ×C;

[0130] Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i ,i=1,2,...,B 1 , the receiving end decoder creates an array (B) that can store the previous set of received pictures during the monitoring task execution.1 , H 1 ×W 1 ×C), directly according to the array index b i ,i=1,2,...,B 1 , from the picture array (B 1 ,H 1 ×W 1 ×C) to take out the restored image without going through the semantic restoration model in the semantic extraction and restoration model;

[0131] Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining needs to be started. The visible satellite will notify the fusion center to retrain the semantic extraction and semantic restoration model.

[0132] Step 4.4, after the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire judgment model. The fire is divided into four levels: large fire, medium fire, small fire and no fire. Then, auxiliary decision-making is output based on the fire results. Finally, the time, longitude and latitude of the fire, equipment number, fire size and auxiliary decision-making are output in the form of frame format.

[0133] In step 3, the similarity comparison is specifically performed as follows: during the execution of the monitoring task, the encoder creates an array to store the index of the sent pictures, and the array size is (B 1 ,N×N), the decoder creates an array to store the received pictures, the size of the array is (B 1 ,H 1 ×W 1 ×C); the index N currently to be transmitted i ×N i ,i=1,2,...,B 1 Compare the similarity with the index in the array on the encoder side. If they are similar, just transfer the array index b i , i = 1, 2, ..., B 1 , the decoder receives the array index b i ,i=1,2,...,B 1 From the decoder array (B 1 , H 1 ×W 1 ×C) Take out the corresponding picture; if they are not similar, the complete codebook index N needs to be transmitted i ×N i ,i=1,2,...,B 1 .

[0134] In summary, the present invention is aimed at a satellite Internet of Things semantic communication method for fire monitoring, which performs a series of processing on fire images such as fire detection, semantic extraction and transmission, and fire intensity judgment. It can not only accurately detect fires, but also reduce transmission load and improve transmission efficiency, which helps to understand the size of the fire in a timely manner.

[0135] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A satellite Internet of Things semantic communication method for fire monitoring, Features: The satellite Internet of Things semantic communication method includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows: Step 1: Model construction and training: Use deep learning to build and train fire detection models, semantic extraction and recovery models, and fire intensity judgment models. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining. Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission cycle of the original image has been reached. If the transmission cycle has been reached, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction. Step 3, semantic extraction and transmission of fire images: Use the semantic extraction model in the semantic extraction and recovery model to semantically encode the monitored fire images to obtain the encoded feature vectors, use the nearest neighbor search for the encoded feature vectors, map the feature vectors into basis vectors in the codebook, and thus obtain the corresponding basis vector indexes. The channel only needs to transmit the obtained basis vector indexes; at the same time, store the previous group of indexes of the basis vector indexes, and compare the similarity between the indexes currently to be transmitted and the previous group of indexes. If a similar one is found, only the image index needs to be transmitted. If it is not similar, the complete codebook index needs to be transmitted. At the same time, the channel will transmit the time, longitude and latitude and device number of the current IoT sensor to collect the image; Step 4, semantic restoration of fire images: After receiving the complete codebook index, the visible satellite restores the feature vector according to the shared codebook and sends the feature vector to the semantic restoration network to restore the complete image; if the received image index is a picture index, the image is directly restored from the most recently transmitted picture without going through the semantic restoration model; after the complete image is restored, the image is input into the fire intensity judgment model to determine the size of the fire, and the final decision is output based on the fire intensity result; the visible satellite will also receive the original image transmitted directly through the channel, and compare the structural similarity index with the image restored by the model. If the image restored by the model is not good, retraining needs to be started. The visible satellite notifies the fusion center to retrain the semantic extraction and semantic restoration model; at the same time, the visible satellite will also receive the time, longitude and latitude of the current image acquisition and the device number.

2. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: The fire intensity judgment model construction and training in step 1 specifically include the following steps: Step 1.1.1, the fire judgment model is to separate the fire area separately, classify each pixel of the image as having fire or not, and complete the image segmentation task. The fire judgment model includes an input layer, a hidden layer and an output layer. The size of the input layer is H×W×C, and each image is designed to have a unified input size and three RGB channels. The hidden layer contains an O-block encoder and an O-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder contains multiple CNNs, multiple residual blocks and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure. Step 1.1.2: After the fire intensity judgment model is constructed, the fire intensity judgment model is trained, and the loss function used is the binary cross entropy: For the dataset of fire image segmentation, the fire area needs to be manually annotated to generate a true label mask, and the fire intensity judgment model is allowed to predict each pixel of the image to generate a predicted label mask. Through model training, the predicted label mask is made to fit the true label mask to achieve the segmentation of the fire area and the judgment of the fire intensity. The proportions of training data and verification data in the fire judgment model dataset are t% and (1-t)% respectively, and t refers to the proportion of training data in the entire dataset. Before importing the model for training, the data is shuffled. The maximum number of iterations is E, but when the performance has no substantial change within P iterations, it will stop early, and the batch size of training is B; Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and the parameters of the fire intensity judgment model to the visible satellites.

3. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: The fire detection model construction and training in step 1 specifically include the following steps: Step 1.2.1, the fire detection model is to classify the entire image as having fire or not, and complete the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus converted into an image segmentation task. The fire detection model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C, and each image is designed to have a unified input size and three RGB channels. The hidden layer includes an L-block encoder and an L-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes multiple CNNs and multiple residual blocks, and the residual block is composed of multiple CNNs. The decoder includes multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire detection model has a symmetrical structure, with the encoder on the left side of the model and the decoder on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure. Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy: The fire detection model and the fire intensity judgment model use the same data set. They also need to manually annotate the fire area and generate a true label mask. The fire detection model predicts each pixel of the image and generates a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to achieve the segmentation of the fire area and the detection of the fire. The proportions of training data and verification data in the fire detection model data set are t% and (1-t)% respectively, where t refers to the proportion of training data in the entire data set. Before importing the model for training, the data is shuffled. The maximum number of iterations is E. However, when the performance has not changed substantially within P iterations, it will stop early. The batch size of the training is B. Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

4. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: The semantic extraction and recovery model in step 1 includes an input layer, a hidden layer and an output layer. The size of the input layer is H. 1 ×W 1 ×C, the input layer preprocesses the image, each image is designed to have a unified input size and three RGB channels, the hidden layer contains a K-block encoder and a K-block decoder, the encoder is used to extract semantic information, the decoder is used to restore semantic information, the encoder contains multiple CNNs, multiple residual blocks, a deconvolution network and an embedding layer, the residual block is composed of multiple CNNs, the decoder contains multiple CNNs, multiple residual blocks and multiple deconvolution neural networks, the residual block is also composed of multiple CNNs, the deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image, the output layer is a two-dimensional deconvolution layer, and finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, the encoder is on the left side of the model, and the decoder is on the right side of the model; The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vector in the shared discrete codebook has the same dimension D as the feature vector encoded by the encoder. The encoder maps the input continuous encoding features into the basis vector in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly restores the original basis vector based on the received discrete index.

5. According to the satellite Internet of Things semantic communication method for fire monitoring according to claim 1 or 4, Features: The training of the semantic extraction and restoration model in step 1 specifically includes the following steps: Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composed of, D for each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), and maps it to the basis vector z by nearest neighbor search b (s), calculated as follows: Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s); Step 1.3.2: Substitute the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradients are approximated using a straight-through estimator and the gradients are fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e (s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and during the back propagation process, the gradient Passed to the encoder, the encoder output and the decoder input have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ; The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as: Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is: The loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j || 2 In order to make the basis vectors and their corresponding encoder outputs as close as possible, ||z e (s)-sg[e j ]|| 2 It is used to constrain the output of the encoder and prevent it from going too far away from the basis vector. γ controls the relative learning speed of the encoder. The decoder only To optimize, the encoder passes and ||z e (s)-sg[e j ]|| 2 To optimize, the basis vectors are obtained by ||sg[z e (s)]-e j || 2 Optimize Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss It is expressed as: in, m Represents the number of pixels in the image. Step 1.3.3: After the semantic extraction and recovery model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and recovery model to the ground terminal by region; the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and recovery model to the visible satellites by region.

6. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: The inter-satellite update and model retraining in step 1 specifically include the following steps: Step 1.4.1: During the monitoring mission, if the satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update. Step 1.4.

2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain; after the retraining is completed, since the models in different areas are the same but the parameters are different, only part of the semantic extraction and recovery model parameters are distributed.

7. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: In step 3, the semantic extraction and transmission process of the fire image is specifically as follows: Step 3.1: Use the semantic extraction model in the trained semantic extraction and recovery model to semantically encode the monitored fire image. The semantic extraction model has two encoders, which are encoded twice to obtain the encoded feature vector (B 1 , M, M, D); Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B 1 , M, M, D) for the first quantization, and obtain the codebook index (B) through the nearest neighbor search 1 , N, N); Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The size of the array is (B 1 , N×N)' the index N that needs to be transmitted currently i ×N i , i = 1, 2, ..., B 1 Compare similarity with the index in the array. If similar, only the array index b needs to be transferred. i , i = 1, 2, ..., B 1 If they are not similar, transmit the complete codebook index N i ×N i , i = 1, 2, ..., B 1 ; Step 3.4: At the same time, the encoder will send the time, longitude and latitude and device number of the current sensor to collect the image.

8. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: In step 4, the semantic restoration of the fire image specifically includes the following steps: Step 4.1: If the data received from the visible satellite is a complete codebook index N i ×N i , i = 1, 2, ..., B 1 , first perform inverse quantization, and the decoder at the receiving end converts the received codebook index N according to the shared codebook ε i ×N i , i = 1, 2, ..., B 1 Recover the codebook basis vector (N i , N i , D), i = 1, 2, ..., B 1 , then, the basis vector (N i , N i , D), i = 1, 2, ..., B 1 Input semantic recovery model in semantic extraction and recovery model. The semantic recovery model has two decoders. After the first decoding, the decoded feature vector is quantized for the second time. The size of the quantized codebook index is the same as the first one, that is, N i ×N i , i = 1, 2, ..., B 1 , and then dequantize to get the eigenvector (M i , M i , D), i = 1, 2, ..., B 1 The feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H 1 ×W 1 ×C; Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i , i = 1, 2, ..., B 1 , the receiving end decoder creates an array (B) that can store the previous set of received pictures during the monitoring task execution. 1 , H 1 ×W 1 ×C), directly according to the array index b i , i=1,2...,B 1 , from the picture array (B 1 , H 1 ×W 1 ×C) to take out the restored image without going through the semantic restoration model in the semantic extraction and restoration model; Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining needs to be started. The visible satellite will notify the fusion center to retrain the semantic extraction and semantic restoration model. Step 4.4, after the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire judgment model. The fire is divided into four levels: large fire, medium fire, small fire and no fire. Then, auxiliary decision-making is output based on the fire results. Finally, the time, longitude and latitude of the fire, equipment number, fire size and auxiliary decision-making are output in the form of frame format.

9. According to claim 1, a satellite Internet of Things semantic communication method for fire monitoring, Features: The similarity comparison in step 3 is specifically as follows: During the execution of the monitoring task, the encoder creates an array to store the index of the sent pictures, and the array size is (B 1 , N×N), the decoder creates an array to store the received pictures, the size of the array is (B 1 , H 1 ×W 1 ×C); the index N currently to be transmitted i ×N i , i = 1, 2, ..., B 1 Compare the similarity with the index in the array on the encoder side. If they are similar, just transfer the array index b i , i = 1, 2, ..., B 1 , the decoder receives the array index b i , i = 1, 2, ..., B 1 From the decoder array (B 1 , H 1 ×W 1 ×C) Take out the corresponding picture; if they are not similar, the complete codebook index N needs to be transmitted i ×N i , i = 1, 2, ..., B 1 .

Citation Information

Patent Citations

  • Method for training image quantization model and method and device for retrieving image

    CN116051917A

  • Agricultural Internet of Things perception data prediction method based on semantic communication

    CN117220826A

  • Semantic information federal learning method and system oriented to digital twinning in industrial Internet of Things scene

    CN117669699A

  • Task-oriented privacy semantic communication method

    CN117835246A

Cited By

  • Mine video semantic communication transmission method and system based on deep learning

    CN119562069A