A satellite Internet of Things semantic communication method for fire monitoring

Abstract

The present invention belongs to the field of satellite communication technology and discloses a satellite Internet of Things semantic communication method for fire monitoring. The method includes four steps: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The model construction and training module constructs and trains a fire detection model, a semantic extraction and restoration model, and a fire intensity judgment model, and deploys the trained models into the satellite Internet of Things. The fire image preprocessing module determines whether an image contains a fire and periodically transmits the original image directly through a channel. The fire image semantic extraction and transmission module performs semantic extraction on the image and transmits it through a channel. The fire image semantic restoration module restores the original image, determines the extent of the fire, and outputs an auxiliary decision based on the fire intensity result. The present invention accurately detects fires, reduces transmission load, and improves transmission efficiency, helping monitoring personnel to promptly understand the extent of the fire and make auxiliary decisions.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite communications, and in particular relates to a satellite Internet of Things semantic communication method for fire monitoring. Background Art

[0002] Traditional forest fire monitoring methods are mainly based on manual observation and the establishment of video surveillance systems. At present, the Internet of Things technology and wireless communication technology can be used to accurately monitor the collected images, conduct data analysis after obtaining specific data, and dispatch resources in time to eliminate hidden dangers in the bud, thereby reducing the occurrence of forest fires.

[0003] However, there are some significant defects in the actual application of terrestrial IoT: the monitoring range of terrestrial IoT devices is usually limited by the terrain and the number of devices, making it difficult to cover vast forest areas; terrestrial devices are easily affected by weather, geographical conditions and natural disasters, resulting in equipment damage and monitoring interruption.

[0004] Satellite IoT technology can effectively overcome these shortcomings. Using satellites for wide-area monitoring and data transmission, the satellite IoT enables more real-time and efficient fire monitoring and data transmission. Traditional communication methods for the satellite IoT primarily include direct satellite communication, store-and-forward communication, and relay satellite communication. Direct satellite communication is the most basic method, in which a ground terminal communicates directly with a ground station via satellite. This method is primarily used for real-time data transmission, but it requires high bandwidth and coverage. E. Del Re et al. explored the application of satellite communication in emergency situations. Store-and-forward communication involves a satellite temporarily storing data received from a ground terminal and then forwarding it to a ground station at the appropriate time. This method is suitable for large-scale data transmission, but it also involves certain latency. J. Pelton discussed the application of store-and-forward communication in remote sensing data transmission. Relay satellite communication uses one or more relay satellites to relay signals to cover a wider geographic area. This method offers wide coverage and high reliability, but the system is complex and costly. Liu et al. introduced a relay satellite system in their research.

[0005] Although these traditional communication methods have been widely used in satellite Internet of Things, they also have some limitations: there is a certain delay in data transmission; the bandwidth of satellite communication is limited and it is difficult to meet the needs of large-scale data transmission; the cost is high, and the initial construction and maintenance costs of relay satellite systems and direct satellite communications are high. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention provides a satellite Internet of Things semantic communication method for fire monitoring. This method effectively detects fire images through a deep learning model, and transmits them through semantic communication, avoiding a large amount of redundant transmission at the source. This method creates a shared codebook between the encoder and decoder, mapping the input continuous encoding features into basis vectors in the codebook. The transmitter only needs to transmit the discrete indexes of the basis vectors. This process converts three-dimensional data into two-dimensional data. Similarity comparison is used on the indexes, and the two-dimensional data can also be converted into one-dimensional data, which greatly reduces transmission overhead. Finally, a fire intensity judgment model is used to judge the fire intensity of the restored image, assess the severity or size of the fire, and output a final decision. This achieves accurate fire detection, reduces transmission load, and improves the system's response speed, enabling operators to promptly understand the severity of the fire and take timely and effective measures to respond to the fire incident.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] The present invention is a satellite Internet of Things semantic communication method for fire monitoring, characterized in that the satellite Internet of Things semantic communication method includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows:

[0009] Step 1: Model construction and training: Build and train a fire detection model, a semantic extraction and recovery model, and a fire intensity assessment model through deep learning. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining.

[0010] Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission period for the original image has been reached. If so, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction.

[0011] Step 3: Semantic extraction and transmission of fire images: Use the semantic extraction model in the semantic extraction and recovery model to semantically encode the monitored fire images to obtain the encoded feature vectors. Use the nearest neighbor search on the encoded feature vectors to map the feature vectors into basis vectors in the codebook, thereby obtaining the corresponding basis vector indexes. The channel only needs to transmit the obtained basis vector indexes. At the same time, the previous set of basis vector indexes are stored, and the index to be transmitted is compared with the previous set of indexes for similarity. If a similar one is found, only the image index needs to be transmitted. If it is not similar, the complete codebook index needs to be transmitted. At the same time, the channel will transmit the time, longitude and latitude and device number of the current IoT sensor to collect the image.

[0012] Step 4: Semantic restoration of fire images: After receiving the complete codebook index, the visible satellite recovers the feature vector based on the shared codebook and feeds the feature vector into the semantic restoration network to restore the complete image. If it receives an image index, it directly recovers the image from the most recently transmitted image without going through the semantic restoration model. After the complete image is restored, it is input into the fire intensity judgment model to determine the size of the fire and output the final decision based on the fire intensity result. The visible satellite also receives the original image transmitted directly through the channel and compares the structural similarity index with the image restored by the model. If the image restored by the model is not good, retraining is required. The visible satellite notifies the fusion center to retrain the semantic extraction and semantic restoration model. At the same time, the visible satellite also receives the time, longitude and latitude of the current image acquisition and the device number.

[0013] A further improvement of the present invention is that the fire intensity judgment model construction and training in step 1 specifically includes the following steps:

[0014] Step 1.1.1. The fire intensity judgment model is to separate the fire area separately, classify each pixel of the image as having fire or not, and complete the image segmentation task. The fire intensity judgment model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer contains an H-block encoder and an H-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder contains multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire intensity judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0015] Step 1.1.2: After the fire intensity judgment model is built, the fire intensity judgment model is trained. The loss function used is binary cross entropy:

[0016]

[0017] For fire image segmentation datasets, the fire areas need to be manually annotated to generate a true label mask. The fire severity assessment model then makes predictions for each pixel in the image, generating a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask, achieving segmentation of the fire area and fire severity assessment. The training data and validation data in the fire assessment model dataset are t% and (1-t)%, respectively, where t refers to the percentage of training data in the entire dataset. Before training the model, the data is shuffled. The maximum number of iterations is E. However, if performance does not change substantially within P iterations, early stopping is performed. The training batch size is B.

[0018] Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and its parameters to visible satellites.

[0019] A further improvement of the present invention is that in step 1, the fire detection model construction and training specifically includes the following steps:

[0020] Step 1.2.1. The fire detection model classifies the entire image as having fire or not to complete the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus converted into an image segmentation task. The fire detection model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer includes an L-block encoder and an L-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder includes multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire detection model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0021] Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy:

[0022]

[0023] The fire detection model and the fire intensity assessment model use the same dataset. Similarly, fire areas must be manually annotated to generate a true label mask. The fire detection model then predicts each pixel in the image, generating a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask, achieving segmentation of the fire area and fire detection. The training data and validation data in the fire detection model dataset consist of t% and (1-t)%, respectively, where t refers to the percentage of training data in the entire dataset. Before training, the data is shuffled. The maximum number of iterations is E. However, if performance does not change substantially within P iterations, early stopping is performed. The training batch size is B.

[0024] Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

[0025] A further improvement of the present invention is that the semantic extraction and restoration model construction in step 1 specifically includes the following steps:

[0026] The semantic extraction and restoration model includes an input layer, a hidden layer and an output layer. The size of the input layer is H1×W1×C. The input layer preprocesses the image. Each image is designed to have a unified input size and three RGB channels. The hidden layer contains a K-block encoder and a K-block decoder. The encoder is used to extract semantic information, and the decoder is used to restore semantic information. The encoder contains multiple CNNs, multiple residual blocks, a deconvolution network and an embedding layer. The residual block is composed of multiple CNNs. The decoder contains multiple CNNs, multiple residual blocks and multiple deconvolution neural networks. The residual block is also composed of multiple CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional deconvolution layer, which finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, with the encoder on the left side of the model and the decoder on the right side of the model.

[0027] The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vectors in the shared discrete codebook have the same dimension D as the feature vectors encoded by the encoder. The encoder maps the input continuous encoded features into the basis vectors in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly recovers the original basis vector based on the received discrete index.

[0028] A further improvement of the present invention is that the semantic extraction and restoration model training in step 1 specifically includes the following steps:

[0029] Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composition, D is each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), which is mapped to the basis vector z by nearest neighbor search b (s), calculated as follows:

[0030]

[0031] Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s);

[0032] Step 1.3.2, the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradient is approximated using a straight-through estimator and the gradient is fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e (s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and in the back propagation process, the gradient Passed to the encoder, the output of the encoder and the input of the decoder have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ;

[0033] The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as:

[0034]

[0035] Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is:

[0036]

[0037] In the loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j ||2 In order to make the basis vector and its corresponding encoder output as close as possible, ||z e (s)-sg[e j ]||2 is used to constrain the output of the encoder, not letting it go too far away from the basis vector, γ controls the relative learning speed of the encoder, and the decoder only passes To optimize, the encoder passes and ||z e (s)-sg[e j ]||2 is optimized, and the basis vectors are optimized by ||sg[z e (s)]-e j ||2 for optimization;

[0038] Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss Expressed as:

[0039]

[0040] Here, m represents the number of pixels in the image.

[0041] Step 1.3.3: After the semantic extraction and restoration model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and restoration model to the ground terminal by region, and the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and restoration model to the visible satellites by region.

[0042] A further improvement of the present invention is that the inter-satellite update and model retraining in step 1 specifically include the following steps:

[0043] Step 1.4.1: During the monitoring mission, if a satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update.

[0044] Step 1.4.2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain. After retraining is completed, since the models in different areas are the same but the parameters are different, only some of the semantic extraction and recovery model parameters are distributed.

[0045] A further improvement of the present invention is that in step 3, the semantic extraction and transmission process of the fire image is specifically as follows:

[0046] Step 3.1: Use the semantic extraction model in the trained semantic extraction and restoration model to semantically encode the monitored fire image. The semantic extraction model has two encoders and performs two encodings to obtain the encoded feature vector (B1, M, M, D).

[0047] Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B1, M, M, D) is quantized for the first time, and the codebook index (B1, N, N) is obtained through the nearest neighbor search.

[0048] Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The array size is (B1, N×N), and the index N that needs to be transmitted is i ×N i ,i=1,2,...,B1 is compared with the index in the array for similarity. If they are similar, only the array index b is transmitted. i ,i=1,2,...,B1, if they are not similar, transmit the complete codebook index N i ×N i ,i=1,2,...,B1;

[0049] Step 3.4: At the same time, the encoder will send the time, longitude and latitude of the current sensor to collect the image, and the device number.

[0050] A further improvement of the present invention is that in step 4, the semantic restoration of the fire image specifically includes the following steps:

[0051] Step 4.1: If the data received from the visible satellite is a complete image, the codebook index N i ×N i ,i=1,2,...,B1, first perform inverse quantization, the decoder at the receiving end converts the received codebook index N according to the shared codebook ε i ×N i , i=1,2,...,B1 recovers the codebook basis vector ( N i ,N i ,D ) ,i=1,2,...,B1, then, the basis vector (N i , N i , D), i = 1, 2, ..., B1 input semantic recovery model in semantic extraction and recovery model, semantic recovery model has two decoders, after the first decoding, the decoded feature vector is quantized for the second time, the quantized codebook index size is the same as the first time, that is, N i ×N i ,i=1,2,...,B1, then dequantize to get the eigenvector (Mi ,M i ,D),i=1,2,...,B1, the feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H1×W1×C;

[0052] Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i ,i=1,2,...,B1, the receiving end decoder creates an array (B1, H1×W1×C) that can store the previous set of received pictures during the monitoring task execution, directly according to the array index b i ,i=1,2,...,B1, take out the restored image from the image array (B1, H1×W1×C) without going through the semantic restoration model in the semantic extraction and restoration model;

[0053] Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining is required. The visible satellite will notify the fusion center to retrain the semantic extraction and restoration model.

[0054] Step 4.4: After the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire intensity judgment model. Fires are divided into four levels: large fire, medium fire, small fire, and no fire. Then, an auxiliary decision is output based on the fire intensity results. Finally, the time, longitude and latitude of the fire, device number, fire size, and auxiliary decision are output in frame format.

[0055] A further improvement of the present invention is that: in the step 3, the index currently to be transmitted is compared with the previous set of indexes for similarity, specifically: during the execution of the monitoring task, the encoder creates an array to store the index of the sent picture, the array size is (B1, N×N), the decoder creates an array to store the received picture, the array size is (B1, H1×W1×C); the index N currently to be transmitted is compared with the previous set of indexes. i ×N i ,i=1,2,...,B1 is compared with the index in the array (B1,N×N) on the encoder side for similarity. If they are similar, only the array index b is transmitted. i ,i=1,2,...,B1, the decoder will receive the array index b i , i = 1, 2, ..., B1 takes the corresponding picture from the decoder array (B1, H1×W1×C); if they are not similar, the complete codebook index N needs to be transmitted i ×N i,i=1,2,...,B1.

[0056] The beneficial effects of the present invention are: the present invention can accurately identify key information in fire images through a deep learning model, thereby improving the accuracy and reliability of fire detection;

[0057] The present invention uses semantic communication for transmission to avoid a large amount of redundant transmission from the source. It creates a shared codebook that can map the input continuous coding features into basis vectors in the codebook. The encoder only needs to transmit the discrete index of the basis vector. Semantic communication does not emphasize the accurate restoration of the transmitted signal at the receiving end. The goal is to communicate the meaning. The sending end directly avoids a large amount of redundant transmission from the source, thereby reducing the amount of transmitted data, saving bandwidth, improving the real-time performance of data transmission, and reducing communication costs.

[0058] Through the fire intensity judgment model, the present invention can evaluate the fire intensity of the restored image, accurately assess the severity or size of the fire, help understand the development of the fire in real time, and take timely and effective rescue measures. This achieves accurate detection of fires, reduces transmission load and improves transmission efficiency, and timely understands the severity of the fire, which helps to quickly take rescue measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an implementation flow chart of the present invention.

[0060] Figure 2 It is a structural schematic diagram of the fire detection model of the present invention.

[0061] Figure 3 It is a structural schematic diagram of the fire intensity judgment model of the present invention.

[0062] Figure 4 It is a structural diagram of the semantic extraction and recovery model of the present invention.

[0063] Figure 5 The following is a performance comparison of training loss, recall rate, and precision rate under different model structures when the model input layer size is 512.

[0064] Figure 6 Performance comparison of the fire semantic extraction and semantic restoration model training loss and structural similarity index under different learning rates and optimizers.

[0065] Figure 7 The performance evaluation results of the structural similarity index of the proposed fire semantic extraction and semantic recovery model and the JPEG method under different modulation modes and signal-to-noise ratios are shown in the figure.

[0066] Figure 8 The transmission data volume evaluation results of JPEG, the proposed fire semantic extraction and semantic restoration model, and the proposed fire semantic extraction and semantic restoration model with fusion similarity comparison.

[0067] Figure 9 The performance comparison results of the structural similarity index of the proposed fire extraction and semantic recovery model considering parameter update and parameter non-update. DETAILED DESCRIPTION

[0068] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described implementation is only a part of the implementation of the present invention, not the entire implementation. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0069] The present invention is a satellite Internet of Things semantic communication method for fire monitoring. The monitoring image is first detected by a fire detection model to determine whether there is a fire. If a fire is detected, the image is transmitted in real time. Then, the monitoring image is compressed and transmitted using a semantic extraction and recovery model based on a CNN. At the same time, a shared codebook is created at the receiving and transmitting ends. The complete coding vector does not need to be transmitted in the channel, only the codebook index needs to be transmitted. The receiving end can recover the coding vector based on the shared codebook. Finally, the fire intensity is judged on the recovered image using a fire intensity judgment model to evaluate the severity of the fire and output a final decision.

[0070] like Figure 1 As shown, the satellite Internet of Things semantic communication method for fire monitoring of the present invention includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows:

[0071] Step 1: Model construction and training: Build and train a fire detection model, a semantic extraction and recovery model, and a fire intensity assessment model through deep learning. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining.

[0072] The fire intensity judgment model construction and training in step 1 specifically include the following steps:

[0073] The implemented fire detection and fire intensity judgment model is based on Convolutional Neural Network (CNN) and is named Fire Detection Fire Judgement Model (FDFJM).

[0074] Table 2 shows the performance evaluation results of the fire intensity judgment task under different model structures.

[0075] Table 2

[0076]

[0077] As can be seen from Table 2, considering the results of the five indicators of accuracy, precision, recall, AUC, F1-score and specificity, the fire intensity judgment model has the best effect when the input layer size is 512 and there are 3 encoders and 3 decoders. Therefore, FDFJM_3En_3De(512) is used as the fire intensity judgment model. Figure 3 Schematic diagram of the FDFJM_3En_3De(512) model structure.

[0078] Step 1.1.1. The fire intensity judgment model is to segment the fire area separately and classify each pixel of the image as fire or non-fire to complete the image segmentation task. The fire intensity judgment model includes an input layer, a hidden layer, and an output layer. The image size of the dataset is 254×254×3. Before inputting the model, the bilinear interpolation method is used to convert the image to a uniform size of H×W×C. Two values ​​are set for the model: H=W=256 or 512. The size of the input layer is H×W×C. Each image is designed to have a uniform input size and three RGB channels, such as Figure 3 As shown, the hidden layer contains O=3 block encoders and O=3 block decoders. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains 7 CNNs and 6 residual blocks. The residual block is composed of 2 CNNs. The decoder contains 4 CNNs, 6 residual blocks and 3 deconvolution neural networks. The residual block is also composed of 2 CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolution layer with a Sigmoid activation function. The fire judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0079] Step 1.1.2: After the fire intensity judgment model is built, the fire intensity judgment model is trained. The loss function used is binary cross entropy:

[0080]

[0081] For fire image segmentation datasets, fire areas need to be manually annotated to generate a true label mask. The fire intensity assessment model then predicts each pixel in the image, generating a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask, achieving segmentation of the fire area and fire intensity assessment. The training and validation data in the fire assessment model dataset consist of t% and (1-t)%, respectively, where t refers to the proportion of training data in the entire dataset; specifically, t% = 85% and (1-t)% = 15%. Before training the model, the data is shuffled. The maximum number of iterations is E = 50, but early stopping is used when performance does not substantially change within P = 5 iterations. The training batch size is B = 16.

[0082] Figure 5 The performance comparison of training loss, recall rate and precision rate under different model structures (model input layer size is 512) is shown in the figure. Figure 5 It can be seen that the training loss has gradually decreased and tended to be stable, and the recall rate and precision rate have gradually increased and tended to be stable, which means that the fire judgment model FDFJM_3En_3De(512) has been trained.

[0083] Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and its parameters to visible satellites.

[0084] The fire detection model construction and training in step 1 specifically include the following steps:

[0085] Table 1 shows the performance evaluation results of fire detection tasks under different model structures.

[0086] Table 1

[0087]

[0088]

[0089] As can be seen from Table 1, the fire detection model has the highest accuracy and the best fire detection task effect when the input layer size is 512 and there are 2 encoders and 2 decoders. Therefore, FDFJM_2En_2De(512) is used as the fire detection model. Figure 2 Schematic diagram of the FDFJM_2En_2De(512) model structure.

[0090] Step 1.2.1, the fire detection model is to classify the entire image as having fire or not, completing the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus converted into an image segmentation task, such as Figure 2As shown, the fire detection model includes an input layer, a hidden layer and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer contains an L=2 block encoder and an L=2 block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains 7 CNNs and 4 residual blocks. The residual block consists of 2 CNNs. The decoder contains 5 CNNs, 4 residual blocks and 3 deconvolution neural networks. The residual block also consists of 2 CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolution layer with a Sigmoid activation function. The fire detection model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side structure of the model is spliced ​​with the corresponding right side structure.

[0091] Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy:

[0092]

[0093] The fire detection model and the fire intensity assessment model use the same dataset. Similarly, fire areas must be manually annotated to generate a true label mask. The fire detection model then predicts each pixel in the image and generates a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask, achieving segmentation of the fire area and fire detection. The training data and validation data in the fire detection model dataset are t% and (1-t)%, respectively, where t refers to the proportion of training data in the entire dataset. Specifically, t% = 85% and (1-t)% = 15%. Before importing the model for training, the data is shuffled. The maximum number of iterations is E = 50, but early stopping is used when performance does not substantially change within P = 5 iterations. The training batch size is B = 16.

[0094] Figure 5 The performance comparison of training loss, recall rate and precision rate under different model structures (model input layer size is 512) is shown in the figure. Figure 5 It can be seen that the training loss has gradually decreased and stabilized, and the recall rate and precision rate have gradually increased and stabilized, indicating that the fire detection model FDFJM_2En_2De(512) has been trained.

[0095] Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

[0096] The semantic extraction and restoration model construction in step 1 specifically includes the following steps:

[0097] The semantic extraction and recovery model implemented is based on CNN and is named Fire Disaster oriented Semantic Extraction Semantic Recovery Model (FDSESRM). Figure 4 Schematic diagram of the FDSESRM model structure.

[0098] like Figure 4 As shown, the semantic extraction and restoration model includes an input layer, a hidden layer and an output layer. The size of the input layer is H1×W1×C, specifically H1=W1=256. The input layer preprocesses the image, and each image is designed to have a unified input size and three RGB channels. The hidden layer contains a K=2 block encoder and a K=2 block decoder. The encoder is used to extract semantic information, and the decoder is used to restore semantic information. The encoder contains 7 CNNs, 4 residual blocks, 1 deconvolution network and 1 embedding layer. The residual block is composed of 2 CNNs. The decoder contains 2 CNNs, 4 residual blocks and 3 deconvolution neural networks. The residual block is also composed of 2 CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional deconvolution layer, which finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, with the encoder on the left side of the model and the decoder on the right side of the model.

[0099] The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vectors in the shared discrete codebook have the same dimension D = 64 as the feature vectors encoded by the encoder. The encoder maps the input continuous encoded features into the basis vectors in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly recovers the original basis vector based on the received discrete index.

[0100] The training of the semantic extraction and restoration model in step 1 specifically includes the following steps:

[0101] Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composition, D is each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), which is mapped to the basis vector z by nearest neighbor search b (s), calculated as follows:

[0102]

[0103] Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s);

[0104] Step 1.3.2, the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradient is approximated using a straight-through estimator and the gradient is fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e (s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and in the back propagation process, the gradient Passed to the encoder, the output of the encoder and the input of the decoder have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ;

[0105] The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as:

[0106]

[0107] Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is:

[0108]

[0109] In the loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j ||2 In order to make the basis vector and its corresponding encoder output as close as possible, ||z e (s)-sg[e j ]||2 is used to constrain the output of the encoder, not letting it go too far away from the basis vector, γ controls the relative learning speed of the encoder, and the decoder only passes To optimize, the encoder passes and ||ze (s)-sg[e j ]||2 is optimized, and the basis vectors are optimized by ||sg[z e (s)]-e j ||2 for optimization;

[0110] Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss Expressed as:

[0111]

[0112] Here, m represents the number of pixels in the image.

[0113] Figure 6 The performance comparison of the fire semantic extraction and semantic restoration model training loss and structural similarity index under different learning rates and optimizers is shown in the figure. Figure 6 It can be seen that the training loss has gradually decreased and stabilized, and the structural similarity index has gradually increased and stabilized, indicating that the semantic extraction and semantic recovery model has been trained. The structural similarity index is used to evaluate image quality. The higher the value, the better the restored image. Therefore, it can be seen that the model works best when lr = 0.0003 and optimizer = Adam. Therefore, the model trained under the condition of lr = 0.0003 and optimizer = Adam is used in the simulation.

[0114] Step 1.3.3: After the semantic extraction and restoration model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and restoration model to the ground terminal by region, and the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and restoration model to the visible satellites by region.

[0115] Figure 7 The performance evaluation results of the fire semantic extraction and semantic restoration model of the present invention and the structural similarity index of the JPEG method under different modulation modes and signal-to-noise ratios are shown in FIG. Figure 7 It can be seen that under the same modulation method, the structural similarity index of FDSESRM is always better than that of JPEG.

[0116] Figure 8 The transmission data volume evaluation result diagram of JPEG, the fire semantic extraction and semantic restoration model of the present invention and the fire semantic extraction and semantic restoration model of the present invention with fusion similarity comparison is shown. Figure 8It can be seen that the amount of data of FDSESRM is significantly less than that of JPEG. Taking the transmission time of 30s as an example, the amount of data of FDSESRM is reduced by 74.55% compared with the JPEG method. The amount of data of FDSESRM with fusion similarity comparison is reduced by 81.05% compared with the JPEG method.

[0117] The inter-satellite update and model retraining in step 1 specifically include the following steps:

[0118] Step 1.4.1: During the monitoring mission, if a satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update.

[0119] Step 1.4.2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain. After retraining is completed, since the models in different areas are the same but the parameters are different, only some of the semantic extraction and recovery model parameters are distributed.

[0120] Figure 9 This is a performance comparison diagram of the structural similarity index of the fire extraction and semantic recovery model of the present invention when the parameters are updated and when the parameters are not updated. In the figure, the first-level update represents the inter-satellite update, and the second-level update represents the model retraining update. Figure 9 It can be seen that the structural similarity index of both the first-level update and the second-level update has been improved, and the image restoration quality has improved.

[0121] Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission period for the original image has been reached. If so, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction.

[0122] Step 3: Fire image semantic extraction and transmission:

[0123] Step 3.1: Use the semantic extraction model in the trained semantic extraction and restoration model to semantically encode the monitored fire image. The semantic extraction model has two encoders and performs two encodings to obtain the encoded feature vector (B1, M, M, D).

[0124] Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B1, M, M, D) is quantized for the first time, and the codebook index (B1, N, N) is obtained through the nearest neighbor search.

[0125] Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The array size is (B1, N×N), and the index N that needs to be transmitted is i ×N i ,i=1,2,...,B1 is compared with the index in the array for similarity. If they are similar, only the array index b is transmitted. i ,i=1,2,...,B1, if they are not similar, transmit the complete codebook index N i ×N i ,i=1,2,...,B1;

[0126] Step 3.4: At the same time, the encoder will send the time, longitude and latitude of the current sensor to collect the image, and the device number.

[0127] Step 4: Fire image semantic restoration:

[0128] The semantic extraction model and the semantic recovery model are trained together. The fire detection model and the fire intensity assessment model are trained in the same way. The data received by the satellite is divided into three categories: the first category is the codebook index of the complete image, the second category is the array index, and the third category is the original image transmitted directly through the channel. The specific steps include the following according to the data type:

[0129] Step 4.1: If the data received from the visible satellite is a complete image, the codebook index N i ×N i ,i=1,2,...,B1, first perform inverse quantization, the receiving end decoder converts the received codebook index N according to the shared codebook ε i ×N i , i=1,2,...,B1 recovers the codebook basis vector (N i , N i , D), i=1, 2, ..., B1, then, the basis vector (N i , N i , D), i = 1, 2, ..., B1 input semantic recovery model in semantic extraction and recovery model, semantic recovery model has two decoders, after the first decoding, the decoded feature vector is quantized for the second time, the quantized codebook index size is the same as the first time, that is, N i ×N i ,i=1,2,...,B1, then dequantize to get the eigenvector (M i ,M i ,D),i=1,2,...,B1, the feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H1×W1×C;

[0130] Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i ,i=1,2,...,B1, the receiving end decoder creates an array (B1, H1×W1×C) that can store the previous set of received pictures during the monitoring task execution, directly according to the array index b i ,i=1,2,...,B1, take out the restored image from the image array (B1,H1×W1×C) without going through the semantic restoration model in the semantic extraction and restoration model;

[0131] Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining is required. The visible satellite will notify the fusion center to retrain the semantic extraction and restoration model.

[0132] Step 4.4: After the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire intensity judgment model. Fires are divided into four levels: large fire, medium fire, small fire, and no fire. Then, an auxiliary decision is output based on the fire intensity results. Finally, the time, longitude and latitude of the fire, device number, fire size, and auxiliary decision are output in frame format.

[0133] In step 3, the similarity comparison is performed as follows: during the execution of the monitoring task, the encoder creates an array to store the index of the sent picture, the array size is (B1, N×N), and the decoder creates an array to store the received picture, the array size is (B1, H1×W1×C); the index N currently to be transmitted is i ×N i ,i=1,2,...,B1 is compared with the index in the array on the encoder side for similarity. If they are similar, only the array index b is transmitted. i , i=1,2,...,B1,the decoder will receive the array index b i ,i=1,2,...,B1 takes the corresponding picture from the decoder array (B1, H1×W1×C); if they are not similar, the complete codebook index N needs to be transmitted i ×N i ,i=1,2,...,B1.

[0134] In summary, the present invention is aimed at a satellite Internet of Things semantic communication method for fire monitoring, which performs a series of processing on fire images, including fire detection, semantic extraction and transmission, and fire intensity judgment. It can not only accurately detect fires, but also reduce transmission load and improve transmission efficiency, which helps to understand the size of the fire in a timely manner.

[0135] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A satellite Internet of Things semantic communication method for fire monitoring, characterized by: The satellite IoT semantic communication method includes four parts: model construction and training, fire image preprocessing, fire image semantic extraction and transmission, and fire image semantic restoration. The specific steps are as follows: Step 1: Model construction and training: Build and train a fire detection model, a semantic extraction and recovery model, and a fire intensity assessment model through deep learning. Distribute the trained models and model parameters, and perform inter-satellite updates and model retraining. Step 2: Fire image preprocessing: After receiving the fire monitoring image collected by the IoT sensor, the ground terminal determines whether the transmission period for the original image has been reached. If so, the original image needs to be directly transmitted through the channel. At the same time, the fire detection model is used to perform feature analysis on the image to determine whether there is a fire in the image. If there is a fire, the fire image is input into the semantic extraction model in the semantic extraction and recovery model for semantic extraction. Step 3: Semantic extraction and transmission of fire images: Use the semantic extraction model in the semantic extraction and recovery model to semantically encode the monitored fire images to obtain the encoded feature vectors. Use the nearest neighbor search on the encoded feature vectors to map the feature vectors into basis vectors in the codebook, thereby obtaining the corresponding basis vector indexes. The channel only needs to transmit the obtained basis vector indexes. At the same time, the previous set of basis vector indexes are stored, and the index to be transmitted is compared with the previous set of indexes for similarity. If a similar one is found, only the image index needs to be transmitted. If it is not similar, the complete codebook index needs to be transmitted. At the same time, the channel will transmit the time, longitude and latitude and device number of the current IoT sensor to collect the image. Step 4: Semantic restoration of fire images: After receiving the complete codebook index, the visible satellite recovers the feature vector based on the shared codebook and feeds the feature vector into the semantic restoration network to restore the complete image. If it receives an image index, it directly recovers the image from the most recently transmitted image without going through the semantic restoration model. After the complete image is restored, it is input into the fire intensity judgment model to determine the size of the fire and output the final decision based on the fire intensity result. The visible satellite also receives the original image transmitted directly through the channel and compares the structural similarity index with the image restored by the model. If the image restored by the model is not good, retraining is required. The visible satellite notifies the fusion center to retrain the semantic extraction and semantic restoration model. At the same time, the visible satellite also receives the time, longitude and latitude of the current image acquisition and the device number.

2. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: The fire intensity judgment model construction and training in step 1 specifically include the following steps: Step 1.1.

1. The fire intensity judgment model is to segment the fire area separately and classify each pixel of the image as having fire or not to complete the image segmentation task. The fire intensity judgment model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C. Each image is designed to have a unified input size and three RGB channels. The hidden layer contains an O-block encoder and an O-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder contains multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder contains multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire intensity judgment model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side of the model is spliced ​​with the corresponding right side structure. Step 1.1.2: After the fire intensity judgment model is built, the fire intensity judgment model is trained. The loss function used is the binary cross entropy: For the fire image segmentation dataset, the fire area needs to be manually annotated to generate a true label mask. The fire intensity judgment model is then asked to predict each pixel of the image and generate a predicted label mask. Through model training, the predicted label mask is made to fit the true label mask to achieve segmentation of the fire area and judgment of the fire intensity. The training data and validation data in the fire judgment model dataset account for t% and (1-t)% respectively, where t refers to the proportion of training data in the entire dataset. Before importing the model for training, the data is shuffled. The maximum number of iterations is E, but when the performance has not changed substantially within P iterations, it will stop early. The training batch size is B; Step 1.1.3: After the fire intensity judgment model training is completed, the fusion center distributes the fire intensity judgment model and its parameters to visible satellites.

3. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: The fire detection model construction and training in step 1 specifically include the following steps: Step 1.2.1, the fire detection model is to classify the entire image as having fire or not, completing the image classification task. As long as there are pixels in the image that are judged to have fire, the entire image is judged to have fire, and the image classification task is thus converted into an image segmentation task. The fire detection model includes an input layer, a hidden layer, and an output layer. The size of the input layer is H×W×C, and each image is designed to have a unified input size and three RGB channels. The hidden layer includes an L-block encoder and an L-block decoder. The encoder is used to reduce the size of the feature map, increase the number of channels, and extract high-level semantic features. The decoder is used to increase the size of the feature map, reduce the number of channels, and restore spatial information. The encoder includes multiple CNNs and multiple residual blocks. The residual block is composed of multiple CNNs. The decoder includes multiple CNNs, multiple residual blocks, and multiple deconvolutional neural networks. The residual block is also composed of multiple CNNs. The deconvolutional neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional convolutional layer with a Sigmoid activation function. The fire detection model has a symmetrical structure. The encoder is on the left side of the model and the decoder is on the right side of the model. The left side of the model is spliced ​​with the corresponding right side structure. Step 1.2.2: After the fire detection model is built, the fire detection model is trained. The training method of the fire detection model is the same as that of the fire intensity judgment model. The loss function used is binary cross entropy: The fire detection model and the fire intensity assessment model use the same dataset. They also require manual annotation of the fire area to generate a true label mask. The fire detection model then predicts each pixel in the image and generates a predicted label mask. Through model training, the predicted label mask is fitted to the true label mask to segment the fire area and detect the fire. The training data and validation data in the fire detection model dataset account for t% and (1-t)% respectively, where t refers to the percentage of training data in the entire dataset. Before importing the model for training, the data is shuffled. The maximum number of iterations is E, but if the performance does not change substantially within P iterations, early stopping is performed. The training batch size is B. Step 1.2.3: After the fire detection model training is completed, the fusion center distributes the fire detection model and the parameters of the fire detection model to the ground terminal.

4. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: The semantic extraction and restoration model in step 1 includes an input layer, a hidden layer, and an output layer. The size of the input layer is H1×W1×C. The input layer preprocesses the image. Each image is designed to have a unified input size and three RGB channels. The hidden layer includes a K-block encoder and a K-block decoder. The encoder is used to extract semantic information, and the decoder is used to restore semantic information. The encoder includes multiple CNNs, multiple residual blocks, a deconvolution network, and an embedding layer. The residual block is composed of multiple CNNs. The decoder includes multiple CNNs, multiple residual blocks, and multiple deconvolution neural networks. The residual block is also composed of multiple CNNs. The deconvolution neural network increases the size of the feature map and restores the spatial resolution of the image. The output layer is a two-dimensional deconvolution layer, and finally outputs the restored image. The semantic extraction and restoration model has a symmetrical structure, with the encoder on the left side of the model and the decoder on the right side of the model. The embedding layer in the semantic extraction and restoration model is a discrete codebook shared by the encoder and decoder. The basis vectors in the shared discrete codebook have the same dimension D as the feature vectors encoded by the encoder. The encoder maps the input continuous encoded features into the basis vectors in the shared discrete codebook. Therefore, the encoder only needs to transmit the discrete index of the basis vector, and the decoder quickly recovers the original basis vector based on the received discrete index.

5. A satellite Internet of Things semantic communication method for fire monitoring according to claim 1 or 4, characterized in that: The training of the semantic extraction and restoration model in step 1 specifically includes the following steps: Step 1.3.1: Define the shared discrete codebook as The shared discrete codebook consists of J basis vectors Composition, D is each basis vector e j The model accepts the input s and passes it through the encoder to encode the feature vector z e (s), which is mapped to the basis vector z by nearest neighbor search b (s), calculated as follows: Consider this forward computation as a deep neural network layer with a specific nonlinear function that transforms the encoded feature vector z e (s) is mapped to the basis vector z b (s); Step 1.3.2, the basis vector z b (s) is input to the decoder for decoding, and the basis vector in the discrete codebook ε The parameters of the encoder and decoder are trained together. In the back-propagation, the gradient is approximated using a straight-through estimator and the gradient is fed from the decoder into the basis vector z. b (s) is copied to the encoder output z e (s), in the forward propagation, the nearest basis vector z b (s) is passed to the decoder, and in the back propagation process, the gradient Passed to the encoder, the output of the encoder and the input of the decoder have the same dimension D, and the gradient contains useful information for training the encoder to update its parameters to minimize the loss function L c ; The loss function consists of reconstruction loss and quantization loss. The reconstruction loss is used to measure the degree of distortion when the semantic extraction and restoration model reconstructs the input data. The quantization loss is used to measure the difference between the encoder output and its nearest neighbor codebook vector. The loss function is expressed as: Among them, s represents the input of the network, represents the output of the network, θ represents the trainable parameters of the neural network in the semantic communication system, γ is a hyperparameter, Represents the reconstruction loss, the symbol "sg" represents "stop gradient", and the operation method is: In the loss function is the reconstruction loss for training the encoder and decoder parameters, ||sg[z e (s)]-e j ||2 In order to make the basis vector and its corresponding encoder output as close as possible, ||z e (s)-sg[e j ]||2 is used to constrain the output of the encoder, not letting it go too far away from the basis vector, γ controls the relative learning speed of the encoder, and the decoder only passes To optimize, the encoder passes and ||z e (s)-sg[e j ]||2 is optimized, and the basis vectors are optimized by ||sg[z e (s)]-e j ||2 for optimization; Reconstruction losses It is designed to calculate the MSE between the reconstructed image and the original image, so the reconstruction loss Expressed as: Where m represents the number of pixels in the image; Step 1.3.3: After the semantic extraction and restoration model training is completed, the fusion center distributes the semantic extraction model and semantic extraction model parameters in the semantic extraction and restoration model to the ground terminal by region; the fusion center distributes the semantic recovery model and semantic recovery model parameters in the semantic extraction and restoration model to the visible satellites by region.

6. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: The inter-satellite update and model retraining in step 1 specifically include the following steps: Step 1.4.1: During the monitoring mission, if a satellite leaves the area and cannot perform the current mission, the model parameters of the current satellite need to be sent to the next visible satellite to ensure that the monitoring performance can be maintained. This process is called inter-satellite update. Step 1.4.

2. At the same time, if the retraining conditions are met, the semantic extraction and recovery model is retrained, and the visible satellites notify whether to retrain. After retraining is completed, since the models in different areas are the same but the parameters are different, only some of the semantic extraction and recovery model parameters are distributed.

7. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: In step 3, the semantic extraction and transmission process of the fire image is specifically as follows: Step 3.1: Use the semantic extraction model in the trained semantic extraction and restoration model to semantically encode the monitored fire image. The semantic extraction model has two encoders and performs two encodings to obtain the encoded feature vector (B1, M, M, D). Step 3.2: The semantic extraction and restoration model is quantized twice. The encoder performs the first quantization. First, the encoded feature vector (B1, M, M, D) is quantized for the first time, and the codebook index (B1, N, N) is obtained through the nearest neighbor search. Step 3.3: During the monitoring task execution, the encoder creates an array to store the index of the previous set of pictures. The array size is (B1, N×N), and the index N that needs to be transmitted is i ×N i , i=1,2,...,B1 is compared with the index in the array for similarity. If they are similar, only the array index b is transmitted. i , i=1,2,...,B1, if they are not similar, transmit the complete codebook index N i ×N i , i=1,2,...,B1; Step 3.4: At the same time, the encoder will send the time, longitude and latitude of the current sensor to collect the image, and the device number.

8. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: In step 4, the semantic restoration of the fire image specifically includes the following steps: Step 4.1: If the data received from the visible satellite is a complete image, the codebook index N i ×N i , i=1,2,...,B1,firstly, dequantization is performed, and the decoder at the receiving end converts the received codebook index N into the shared codebook ε. i ×N i , i=1,2,...,B1 recovers the codebook basis vector (N i , N i , D), i=1, 2, ..., B1, then, the basis vector (N i , N i , D), i = 1, 2, ..., B1 input semantic recovery model in semantic extraction and recovery model, semantic recovery model has two decoders, after the first decoding, the decoded feature vector is quantized for the second time, the quantized codebook index size is the same as the first time, that is, N i ×N i , i=1,2,...,B1,then perform inverse quantization to obtain the eigenvector (M i , M i , D), i = 1, 2, ..., B1, the feature vector obtained after dequantization is input into the second decoder, and the output of the second decoder is the complete restored image H1×W1×C; Step 4.2: If the data received from the visible satellite is an array index, the received array index is b i , i=1,2,...,B1,The receiving end decoder creates an array (B1,H1×W1×C) that can store the previous set of received pictures during the monitoring task execution. i , i=1, 2, ..., B1, take out the restored image from the image array (B1, H1×W1×C) without going through the semantic restoration model in the semantic extraction and restoration model; Step 4.3: If the data received by the visible satellite is the original image directly transmitted through the channel, the structural similarity index of the received original image is compared with the image restored by the semantic recovery model in the semantic extraction and restoration model. If the image restored by the model is not good, retraining is required. The visible satellite will notify the fusion center to retrain the semantic extraction and restoration model. Step 4.4: After the decoder at the receiving end obtains the restored image, it first determines the size of the fire through the fire intensity judgment model. Fires are divided into four levels: large fire, medium fire, small fire, and no fire. Then, an auxiliary decision is output based on the fire intensity results. Finally, the time, longitude and latitude of the fire, device number, fire size, and auxiliary decision are output in frame format.

9. The satellite Internet of Things semantic communication method for fire monitoring according to claim 1, characterized in that: The similarity comparison in step 3 is specifically as follows: during the execution of the monitoring task, the encoder creates an array to store the index of the sent picture, the array size is (B1, N×N), and the decoder creates an array to store the received picture, the array size is (B1, H1×W1×C); the index N currently to be transmitted is i ×N i , i = 1, 2, ..., B1 is compared with the index in the array on the encoder side for similarity. If they are similar, only the array index b is transmitted. i , i=1,2,...,B1,the decoder will receive the array index b i , i = 1, 2, ..., B1 takes the corresponding picture from the decoder array (B1, H1×W1×C); if they are not similar, the complete codebook index N needs to be transmitted i ×N i , i=1,2,...,B1.

Images