A mine video semantic communication transmission method and system based on deep learning

Abstract

The application provides a mine video semantic communication transmission method and system based on deep learning, which can realize the recovery of observable data and the execution of pragmatic tasks. The application constructs a semantic communication system model under a mine mining working face, including a semantic encoder and a semantic decoder realized based on a deep neural network, the semantic encoder is located at a transmitting end and is used for extracting data containing observable data and semantic information, and the semantic decoder is located at a receiving end and is used for reconstructing data related to experience data and pragmatic tasks. The application trains the semantic encoder and the semantic decoder based on a DL-SC algorithm, in the design of a loss function, considers two aspects of image recovery and pragmatic task execution, selects appropriate distortion measurement functions for the two aspects respectively, then obtains a weighted sum to constitute a semantic distortion measurement function as the loss function. The application can realize more efficient information interaction with lower data transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to video transmission communication, and in particular to a mine video semantic communication transmission method and system based on deep learning under a mine mining working face. BACKGROUND

[0002] With the continuous development of communication technology, the sending and receiving ends of information are becoming more and more intelligent, and the demand for intelligent application of wireless communication is also rapidly increasing. The amount of data generated by the system is also increasing, and the communication traffic will also increase, making wireless communication face problems such as scarcity of spectrum resources. Unlike traditional communication, semantic communication focuses on the transmission of semantic information rather than the accurate transmission of raw data, so as to eliminate the transmission of redundant data, achieve data compression, and thus save bandwidth to achieve more efficient information interaction with lower data transmission. Under the mining, communication resources are more valuable, and faster and more efficient data transmission is more needed, so it is necessary to study the semantic communication method of perception information in mining.

[0003] At present, the research on semantic communication mainly includes the extraction and quantization of semantic information and the utilization of pragmatic level semantic information. Based on the expression of semantic information theory, there are many researches on semantic communication systems based on deep learning. The frameworks of these communication systems are similar. Specifically, semantic information is regarded as the hidden features of observable information, and the relationship between the two is not directly given. However, the experience data tuples of observable information and related semantic information are provided by a large enough knowledge base. Since the joint distribution between semantic information and observable information may be too complex to be accurately estimated by traditional methods. SUMMARY

[0004] The purpose of the present application is to provide a mine video semantic communication transmission method and system based on deep learning, which is aimed at the application scene of the mine mining working face to realize data compression, reduce data transmission, improve transmission efficiency and save communication resources.

[0005] Technical scheme: In order to achieve the above-mentioned purpose of the present application, the present application adopts the following technical scheme:

[0006] In the first aspect, the present application provides a mine video semantic communication transmission method based on deep learning, which comprises the following steps:

[0007] A semantic communication system model under the mine mining working face is established, including a semantic encoder and a semantic decoder. The semantic encoder is located at the transmitting end and is realized based on a deep neural network, and is used for extracting data containing observable data and semantic information. The semantic decoder is located at the receiving end and is realized based on a deep neural network, and is used for reconstructing data related to experience data and pragmatic tasks.

[0008] The semantic distortion function is used as a loss function during training, and the semantic encoder and the semantic decoder are trained based on the DL-SC algorithm; including: the transmitting end extracts a batch of training samples, encodes the samples through the semantic encoder, and sends them to the receiving end through the channel; the receiving end decodes the received data through the semantic decoder and performs a pragmatic task; the receiving end calculates the gradient of the loss function on the network parameters of the semantic decoder, and updates the network parameters of the semantic decoder; the receiving end calculates the gradient of the loss function with respect to the channel output, and feeds it back to the transmitting end together with the channel output; the transmitting end calculates the gradient of the channel output on the network parameters of the semantic encoder, and updates the network parameters of the semantic encoder in combination with the gradient fed back by the receiving end; the semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function.

[0009] Further, the semantic encoder and the semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as consisting of J basis vectors , Q is the dimension of each basis vector e j , represents a real number; the input s generates an encoded feature vector z e (s) through the semantic encoder, z e (s) is mapped to a basis vector z b (s) through nearest neighbor search, and z b (s) is input to the semantic decoder.

[0010] Further, the basis vectors in the codebook ε are trained together with the parameters of the semantic encoder and the parameters of the semantic decoder, and the loss function is as follows:

[0011]

[0012] wherein, and z respectively represent the output and the true label, and α is a hyperparameter; the symbol ng[z e (s)] represents that the gradient of z e (s) is zero, and ng[e j ] represents that the gradient of e j is zero; ||·||2 is the 2-norm.

[0013] Further, the semantic distortion function is represented as:

[0014]

[0015] wherein, θ1 and θ2 respectively represent the network parameters of the semantic encoder and the network parameters of the semantic decoder, respectively represent empirical data, observable reconstructed data, execution result and reconstructed pragmatic task output, denotes a distortion metric function of observable data, denotes a distortion metric function of pragmatic output, β is a hyper-parameter for dynamic range alignment, and λ is a hyper-parameter for balancing observable data and pragmatic information.

[0016] Further, given a training batch When the compression rate CR is given, for considering both image classification and semantic segmentation pragmatic tasks, the joint optimization problem of network parameters is represented as:

[0017]

[0018] where, is the expectation with respect to the empirical distribution of training batch , S MSE and S CE respectively represent mean square error and cross-entropy with respect to the empirical distribution of training batch , S denotes a semantic encoder, denotes a semantic decoder, denotes a pragmatic task function, cr is a fixed compression rate, and N is the power of channel noise.

[0019] Further, the DL-SC algorithm-based training of the semantic encoder and the semantic decoder comprises the following steps:

[0020] Step 1, set a training round counter t = 1;

[0021] Step 2, judge whether the maximum training round is reached, if not, go to step 3, otherwise end;

[0022] Step 3, the sender extracts a batch of training samples from the data set

[0023] Step 4, encode the samples in and send them out, denotes a semantic encoder at t rounds, and X denotes the semantic encoder output;

[0024] Step 5, the receiver decodes the received data denotes a semantic decoder at t rounds, and Y denotes the channel output;

[0025] Step 6, the receiver executes a pragmatic task denotes a pragmatic task function;

[0026] Step 7, the receiver calculates the gradient according to the following formula and update network parameters;

[0027]

[0028] Step 8, the receiving end calculates the gradient

[0029] Step 9, the receiving end feeds back X and Y to the sending end;

[0030] Step 10, the sending end calculates the gradient of the channel output on the semantic encoder network parameters according to the following formula and update network parameters;

[0031]

[0032] Step 11, the training round counter t is incremented by 1, and step 2 is entered.

[0033] Further, the semantic encoder and the semantic decoder are locally pre-trained at the receiving end by minimizing the observable data reconstruction loss function.

[0034] In a second aspect, the application provides a mine video semantic communication transmission system based on deep learning, comprising a semantic encoder and a semantic decoder, the semantic encoder is located at the sending end and is realized based on a deep neural network, and is used for extracting data containing observable data and semantic information; the semantic decoder is located at the receiving end and is realized based on a deep neural network, and is used for reconstructing data related to experience data and pragmatic tasks;

[0035] The semantic encoder and the semantic decoder are optimized based on the DL-SC algorithm training with the semantic distortion function as the loss function during training; including: the sending end extracts a batch of training samples, encodes the samples through the semantic encoder, and sends them to the receiving end through the channel; the receiving end decodes the received data through the semantic decoder and performs a pragmatic task; the receiving end calculates the gradient of the loss function on the network parameters of the semantic decoder and updates the network parameters of the semantic decoder; the receiving end calculates the gradient of the loss function with respect to the channel output and feeds it back to the sending end; the sending end calculates the gradient of the channel output on the network parameters of the semantic encoder, combines the gradient fed back by the receiving end, and updates the network parameters of the semantic encoder; the semantic distortion function is the weighted sum of the semantic information loss function and the observable data loss function.

[0036] ​Beneficial effects: For the fusion of communication and perception in the mining working face, the mine video semantic communication transmission method and system based on deep learning proposed by the application can realize the recovery of observable data and the execution of pragmatic tasks. The application constructs a semantic communication model based on a neural network, constructs a semantic encoder and decoder into two deep neural networks, and adopts a joint source and channel coding method to realize semantic coding. During training, the receiving end is dominant and feeds back data to the sending end, realizing joint training of the encoder and decoder while not leaking relevant information of the pragmatic task to the sending end. In the design of the loss function, both video recovery and pragmatic task execution are considered, appropriate distortion measurement functions are selected for the two, and then their weighted sum is calculated to form a semantic distortion measurement function as the loss function. The training target is to minimize the loss function, and the network training algorithm is formed to realize more efficient information interaction with lower data transmission volume. In summary, in the scene of the mining working face, the method of the application is efficient in information interaction, can realize data compression, reduce data transmission volume, improve transmission efficiency and save communication resources. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The semantic communication system model diagram in the embodiment of the application.

[0038] Figure 2 The DL-SC algorithm flowchart in the embodiment of the application.

[0039] Figure 3 The semantic encoder and decoder network structure schematic diagram in the embodiment of the application. DETAILED DESCRIPTION

[0040] The technical solutions of the application will be further described below in combination with the drawings and specific embodiments.

[0041] The embodiment of the application discloses a mine video semantic communication transmission method based on deep learning, first, a semantic communication system model under a mine mining working face is established, wherein the semantic communication system model adopts a semantic communication system model based on a discrete codebook, and comprises a semantic encoder and a semantic decoder; the semantic encoder is located at a transmitting end, is realized based on a deep neural network, and is used for extracting data containing observable data and semantic information; the semantic decoder is located at a receiving end, is realized based on a deep neural network, and is used for reconstructing data related to experience data and pragmatic tasks; then, a loss function of loss minimization of the semantic information and the observable data is designed, and a DL-SC algorithm is used for training a deep learning model. Specifically, the transmitting end extracts a batch of training samples, encodes the samples through the semantic encoder, and then sends the samples to the receiving end through a channel; the receiving end decodes the received data through the semantic decoder, and executes a pragmatic task; the receiving end calculates the gradient of the loss function on the network parameters of the semantic decoder, and updates the network parameters of the semantic decoder; the receiving end calculates the gradient of the loss function relative to the channel output, and feeds back the gradient and the channel output to the transmitting end; the transmitting end calculates the gradient of the channel output on the network parameters of the semantic encoder, and updates the network parameters of the semantic encoder in combination with the gradient fed back by the receiving end.

[0042] The semantic communication system model, the loss function and the DL-SC algorithm related to the embodiment of the application are described in detail below.

[0043] I. Establishing a semantic communication system model under a mine mining working face

[0044] Dataset is a set of experience data D from background knowledge and the corresponding pragmatic task execution results Z. The experience data D is a picture obtained by processing a video by frames, and the experience data D is first encoded by the semantic encoder f (·) to obtain X input channel, and the basic element is denoted as Γ X . In order to be consistent with the image source in RGB format, X is also set as a three-dimensional vector sequence with 8-bit symbols. The compression rate (CR) is defined as:

[0045]

[0046] Wherein, Γ D is the basic element of D, |·| is an absolute value symbol, and CR is equivalent to the ratio of image resolution to channel input sequence length.

[0047] The encoded signal X will be transmitted through an additive white Gaussian noise channel, and the channel output is denoted as Y=X+N. The power N of the channel noise is fixed but unknown. After receiving Y, the receiving end can reconstruct the observable data as according to the decoder g (·), and the pragmatic task function Further reconstruction of pragmatic task output

[0048] The encoder f(·) and decoder g(·) in the system are both implemented through deep neural network DNN, and the pragmatic task function Set to a given value. It can be regarded as a mathematical abstraction of the pragmatic task, and how to obtain it is not the research focus of this invention. Semantic communication based on joint source-channel coding can also be regarded as task-oriented semantic communication, and the encoder takes into account both semantic information and channel influence. Therefore, the goal of the encoder is to extract data containing the most observable data and semantic information, while the goal of the decoder is to reconstruct data and observable data related to the pragmatic task. Since the pragmatic task information is only known to the receiver, the receiver dominates the training process of the encoder and decoder. By comparing the reconstructed data with the original data, the receiver can guide the transmitter on how to improve the encoder.

[0049] In the perception information transmission model, ground sensor nodes can be considered the transmitter of a semantic communication system, including a semantic encoder, and the receiver, including a semantic decoder, as the receiver. These decoders are required to perform specific pragmatic tasks. Given the low transmission power of sensors in practice, only the transmission rate between the sensor node and the receiver is optimized. The expression for the transmission rate between the sensor node and the receiver is:

[0050] R s,r =B1log2(1+γ s,r )

[0051] Where B1 represents the channel bandwidth of additive white Gaussian noise, γ s,r It represents the signal-to-noise ratio of the link between the sensor node and the receiving device. From this, we can see that the larger the signal-to-noise ratio of the channel, the higher the transmission rate. Therefore, while maximizing the transmission rate, we are also optimizing the signal-to-noise ratio of the link.

[0052] In semantic communication systems, the signal-to-noise ratio of the channel also determines the effect of image transmission. Let the signal-to-noise ratio be γ SNR , the channel input signal and noise signal are x and n respectively, and the signal length is L, then:

[0053]

[0054] Then, under the condition that the signal-to-noise ratio is known, the formula of the noise signal can be derived as follows:

[0055]

[0056] Finally, the noise signal is amplified according to the Gaussian distribution to obtain the final noise signal:

[0057]

[0058] Here, random(L) represents a randomly generated integer in the open interval [0, L-1). The resulting noise signal is then superimposed on the channel input signal to obtain the channel output signal. Clearly, the higher the channel signal-to-noise ratio and the lower the noise, the closer the channel output signal is to the original input signal, and the better the performance of the restoration and pragmatic tasks.

[0059] 2. Building a Semantic Communication System Based on Discrete Codebook

[0060] The discrete codebook is expressed as It consists of J basis vectors Composition, Q is each basis vector e j The dimension of the semantic communication model is s, and an encoder is used to generate the encoded feature vector z. e (s), which is then mapped to a basis vector z via nearest neighbor lookup b (s) on:

[0061]

[0062] Among them, ||·||2 is the 2-norm. Then z b (s) is input to the decoder. This forward calculation is a layer of DNN with a specific nonlinear function that converts the encoded feature vector z e (s) is mapped to a basis vector z b (s), the basis vectors in the codebook ε Trained together with the encoder and decoder parameters, in backpropagation, the gradient is approximated by a direct estimator and the gradient is fed from the decoder to z b (s) is copied to the encoder output z e (s), the nearest basis vector z b (s) is passed to the decoder, and during the back-propagation process, the gradient is passed unchanged to the encoder.

[0063] The loss function consists of three components, representing different parts of the parameters:

[0064]

[0065] Among them, s, and z represent the input, output and true label of the network respectively, and α is a hyperparameter. e (s)] means not passed to z e The gradient of (s), i.e. z e The gradient of (s) is zero, so ze (s) is a constant that is not updated. The first term of the loss function above is the reconstruction loss that trains the encoder and decoder parameters. Since the mapping from z e (s) to z b (s) is direct gradient estimation, the basis vector does not receive gradients from the reconstruction loss To train the basis vector, the l2 error is used to move the basis vector to the encoded feature z e (s) as shown in the second term. Since the volume of the encoded feature space is dimensionless, if the training speed of the basis vector is not faster than the encoder parameters, the codebook can grow arbitrarily, causing the training process to diverge. To solve this problem, the third term is added in the equation, where ng[e j ] represents the gradient of e j is zero.

[0066] III. Semantic Distortion Function

[0067] The goal of the system is to find the optimal encoder and decoder that minimizes the loss of semantic information and observable data, so the semantic distortion function is introduced, and the optimization objective is to minimize the semantic distortion function Considering the joint optimization scenario of semantic encoder and decoder, the semantic distortion can be expressed as:

[0068]

[0069] where, and represent the distortion measure functions of observable data and pragmatic output, respectively, which can be represented by cross-entropy, mean square error, KL divergence, etc., and θ1, θ2 represent the network parameters of semantic encoder and semantic decoder. When their definitions are different, β is a hyperparameter used to scale the distortion of the observable data to align with the dynamic range of the distortion of the pragmatic information, and λ is a hyperparameter used to weigh the observable data and pragmatic information. Semantic information is determined by a specific task, not the inherent characteristics of the information source. Therefore, both the observable data and the pragmatic information are considered in the definition of the loss function. In addition, the above semantic distortion (SD) is a general measure, where can be extended to multi-task or multi-semantic scenarios.

[0070] When the data set is given, the form of the loss function can be further refined. Given a training batch is a set of training samples , the SD function in this case is called empirical semantic distortion and can be expressed as:

[0071]

[0072] where, is the expectation over the empirical distribution of the training batch .

[0073] The hyperparameter λ mainly balances the weight of the observable data recovery and the pragmatic task. Generally, the trade-off ratio between the two is not explicit, so the value of λ varies according to the actual situation and needs to be adjusted manually. In this embodiment, a flexible setting method is introduced to set the value of λ. The flexible weight hyperparameter λ s can be expressed as

[0074]

[0075] λ s Given the initial value of the hyperparameter λ, it is not a fixed value in practical application. The reason for selecting this value is as follows: the compression rate of semantic communication is [0, 1], which is a kind of lossy joint semantic-channel coding (JSCC) with limited coding rate. And when the compression rate CR is larger, the semantic encoder can choose to encode semantic information or non-semantic information, and its performance tends to be the same as that of the traditional encoder, at this time, the optimization of the pragmatic task part in the loss function is dominant. In the case of smaller compression rate, the semantic encoder can give priority to semantic information and correspondingly converge to the pragmatic encoder, at this time, the observable data part in the loss function is mainly optimized.

[0076] Therefore, when the compression rate CR = cr is given, the joint optimization problem of the network parameters can be expressed as

[0077]

[0078] The distortion measure function of the observable data is selected as the mean square error (MSE). The distortion function of the pragmatic output is set according to the type of the pragmatic task. This application mainly considers two pragmatic tasks, image classification and semantic segmentation, so the distortion measure function is selected as the cross entropy (CE). Based on this, the above optimization problem can be further refined as

[0079]

[0080] where, S MSE and S CE respectively represent the MSE and CE over the empirical distribution of the training batch .

[0081] IV. Training deep learning model based on DL-SC algorithm

[0082] The network training method combining deep learning (DL) and semantic communication (SC), i.e., DL-SC algorithm, models the semantic encoder and the semantic decoder as two deep neural networks (DNNs), respectively. In each training round t∈{1, 2, …}, the transmitter randomly and uniformly extracts samples D from the dataset to form a training batch For each sample The transmitter encodes it and then sends the encoded signal to the receiver through the channel. The receiver can obtain the reconstructed data and the corresponding reconstructed pragmatic output and represent the semantic encoder and the semantic decoder at the t-th round, respectively, which gradually converges with the training process. Meanwhile, the receiver knows the original sample D and the corresponding pragmatic output Z, therefore, a complete training sample is defined as a four-tuple The set of these training samples forms a complete random training batch, denoted as All the contents in the training batch are completely available to the receiver, while the transmitter can only use in the training batch.

[0083] The goal of the system is to minimize the semantic distortion by continuously updating the network parameters, therefore, the semantic distortion function is taken as the loss function during training, denoted as When the transmitter and the receiver know the average gradients and on the training batch, respectively, the network parameters are updated using the stochastic gradient descent method. The receiver can obtain the complete training sample, therefore, it can easily obtain which can be represented as:

[0084]

[0085] Since the transmitter only knows the information of D, it requires the receiver to feedback some necessary values to obtain According to the chain rule, the gradient of the loss function on the encoder network parameters, i.e., can be derived by

[0086] ​

[0087] Specifically, the first term represents the gradient of the loss function with respect to the channel output Y, which can be calculated numerically at the receiving end and then fed back to the sending end. At the same time, since θ1 is not known to the receiving end, the receiving end returns the channel output Y to the sending end and lets it locally calculate That is, the second term of the above formula. Therefore, the content of the feedback can be written as a data tuple It can be seen that no direct information about the pragmatic task function is leaked to the sending end, which guarantees security to some extent. That is, only knowing is not enough for the sending end to determine the actual use of its transmitted data at the receiving end. In addition, from the security perspective of information theory, although secrecy is a quantitative problem, the feedback method of the present embodiment is still secure enough to prevent privacy leakage when the size of the transmitted data is large enough than the size of the actual output data. Using the data tuple The sending end can make corresponding improvements to the network parameters of its semantic encoder.

[0088] The training process of the above algorithm is as follows:

[0089]

[0090] Considering the communication cost in the training process, the size of the original data, the compression rate, the training period and the complexity of the semantic decoder network. In order to reduce the communication overhead in the training process, the observable data reconstruction loss function Locally pre-train the semantic encoder and semantic decoder at the receiving end.

[0091] In the execution phase, after the pre-training is completed, the receiving end sends the parameters of the encoder network to the sending end. Then the sending end and the receiving end collectively train the entire network with the overall loss function;

[0092] In Figure 1 , a semantic communication system model in a mine mining working face is described, and the goal of the sending end is to send some data to the receiving end for completing the pragmatic task and the recovery of observable data. In Figure 2 , the DL-SC algorithm flowchart is described, taking minimizing the loss function as the training goal to form a network training algorithm. In Figure 3 , a semantic encoding and decoding network in a mine mining working face is described, modeling the semantic encoder and decoder as two deep neural networks, respectively.

[0093] In summary, the embodiment of the present application is aimed at the application scene of the mining working face to realize data compression, reduce data transmission amount, significantly improve transmission efficiency, save communication resources, and proposes a communication perception joint resource optimization algorithm based on deep learning and semantic communication, that is, the DL-SC algorithm. The constructed semantic communication system model constructs the semantic encoder and decoder into two deep neural networks, trains the encoder and decoder based on the database, in order to seek the best encoding and decoding function, realizes the goal of minimizing pragmatic information and observable data loss, and sets the loss function as semantic distortion. The DL-SC algorithm in the embodiment can recover key video information while well performing the pragmatic task of video classification and video segmentation under different channel signal-to-noise ratio conditions, and shows significant superiority and feasibility.

[0094] According to the description of the present application, it should not be difficult for those skilled in the art to see that the present application applies semantic communication to the video transmission method of the mining working face, which can effectively eliminate the transmission of redundant data, realize data compression, and further save bandwidth to realize more efficient information interaction with lower data transmission amount.

[0095] Based on the same inventive concept, the embodiment of the present application discloses a mine video semantic communication transmission system based on deep learning, which comprises a semantic encoder and a semantic decoder. The semantic encoder is located at the transmitting end and is realized based on a deep neural network, and is used for extracting data containing observable data and semantic information. The semantic decoder is located at the receiving end and is realized based on a deep neural network, and is used for reconstructing data related to experience data and pragmatic tasks. The semantic encoder and the semantic decoder are optimized based on the DL-SC algorithm with a semantic distortion function as the loss function during training. The embodiment comprises the following steps: the transmitting end extracts a batch of training samples, encodes the samples through the semantic encoder, and then sends them to the receiving end through the channel; the receiving end decodes the received data through the semantic decoder and performs pragmatic tasks; the receiving end calculates the gradient of the loss function on the network parameters of the semantic decoder and updates the network parameters of the semantic decoder; the receiving end calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitting end; the transmitting end calculates the gradient of the channel output on the network parameters of the semantic encoder, combines the gradient fed back by the receiving end, and updates the network parameters of the semantic encoder; and the semantic distortion function is the weighted sum of the semantic information loss function and the observable data loss function. For details of the implementation process, refer to the foregoing method embodiment, which will not be repeated. The contents not described in detail in the present application belong to the prior art known to those skilled in the art.

Claims

1. A mine video semantic communication transmission method based on deep learning, characterized in that: The steps include: A semantic communication system model was established beneath the mining face, including a semantic encoder and a semantic decoder. The semantic encoder, located at the transmitter and implemented based on a deep neural network, was used to extract data containing observable data and semantic information. The semantic decoder, located at the receiver and implemented based on a deep neural network, was used to reconstruct data related to empirical data and pragmatic tasks. The semantic encoder and semantic decoder are trained based on the DL-SC algorithm, using a semantic distortion function as the loss function during training. The method includes: the transmitter extracts a batch of training samples, encodes the samples through the semantic encoder, and then sends them to the receiver through the channel; the receiver decodes the received data through the semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the semantic decoder network parameters and updates the semantic decoder network parameters; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the semantic encoder network parameters and updates the semantic encoder network parameters in combination with the gradient fed back by the receiver. The semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function. The semantic distortion function is expressed as: ； in, and denote the network parameters of the semantic encoder and the semantic decoder respectively, Represents experience data, observable reconstruction data, execution results and reconstruction pragmatic task output, respectively. represents the distortion measure function of the observable data, represents the distortion measure function of pragmatic output, is a hyperparameter for dynamic range alignment, is a hyperparameter used to weigh observable data and pragmatic information; Given a training batch , when the compression rate CR is given, considering the two pragmatic tasks of image classification and semantic segmentation, the joint optimization problem of network parameters is expressed as: ； in, It's about the training batch The expectation of the empirical distribution of and Represents the mean square error and cross entropy with respect to the training batch The empirical distribution of represents the semantic encoder, represents a semantic decoder, represents the pragmatic task function, is a fixed compression rate, N is the power of channel noise; The method of training the semantic encoder and semantic decoder based on the DL-SC algorithm includes the following steps: Step 1. Set the training round counter ; Step 2: Determine whether the maximum number of training rounds has been reached. If not, proceed to step 3; otherwise, end. Step 3: The sender obtains the data from the dataset Extract a batch of training samples ; Step 4: Encode the samples in , and send it out, express Semantic encoder at the round, X represents the output of the semantic encoder; Step 5: The receiving end decodes the received data , express Semantic decoder at the turn, Y represents the channel output; Step 6: The receiving end performs pragmatic tasks , represents pragmatic task function; Step 7: The receiver calculates the gradient according to the following formula , and update Network parameters; ； Step 8: Calculate the gradient at the receiving end ; Step 9: The receiving end will and Feedback to the sender; Step 10: The sender calculates , and update Network parameters; ； Step 11: Add 1 to the training round counter t and go to step 2.

2. The method for mine video semantic communication transmission based on deep learning according to claim 1, characterized in that: The semantic encoder and semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as , consisting of J basis vectors composition, Each basis vector The dimension of Represents a real number; the input s is passed through the semantic encoder to generate the encoded feature vector , by searching for the nearest neighbor, is mapped to a basis vector On, will Input to the semantic decoder.

3. The method for mine video semantic communication transmission based on deep learning according to claim 2 is characterized in that: The codebook The basis vectors in Trained together with the parameters of the semantic encoder and the semantic decoder, the loss function is as follows: ； in, and z represent the output and true label respectively, is a hyperparameter; the symbol express The gradient of is zero, express The gradient of is zero; is the 2-norm.

4. The method for mine video semantic communication transmission based on deep learning according to claim 1, characterized in that: The semantic encoder and semantic decoder are locally pre-trained at the receiving end by minimizing the observable data reconstruction loss function.

5. A deep learning-based mine video semantic communication transmission system, characterized in that: include: A semantic encoder and a semantic decoder, wherein the semantic encoder is located at the transmitter and is implemented based on a deep neural network to extract data containing observable data and semantic information; The semantic decoder is located at the receiving end and is implemented based on a deep neural network to reconstruct data related to the empirical data and pragmatic tasks; The semantic encoder and semantic decoder use the semantic distortion function as the loss function during training and are optimized based on DL-SC algorithm training; the steps include: the transmitter extracts a batch of training samples, encodes the samples through the semantic encoder, and then sends them to the receiver through the channel; the receiver decodes the received data through the semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the semantic decoder network parameters and updates the semantic decoder network parameters; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the semantic encoder network parameters and updates the semantic encoder network parameters in combination with the gradient fed back by the receiver; the semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function; The semantic distortion function is expressed as: ； in, and denote the network parameters of the semantic encoder and the semantic decoder respectively, Represents experience data, observable reconstruction data, execution results and reconstruction pragmatic task output, respectively. represents the distortion measure function of the observable data, represents the distortion measure function of pragmatic output, is a hyperparameter for dynamic range alignment, is a hyperparameter used to weigh observable data and pragmatic information; Given a training batch , when the compression rate CR is given, considering the two pragmatic tasks of image classification and semantic segmentation, the joint optimization problem of network parameters is expressed as: ； in, It's about the training batch The expectation of the empirical distribution of and Represents the mean square error and cross entropy with respect to the training batch The empirical distribution of represents the semantic encoder, represents a semantic decoder, represents the pragmatic task function, is a fixed compression rate, N is the power of channel noise; The method of training the semantic encoder and semantic decoder based on the DL-SC algorithm includes the following steps: Step 1. Set the training round counter ; Step 2: Determine whether the maximum number of training rounds has been reached. If not, proceed to step 3; otherwise, end. Step 3: The sender obtains the data from the dataset Extract a batch of training samples ; Step 4: Encode the samples in , and send it out, express Semantic encoder at the round, X represents the output of the semantic encoder; Step 5: The receiving end decodes the received data , express Semantic decoder at the turn, Y represents the channel output; Step 6: The receiving end performs pragmatic tasks , represents pragmatic task function; Step 7: The receiver calculates the gradient according to the following formula , and update Network parameters; ； Step 8: Calculate the gradient at the receiving end ; Step 9: The receiving end will and Feedback to the sender; Step 10: The sender calculates , and update Network parameters; ； Step 11: Add 1 to the training round counter t and go to step 2.

6. A deep learning-based mine video semantic communication transmission system according to claim 5, characterized in that: The semantic encoder and semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as , consisting of J basis vectors composition, Each basis vector The dimension of represents a real number; let the input of the semantic communication model be , the encoded feature vector is generated by the semantic encoder , by searching for the nearest neighbor, is mapped to a basis vector On, will Input to the semantic decoder.

Images