Mine video semantic communication transmission method and system based on deep learning

Abstract

The invention provides a mine video semantic communication transmission method and system based on deep learning, which can realize the recovery of observable data and the execution of pragmatic tasks. According to the method, a semantic communication system model under a mining working face is constructed, the semantic communication system model comprises a semantic encoder and a semantic decoder which are realized based on a deep neural network, and the semantic encoder is located at a transmitting end and used for extracting data containing observable data and semantic information; the semantic decoder is located at the receiving end and used for reconstructing data related to empirical data and pragmatic tasks. According to the method, a semantic encoder and a semantic decoder are trained based on a DL-SC algorithm, image restoration and pragmatic task execution are considered in the aspect of design of a loss function, appropriate distortion measure functions are selected for the image restoration and pragmatic task execution respectively, and then the weighted sum of the distortion measure functions is solved to form a semantic distortion measure function to serve as the loss function. According to the invention, more efficient information interaction can be realized with lower data transmission quantity.

Description

Technical Field

[0001] The present invention relates to video transmission communication, and in particular to a method and system for transmitting mine video semantic communication based on deep learning under a mining working face. Background Art

[0002] With the continuous development of communication technology, the information receiving and sending ends are becoming more and more intelligent, the demand for intelligent applications of wireless communication is also increasing rapidly, the amount of data generated by the system is also increasing, and the communication traffic will also increase accordingly, making wireless communication face problems such as scarce spectrum resources. Unlike traditional communication, semantic communication focuses on the transmission of semantic information rather than the accurate transmission of original data, thereby eliminating the transmission of redundant data, achieving data compression, and thus saving bandwidth, and achieving more efficient information interaction with lower data transmission volume. In mining, communication resources are more precious, and there is a greater need for fast and efficient transmission of data, so it is very necessary to study semantic communication methods for perceiving information in mining.

[0003] At present, the research on semantic communication mainly includes the extraction and quantification of semantic information and the utilization of semantic information at the pragmatic level. Based on the statement of semantic information theory, there are many studies on semantic communication systems based on deep learning, and the frameworks of these communication systems are roughly similar. Specifically, semantic information is regarded as a hidden feature of observed information, and the relationship between the two is not directly given. However, the empirical data tuples of observable information and related semantic information are provided by a sufficiently large knowledge base. Because the joint distribution between semantic information and observable information may be too complex to be accurately estimated by traditional methods. Summary of the invention

[0004] Purpose of the invention: In view of the above-mentioned problems existing in the prior art, the present invention provides a deep learning-based mine video semantic communication transmission method and system, which is aimed at the application scenario of mine excavation working face to achieve data compression, reduce data transmission volume, improve transmission efficiency and save communication resources.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for transmitting mine video semantic communication based on deep learning, comprising the following steps:

[0007] Establish a semantic communication system model under the mining face of a mine, including a semantic encoder and a semantic decoder. The semantic encoder is located at the transmitting end and is implemented based on a deep neural network to extract data containing observable data and semantic information; the semantic decoder is located at the receiving end and is implemented based on a deep neural network to reconstruct data related to empirical data and pragmatic tasks;

[0008] The semantic encoder and semantic decoder are trained based on the DL-SC algorithm, using a semantic distortion function as the loss function during training; the method includes: the transmitter extracts a batch of training samples, encodes the samples through a semantic encoder, and then sends the samples to the receiver through a channel; the receiver decodes the received data through a semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the network parameters of the semantic decoder and updates the network parameters of the semantic decoder; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the network parameters of the semantic encoder and updates the network parameters of the semantic encoder in combination with the gradient fed back by the receiver; the semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function.

[0009] Furthermore, the semantic encoder and the semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as By J basis vectors Composition, Q is each basis vector e j The dimension of Represents a real number; the input s is passed through the semantic encoder to generate the encoded feature vector z e (s), through nearest neighbor search, z e (s) is mapped to a basis vector z b (s) on, z b (s) is input to the semantic decoder.

[0010] Furthermore, the basis vectors in the codebook ε are Trained together with the parameters of the semantic encoder and the parameters of the semantic decoder, the loss function is as follows:

[0011]

[0012] in, and z represent the output and true label respectively, α is a hyperparameter; the symbol ng[z e (s)] indicates z e The gradient of (s) is zero, ng[e j ] means e j The gradient of is zero; ||·||2 is the 2-norm.

[0013] Furthermore, the semantic distortion function is expressed as:

[0014]

[0015] Among them, θ1 and θ2 represent the network parameters of the semantic encoder and the semantic decoder respectively. They represent empirical data, observable reconstruction data, execution results, and reconstruction pragmatic task outputs, respectively. represents the distortion measure function of the observable data, represents the distortion measure function of pragmatic output, β is a hyperparameter for dynamic range alignment, and λ is a hyperparameter for weighing the observable data and pragmatic information.

[0016] Furthermore, given a training batch When the compression rate CR is given, considering two pragmatic tasks, image classification and semantic segmentation, the joint optimization problem of network parameters is expressed as:

[0017]

[0018] in, It is about the training batch The expectation of the empirical distribution of MSE and S CE Respectively represent the mean square error and cross entropy with respect to the training batch The experience distribution of represents the semantic encoder, represents a semantic decoder, represents the pragmatic task function, cr is the fixed compression rate, and N is the power of the channel noise.

[0019] Furthermore, the training of the semantic encoder and the semantic decoder based on the DL-SC algorithm comprises the following steps:

[0020] Step 1, set the training round counter t=1;

[0021] Step 2: Determine whether the maximum number of training rounds has been reached. If not, proceed to step 3, otherwise end;

[0022] Step 3: The sender obtains the data set Extract a batch of training samples from

[0023] Step 4: Encode the samples in And send it out, represents the semantic encoder at round t, and X represents the output of the semantic encoder;

[0024] Step 5: The receiving end decodes the received data represents the semantic decoder at round t, and Y represents the channel output;

[0025] Step 6: The receiving end performs pragmatic tasks Represents pragmatic task function;

[0026] Step 7: The receiver calculates the gradient according to the following formula and update Network parameters;

[0027]

[0028] Step 8: Calculate the gradient at the receiving end

[0029] Step 9: The receiving end will and Y are fed back to the sending end;

[0030] Step 10: The sender calculates according to the following formula and update Network parameters;

[0031]

[0032] Step 11: Add 1 to the training round counter t and go to step 2.

[0033] Furthermore, the semantic encoder and semantic decoder are locally pre-trained at the receiving end by minimizing the observable data reconstruction loss function.

[0034] In a second aspect, the present invention provides a deep learning-based mine video semantic communication transmission system, comprising: a semantic encoder and a semantic decoder, wherein the semantic encoder is located at a transmitting end, implemented based on a deep neural network, and used to extract data containing observable data and semantic information; the semantic decoder is located at a receiving end, implemented based on a deep neural network, and used to reconstruct data related to empirical data and pragmatic tasks;

[0035] The semantic encoder and semantic decoder use the semantic distortion function as the loss function during training and are optimized based on DL-SC algorithm training; including: the transmitter extracts a batch of training samples, encodes the samples through the semantic encoder and sends them to the receiver through the channel; the receiver decodes the received data through the semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the semantic decoder network parameters and updates the semantic decoder network parameters; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the semantic encoder network parameters, and updates the semantic encoder network parameters in combination with the gradient fed back by the receiver; the semantic distortion function is the weighted sum of the semantic information loss function and the observable data loss function.

[0036] Beneficial effects: Aiming at the integration of communication and perception in the mining face, the deep learning-based mine video semantic communication transmission method and system proposed in the present invention can realize the recovery of observable data and the execution of pragmatic tasks. The present invention constructs a semantic communication model based on neural networks, constructs the semantic codec into two deep neural networks, and adopts the method of joint source channel coding to realize semantic coding. During training, the receiving end is dominant and feeds back data to the sending end, realizing the joint training of the codec without leaking the relevant information of the pragmatic task to the sending end. In terms of the design of the loss function, the two aspects of video recovery and pragmatic task execution are considered. For both, appropriate distortion metric functions are selected, and then their weighted sum is calculated to form a semantic distortion metric function as the loss function, and the minimization of the loss function is used as the training goal to form a network training algorithm, so as to realize more efficient information interaction with lower data transmission volume. In summary, in the scenario of the mining face of the mine, the method of the present invention is efficient in information interaction, and can achieve data compression, reduce data transmission volume, improve transmission efficiency and save communication resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a model diagram of the semantic communication system in an embodiment of the present invention.

[0038] Figure 2 4 is a flow chart of the DL-SC algorithm in an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of the semantic encoding and decoding network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] The embodiment of the present invention discloses a method for transmitting semantic communication of mine videos based on deep learning. First, a semantic communication system model under the mining face of the mine is established. The semantic communication system model adopts a semantic communication system model based on a discrete codebook, including a semantic encoder and a semantic decoder. The semantic encoder is located at the transmitting end and is implemented based on a deep neural network to extract data containing observable data and semantic information. The semantic decoder is located at the receiving end and is implemented based on a deep neural network to reconstruct data related to empirical data and pragmatic tasks. Then, a loss function that minimizes the loss of semantic information and observable data is designed, and a DL-SC algorithm is used to train the deep learning model. Specifically, the transmitter extracts a batch of training samples, encodes the samples through the semantic encoder, and then sends them to the receiver through the channel; the receiver decodes the received data through the semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the semantic decoder network parameters and updates the semantic decoder network parameters; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the semantic encoder network parameters, and updates the semantic encoder network parameters in combination with the gradient fed back by the receiver.

[0042] The semantic communication system model, loss function and DL-SC algorithm involved in the embodiments of the present invention are described in detail below.

[0043] 1. Establishing a semantic communication system model under the mining face

[0044] Dataset It is a collection of experience data D from background knowledge and its corresponding pragmatic task execution result Z. The experience data D is obtained by processing the video frame by frame. The experience data D is first encoded by the semantic encoder f(·) to obtain the X input channel, whose basic element is recorded as Γ X In order to match the image source in RGB format, X is also set to a three-dimensional vector sequence with 8-bit symbols. The compression rate (CR) is defined as:

[0045]

[0046] Among them, Γ D is the basic element of D, |·| is the absolute value symbol, and CR is equivalent to the ratio of the image resolution to the length of the channel input sequence.

[0047] The encoded signal X will be transmitted through an additive white Gaussian noise channel, and the channel output is recorded as Y = X + N. The power of the channel noise N is fixed but unknown. After receiving Y, the receiver can reconstruct the observable data according to the decoder g(·) as And through pragmatic task function Further reconstruction of pragmatic task output

[0048] The encoder f(·) and decoder g(·) in the system are both implemented by deep neural network DNN, and the pragmatic task function Set to a given value. It can be regarded as a mathematical abstraction of pragmatic tasks, and how to obtain it is not the research focus of this invention. Semantic communication based on joint source-channel coding can also be regarded as task-oriented semantic communication, and the encoder considers both semantic information and channel influence. Therefore, the goal of the encoder is to extract data containing the most observable data and semantic information, while the goal of the decoder is to reconstruct data and observable data related to the pragmatic task. Since the pragmatic task information is only known to the receiver, the receiver dominates the training process of the encoder and decoder. By comparing the reconstructed data with the original data, the receiver can guide the transmitter on how to improve the encoder.

[0049] In the perception information transmission model, the ground sensor node can be regarded as the transmitter of the semantic communication system, which includes a semantic encoder, and the receiving device can be regarded as the receiving end, which includes a semantic decoder and needs to perform specific pragmatic tasks. Considering the actual situation, the sensor's transmission power is very low, so only the transmission rate between the sensor node and the receiving device is optimized. The expression of the transmission rate between the sensor node and the receiving device is:

[0050] R s,r =B1log2(1+γ s,r )

[0051] Where B1 represents the channel bandwidth of additive white Gaussian noise, γ s,r It represents the signal-to-noise ratio of the link between the sensor node and the receiving device. It can be seen that the larger the signal-to-noise ratio of the channel, the higher the transmission rate. Therefore, while maximizing the transmission rate, it is also optimizing the signal-to-noise ratio of the link.

[0052] In the semantic communication system, the signal-to-noise ratio of the channel also determines the effect of image transmission. Let the signal-to-noise ratio be γ SNR , the channel input signal and noise signal are x and n respectively, and the signal length is L, then:

[0053]

[0054] Then, under the condition that the signal-to-noise ratio is known, the formula of the noise signal can be derived as follows:

[0055]

[0056] Finally, the noise signal is amplified according to the Gaussian distribution to obtain the final noise signal:

[0057]

[0058] Among them, random(L) represents the random generation of an integer in the open interval [0, L-1). Then the final noise signal is superimposed on the channel input signal to obtain the channel output signal. Obviously, the larger the signal-to-noise ratio of the channel, the smaller the noise, the closer the channel output signal is to the original input signal, and the better the recovery and pragmatic task performance.

[0059] 2. Building a semantic communication system based on discrete codebook

[0060] The discrete codebook is represented as It consists of J basis vectors Composition, Q is each basis vector e j The dimension of the semantic communication model is s, and an encoder is used to generate the encoded feature vector z e (s), which is then mapped to a basis vector z through nearest neighbor search b (s) on:

[0061]

[0062] Among them, ||·||2 is the 2-norm. Then z b (s) is input to the decoder. This forward calculation is a layer of DNN with a specific nonlinear function that converts the encoded feature vector z e (s) is mapped to a basis vector z b (s), the basis vectors in the codebook ε Together with the parameters of the encoder and decoder, the gradient is approximated by a direct estimator during backpropagation and the gradient is fed from the decoder to z b (s) is copied to the encoder output z e (s), the nearest basis vector z b (s) is passed to the decoder, and during the back-propagation process, the gradient is passed to the encoder unchanged.

[0063] The loss function consists of three components, representing different parts of the parameters:

[0064]

[0065] Among them, s, and z represent the input, output and true label of the network respectively, and α is a hyperparameter. e (s)] means not passed to z e The gradient of (s), i.e. z e The gradient of (s) is zero, so ze (s) is constrained to a constant that is not updated. The first term of the above loss function is the reconstruction loss for training the encoder and decoder parameters. e (s) to z b (s) is the direct gradient estimate of the mapping, the basis vector No losses from reconstruction To train the basis vectors, we use the l2 error to move the basis vectors to the encoded features z e (s), as shown in the second term. Since the volume of the encoded feature space is dimensionless, if the basis vector The training speed of is not as fast as that of the encoder parameters, and the codebook can grow arbitrarily, causing the training process to diverge. To solve this problem, a third term is added to the formula, where ng[e j ] means e j The gradient of is zero.

[0066] 3. Semantic Distortion Function

[0067] The goal of the system is to find the optimal encoder and decoder to minimize the loss of semantic information and observable data, so the semantic distortion function is introduced, and the optimization goal is to minimize the semantic distortion function. Considering the scenario of joint optimization of semantic encoder and decoder, the semantic distortion can be expressed as:

[0068]

[0069] in, and They represent the distortion metric functions of observable data and pragmatic output, respectively, which can be represented by cross entropy, mean square error, KL divergence, etc. θ1, θ2 represent the network parameters of semantic encoder and semantic decoder. When their definitions are different, β is used to scale the distortion The value of The dynamic range of the distortion is aligned with the hyperparameter, while λ is a hyperparameter used to weigh the observable data and pragmatic information. Semantic information is determined by the specific task rather than the inherent characteristics of the information source. Therefore, both observable data and pragmatic information are considered in the definition of the loss function. In addition, the above-mentioned semantic distortion (SD) is a general metric, in which Can be extended to multi-task or multi-semantic scenarios.

[0070] When the dataset is given, the form of the loss function can be further refined. Given a training batch For training samples The SD function in this case is called empirical semantic distortion and can be expressed as:

[0071]

[0072] in, It is about the training batch The expectation of the empirical distribution of .

[0073] The main function of the hyperparameter λ is to balance the weight of observable data recovery and pragmatic tasks. Usually, the trade-off ratio between the two is unclear, so the selection of the λ value will vary according to the actual situation and needs to be adjusted manually. This embodiment introduces a flexible setting method to set the value of λ. Elastic weight hyperparameter λ s It can be expressed as

[0074]

[0075] λ s Given the initial value of the hyperparameter λ, it is not a fixed value in practical applications. The reasons for choosing this value are as follows: the compression rate of semantic communication is [0,1], which is a lossy joint semantic channel coding (JSCC) with limited coding rate. And when the compression rate CR is large, the semantic encoder can choose to encode both semantic information and non-semantic information, and its performance converges with the traditional encoder. At this time, the optimization of the pragmatic task part in the loss function dominates. When the compression rate is small, the semantic encoder can give priority to semantic information and converge to the pragmatic encoder accordingly. At this time, the observable data part in the loss function is mainly optimized.

[0076] Therefore, when the compression ratio CR = cr is given, the joint optimization problem of network parameters can be expressed as

[0077]

[0078] The distortion metric function of observable data is the mean square error (MSE). The distortion function of pragmatic output is set according to the type of pragmatic task. The present invention mainly considers two pragmatic tasks, image classification and semantic segmentation, so the distortion metric function is cross entropy (CE). Based on this, the above optimization problem can be further refined as follows:

[0079]

[0080] Among them, S MSE and S CE Respectively represent MSE and CE on the training batch The empirical distribution of .

[0081] 4. Training deep learning models based on DL-SC algorithm

[0082] A network training method that combines deep learning (DL) with semantic communication (SC), namely the DL-SC algorithm, which combines the semantic encoder and semantic decoder They are modeled as two deep neural networks (DNNs). In each training round t∈{1,2,…}, the transmitter randomly and uniformly selects D samples are extracted from the training set to form a training batch For each sample The transmitter encodes it The encoded signal is then sent to the receiving end through the channel. The receiving end can obtain the reconstructed data and the corresponding reconstructed pragmatic output and They represent the semantic encoder and semantic decoder at round t, which will gradually converge with the training process. At the same time, the receiver knows the original sample D and the pragmatic output Z corresponding to D. Therefore, a complete training sample is defined as a four-tuple The collection of these training samples forms a complete random training batch, denoted as Training batches All the contents in the training batch are fully available to the receiver, while the transmitter only has the contents in the training batch. Can be used.

[0083] The goal of the system is to minimize the semantic distortion by continuously updating the network parameters, so the semantic distortion function As the loss function during training, it is abbreviated as When the transmitter and receiver know the average gradient of the training batch respectively and When , the network parameters are updated using the stochastic gradient descent method. The complete training samples can be obtained at the receiving end, so the receiving end can easily obtain the training data of each sample T. It can be expressed as:

[0084]

[0085] Since the sender only knows the information of D, it requires the receiver to feedback some necessary values ​​in order to obtain According to the chain rule, the gradient of the loss function on the encoder network parameters is It can be derived by the following formula

[0086]

[0087] Specifically, the first term represents the gradient of the loss function with respect to the channel output Y, which can be numerically calculated at the receiving end and then fed back to the sending end. At the same time, since θ1 is unknown to the receiving end, the receiving end returns the channel output Y to the sending end and lets it calculate locally That is the second term of the above formula. Therefore, the feedback content can be written as a data tuple It can be seen that direct information about the pragmatic task function is not leaked to the sender, which ensures security to a certain extent. It is not enough for the sender to determine the actual use of its transmitted data at the receiving end. In addition, from the perspective of information theory security, although confidentiality is a quantitative problem, the feedback method of this embodiment is still sufficiently secure when the size of the transmitted data is large enough to prevent privacy leakage. The transmitter can make corresponding improvements to the network parameters of its semantic encoder.

[0088] The training process of the above algorithm is as follows:

[0089]

[0090] Considering that the communication cost during training is related to the size of the original data, the compression rate, the training cycle, and the complexity of the semantic decoder network, in order to reduce the communication overhead during training, the observable data reconstruction loss function can be minimized. The semantic encoder and semantic decoder are locally pre-trained at the receiving end.

[0091] In the execution phase, after pre-training is completed, the receiver sends the parameters of the encoder network to the transmitter. Then the transmitter and receiver collectively train the entire network with the overall loss function;

[0092] exist Figure 1 In , a semantic communication system model in a mining face is described. The goal of the transmitter is to send some data to the receiver to complete pragmatic tasks and recover observable data. Figure 2 In , the DL-SC algorithm flow chart is described, with minimizing the loss function as the training goal to form a network training algorithm. Figure 3 In this paper, the semantic encoding and decoding network in the mining face is described, and the semantic encoder and decoder are modeled as two deep neural networks respectively.

[0093] In summary, the embodiment of the present invention is aimed at the application scenario of mining working face, with the purpose of realizing data compression, reducing the amount of data transmission, significantly improving transmission efficiency, and saving communication resources, and proposes a communication-aware joint resource optimization algorithm based on deep learning and semantic communication, namely the DL-SC algorithm. In the constructed semantic communication system model, the semantic codec is constructed into two deep neural networks, and the codec is jointly trained based on the database. In order to seek the best coding function and achieve the goal of minimizing the loss of pragmatic information and observable data, the loss function is set to semantic distortion. Under different channel signal-to-noise ratio conditions, the DL-SC algorithm in this embodiment can perform the pragmatic tasks of video classification and video segmentation well while restoring key video information, showing significant superiority and feasibility.

[0094] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the present invention proposes a method for video transmission of a mining face using semantic communication, which can effectively eliminate the transmission of redundant data, achieve data compression, and thereby save bandwidth, thereby achieving more efficient information interaction with a lower data transmission volume.

[0095] Based on the same inventive concept, an embodiment of the present invention discloses a deep learning-based mine video semantic communication transmission system, comprising: a semantic encoder and a semantic decoder, wherein the semantic encoder is located at the transmitting end, implemented based on a deep neural network, and used to extract data containing observable data and semantic information; the semantic decoder is located at the receiving end, implemented based on a deep neural network, and used to reconstruct data related to empirical data and pragmatic tasks; the semantic encoder and the semantic decoder use a semantic distortion function as a loss function during training, and are optimized based on DL-SC algorithm training; comprising: the transmitting end extracts a batch of training samples, encodes the samples through a semantic encoder, and then sends them to the receiving end through a channel; the receiving end decodes the received data through a semantic decoder and performs pragmatic tasks; the receiving end calculates the gradient of the loss function on the network parameters of the semantic decoder, and updates the network parameters of the semantic decoder; the receiving end calculates the gradient of the loss function relative to the channel output, and feeds it back to the transmitting end together with the channel output; the transmitting end calculates the gradient of the channel output on the network parameters of the semantic encoder, and updates the network parameters of the semantic encoder in combination with the gradient fed back by the receiving end; the semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function. The detailed implementation process is referred to the above method embodiment, which will not be described in detail. The contents not described in detail in the present invention belong to the prior art known to professionals in the field.

Claims

1. A mine video semantic communication transmission method based on deep learning, characterized in that: The steps include: Establish a semantic communication system model under the mining face of a mine, including a semantic encoder and a semantic decoder. The semantic encoder is located at the transmitting end and is implemented based on a deep neural network to extract data containing observable data and semantic information; the semantic decoder is located at the receiving end and is implemented based on a deep neural network to reconstruct data related to empirical data and pragmatic tasks; The semantic encoder and semantic decoder are trained based on the DL-SC algorithm, using a semantic distortion function as the loss function during training; the method includes: the transmitter extracts a batch of training samples, encodes the samples through a semantic encoder, and then sends the samples to the receiver through a channel; the receiver decodes the received data through a semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the network parameters of the semantic decoder and updates the network parameters of the semantic decoder; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the network parameters of the semantic encoder and updates the network parameters of the semantic encoder in combination with the gradient fed back by the receiver; the semantic distortion function is a weighted sum of the semantic information loss function and the observable data loss function.

2. According to a method for transmitting mine video semantic communication based on deep learning according to claim 1, it is characterized in that: The semantic encoder and the semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as By J basis vectors Composition, Q is each basis vector e j The dimension of Represents a real number; the input s is passed through the semantic encoder to generate the encoded feature vector z e (s), through nearest neighbor search, z e (s) is mapped to a basis vector z b (s) on, z b (s) is input to the semantic decoder.

3. According to a deep learning-based mine video semantic communication transmission method according to claim 2, it is characterized in that: The basis vectors in the codebook ε Trained together with the parameters of the semantic encoder and the parameters of the semantic decoder, the loss function is as follows: in, and z represent the output and true label respectively, α is a hyperparameter; the symbol ng[z e (s)] indicates z e The gradient of (s) is zero, ng[e j ] means e j The gradient of is zero; ||·||2 is the 2-norm.

4. According to a method for mine video semantic communication transmission based on deep learning according to claim 1, it is characterized in that: The semantic distortion function is expressed as: Among them, θ1 and θ2 represent the network parameters of the semantic encoder and the semantic decoder respectively. They represent empirical data, observable reconstruction data, execution results, and reconstruction pragmatic task outputs, respectively. represents the distortion measure function of the observable data, represents the distortion measure function of pragmatic output, β is a hyperparameter for dynamic range alignment, and λ is a hyperparameter for weighing the observable data and pragmatic information.

5. A method for mine video semantic communication transmission based on deep learning according to claim 4, characterized in that: Given a training batch , when the compression rate CR is given, considering the two pragmatic tasks of image classification and semantic segmentation, the joint optimization problem of network parameters is expressed as: stCR=cr,λ=1-CR in, It is about the training batch The expectation of the empirical distribution of MSE and S CE Respectively represent the mean square error and cross entropy with respect to the training batch The experience distribution of represents the semantic encoder, represents a semantic decoder, represents the pragmatic task function, cr is the fixed compression rate, and N is the power of the channel noise.

6. The method for transmitting mine video semantic communication based on deep learning according to claim 4, characterized in that: The method of training a semantic encoder and a semantic decoder based on the DL-SC algorithm comprises the following steps: Step 1, set the training round counter t=1; Step 2: Determine whether the maximum number of training rounds has been reached. If not, proceed to step 3, otherwise end; Step 3: The sender obtains the data set Extract a batch of training samples from Step 4: Encode the samples in And send it out, represents the semantic encoder at round t, and X represents the output of the semantic encoder; Step 5: The receiving end decodes the received data represents the semantic decoder at round t, and Y represents the channel output; Step 6: The receiving end performs pragmatic tasks Represents pragmatic task function; Step 7: The receiver calculates the gradient according to the following formula and update Network parameters; Step 8: Calculate the gradient at the receiving end Step 9: The receiving end will and Y are fed back to the sending end; Step 10: The sender calculates according to the following formula and update Network parameters; Step 11: Add 1 to the training round counter t and go to step 2.

7. The method for mine video semantic communication transmission based on deep learning according to claim 1, characterized in that: The semantic encoder and semantic decoder are locally pre-trained at the receiving end by minimizing the observable data reconstruction loss function.

8. A deep learning-based mine video semantic communication transmission system, characterized in that: include: A semantic encoder and a semantic decoder, wherein the semantic encoder is located at the transmitting end and is implemented based on a deep neural network and is used to extract data containing observable data and semantic information; The semantic decoder is located at the receiving end and is implemented based on a deep neural network to reconstruct data related to the empirical data and pragmatic tasks; The semantic encoder and semantic decoder use the semantic distortion function as the loss function during training and are optimized based on DL-SC algorithm training; including: the transmitter extracts a batch of training samples, encodes the samples through the semantic encoder and sends them to the receiver through the channel; the receiver decodes the received data through the semantic decoder and performs pragmatic tasks; the receiver calculates the gradient of the loss function on the semantic decoder network parameters and updates the semantic decoder network parameters; the receiver calculates the gradient of the loss function relative to the channel output and feeds it back to the transmitter together with the channel output; the transmitter calculates the gradient of the channel output on the semantic encoder network parameters, and updates the semantic encoder network parameters in combination with the gradient fed back by the receiver; the semantic distortion function is the weighted sum of the semantic information loss function and the observable data loss function.

9. A deep learning-based mine video semantic communication transmission system according to claim 8, characterized in that: The semantic encoder and the semantic decoder perform semantic communication based on a discrete codebook; the codebook is represented as By J basis vectors Composition, Q is each basis vector e j The dimension of represents a real number; let the input of the semantic communication model be s, and the encoded feature vector z is generated by the semantic encoder e (s), through nearest neighbor search, z e (s) is mapped to a basis vector z b (s) on, z b (s) is input to the semantic decoder.

10. The deep learning-based mine video semantic communication transmission system according to claim 8, characterized in that: The semantic distortion function is expressed as: Among them, θ1 and θ2 represent the network parameters of the semantic encoder and the semantic decoder respectively. They represent empirical data, observable reconstruction data, execution results, and reconstruction pragmatic task outputs, respectively. represents the distortion measure function of the observable data, represents the distortion measure function of pragmatic output, β is a hyperparameter for dynamic range alignment, and λ is a hyperparameter for weighing the observable data and pragmatic information.

Images