A mine video semantic communication transmission method based on feature compression

Abstract

This invention discloses a mine video semantic communication and transmission method based on feature compression. The method involves sampling videos collected within the mine frame by frame into images, which are then packaged and fed into a semantic encoder. The image vectors are mapped to feature vectors in a feature space and then divided into blocks. The semantic encoder performs feature-aware compression on the feature vectors, while using a tracking matrix to record the compression path. The compressed features and the tracking matrix are then transmitted to a channel via a transmitter. The semantic decoder first maps the feature vectors to a low-dimensional space using an embedding layer, then recovers the spatial position information of the features based on the tracking matrix. The feature vectors with recovered spatial position information are semantically decoded. The proposed semantic encoder and decoder are jointly trained to minimize the loss function of the target task, and an adaptive estimation method is used to update the parameters of the language model. This invention significantly reduces the amount of transmitted data while barely compromising the performance of the semantic task.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication, and in particular relates to a mine video semantic communication transmission method based on feature compression. Background Art

[0002] With the continuous advancement of communications technology, the efficiency and reliability of communications are constantly breaking new boundaries. This development has fostered the emergence of downstream applications such as intelligent transportation, the Internet of Things, and virtual reality, greatly enriching people's lives. However, these applications have also significantly increased the demand for data transmission, leading to a shortage of communications resources. There are two potential approaches to address this issue. First, expanding to higher frequency bands can alleviate communication pressures in the short term, but higher frequency bands require higher energy consumption. Furthermore, this approach relies on new hardware technologies, making it unsuitable for the current shortage of communications resources. Second, developing more efficient communication methods based on existing communications technologies can fully utilize the limited resources currently available.

[0003] Semantic communication is a novel and efficient communication technology. Rather than understanding data transmission reliability in bits, it focuses more on the success of semantic tasks. In other words, a few bit errors are irrelevant as long as they don't affect the performance of the final task. This significantly improves interference resistance compared to traditional communication. Furthermore, semantic communication interprets input data as features. By extracting these essential features, it can significantly reduce the amount of data required for transmission. Furthermore, compared to pure data information, semantic information encompasses more aspects of the input data, more fully describing the input information and further improving communication reliability. Currently, there are two main approaches to information compression in semantic communication: feature clipping and feature compression. Feature clipping involves discarding unimportant feature blocks, which can significantly reduce the amount of data transmitted. However, this approach has a drawback: if too many blocks are clipped, the retained features cannot effectively describe the input data, which in turn impairs communication reliability. In contrast, the idea of ​​feature compression is to merge the blocks that should have been cropped in feature cropping into the retained blocks. This can greatly reduce the amount of data while retaining sufficient features. The compression rate of this method can vary over a wide range.

[0004] Coal mining also faces severe communication challenges. The underground environment of coal mines is complex, subject to multipath fading, severe electromagnetic interference, and harsh channel conditions. Furthermore, the mining process requires the real-time transmission of a large amount of critical data, such as equipment operating data (vibration, pressure, temperature, and other parameters), underground environmental monitoring data (gas concentration, dust content, roof displacement, etc.), and video image data from the mining face. This data is crucial for ensuring mining safety and improving production efficiency. However, the real-time transmission of this massive amount of data places a huge demand on communication resources. Existing communication methods are susceptible to interference in the complex environment of coal mines, resulting in poor transmission reliability. Furthermore, communication bandwidth is limited, making it difficult to meet the requirements for efficient data transmission. For example, traditional communications use bits as units to ensure reliability. In the strong interference environment of mines, the bit error rate is high, which seriously affects data availability. Conventional data transmission methods do not fully consider semantics, resulting in large amounts of transmitted data and occupying too many communication resources, but cannot accurately meet the demand for "effective semantics" of data in mining operations. For example, when transmitting mining face videos, if a large amount of redundant pixel data is transmitted in a traditional way, it will cause bandwidth congestion, and key semantic information (such as equipment failure characteristics, personnel violation characteristics, etc.) may not be transmitted in a timely and accurate manner due to resource crowding. Summary of the Invention

[0005] Purpose of the invention: In response to the existing shortage of communication resources, the present invention provides a mine video semantic communication transmission method based on feature compression, which is aimed at the application scenario of mine excavation working face, with the purpose of achieving data compression, reducing data transmission volume while ensuring communication performance.

[0006] Technical solution: The method for semantic communication and transmission of mine videos based on feature compression described in the present invention specifically includes the following steps:

[0007] (1) The video captured by the sensors inside the mine is sampled into images by frame, and then the images are packaged and sent to the semantic encoder;

[0008] (2) The semantic encoder maps the received image vector to a feature vector in the feature space through an embedding layer, and then divides the feature vector into blocks;

[0009] (3) The semantic encoder performs feature-aware compression on the feature vector and uses a tracking matrix to record the compression path. The compressed features and tracking matrix are sent to the channel through the transmitter.

[0010] (4) The semantic decoder first maps the received feature vector to a low-dimensional space through an embedding layer, and then recovers the spatial position information of the feature according to the tracking matrix;

[0011] (5) The semantic decoder performs semantic decoding on the feature vector that restores the spatial position information;

[0012] (6) With the goal of minimizing the loss function of the target task, the proposed semantic encoder and semantic decoder are jointly trained, and the parameters of the model composed of the semantic encoder and semantic decoder are updated using an adaptive estimation method.

[0013] Furthermore, the implementation process of step (2) is as follows:

[0014] The image received by the semantic encoder is , B is the batch size of the image, C is the number of channels of the image, H is the length of the image, and W is the width of the image; the image X is input into an embedding layer to obtain the feature vector of the feature space , D is the dimension of the eigenvector, N is the number of eigenvectors; assuming the size of the eigenvector is b, the number of eigenvectors N can be obtained by the following formula:

[0015] .

[0016] Furthermore, the semantic encoder in step (3) performs feature-aware compression processing on the feature vector as follows:

[0017] The processing of the semantic encoder self-attention layer is determined by the following formula:

[0018] ;

[0019] in, It is an input image that generates a feature vector. are called query, key and value respectively, is a trainable weight matrix, D is the dimension of the feature vector; X is obtained by The processed value is called attention score, which is an important indicator to measure the relevance of features;

[0020] The feature compression layer groups and fuses the feature vectors processed by the self-attention layer; the semantic encoder input is After processing by the self-attention layer, we get ,in is the i-th eigenvector, , N is the number of eigenvectors, D is the dimension of the eigenvectors; the N eigenvectors are divided into two groups in an odd-even alternating manner, that is, the eigenvectors at odd positions are divided into one group, and the eigenvectors at even positions are divided into another group; each group contains eigenvectors, respectively recorded as group A and group B;

[0021] A multi-scale aggregation criterion is used to guide the compression of feature vectors. The process is as follows:

[0022] ;

[0023] in, is the compression weight of the i-th eigenvector and the j-th eigenvector, is the compression weight of the i-th eigenvector and the j-th eigenvector under the k-th compression standard, M is the number of compression standards, is the scaling factor of the kth compression standard; the larger the compression weight, the greater the probability that the two features are compressed together;

[0024] For each feature vector in group A, the compression weight is calculated for each feature vector in group B. The number of feature vectors that need to be compressed in each layer of the semantic encoder is r. Then, the feature vectors corresponding to the r largest compression weights are selected in group A and compressed with the feature vectors corresponding to the compression weights in group B.

[0025] Furthermore, the compression criteria include cosine similarity, semantic importance and size;

[0026] Cosine similarity describes the relationship between features at the data level and fuses similar features. The compression weight of cosine similarity is defined as follows:

[0027] ;

[0028] in, Indicates finding the modulus of the j-th eigenvector;

[0029] Semantic importance is represented by the query and key obtained by the attention layer:

[0030] ;

[0031] in, is the compression weight under the semantic importance standard of the i-th eigenvector and the j-th eigenvector, , , represents the influence of the jth eigenvector on other eigenvectors;

[0032] The size of each eigenvector is set to 1, which means that this eigenvector is "compressed" by 1 eigenvector; each time this eigenvector is compressed with other features, the size value is increased by 1; the size of the feature before compression is , N is the number of feature vectors; the compression weight under the size standard is obtained as follows:

[0033] ;

[0034] The smaller the size of two features, the greater the probability that they will be compressed together, making the final feature distribution smoother.

[0035] Furthermore, the feature-aware compression processing of the feature vector in step (3) adopts a layer-by-layer decreasing feature compression strategy, which is as follows:

[0036] r is the number of feature vectors that need to be compressed in each layer of the semantic encoder. A decreasing r distribution is adopted, that is, while ensuring that the total number of reduced feature vectors remains unchanged, the r of the previous layer is made larger and the r of the later layer is made smaller, that is, more features are compressed as early as possible.

[0037] Furthermore, the implementation process of using the tracking matrix to record the compression path in step (3) is as follows:

[0038] The tracking matrix is ​​initially an identity matrix, expressed as: , where B is the batch size and N is the number of feature vectors; after each compression, the same compression operation is performed on the tracking matrix. Obviously, if the number of compressed vectors in the first layer is r, then the feature shape after compression will become , accordingly, the shape of the tracking matrix will become .

[0039] Furthermore, the process of recovering the spatial position information of the feature according to the tracking matrix in step (4) is as follows:

[0040] The spatial features are restored at the semantic decoder. Assume that the number of original feature vectors is 3, that is, the tracking matrix is ​​initially a third-order unit matrix , assuming that the first feature is compressed into the second feature, then the tracking matrix will become: , where the first row corresponds to the first eigenvector after compression, and the second row corresponds to the second eigenvector after compression. The compressed first eigenvector is compressed from the original first and second eigenvectors. The spatial feature is restored by assigning the value of the compressed first eigenvector to the positions corresponding to the original first and second eigenvectors. The entire spatial feature recovery process is expressed as:

[0041] ;

[0042] in, is the eigenvector after spatial feature recovery, is the feature vector received by the semantic decoder, is the tracking matrix, is the spatial feature recovery function.

[0043] Furthermore, the implementation process of the proposed semantic encoder and semantic decoder for joint training in step (6) is as follows:

[0044] (S1) For each training round t, the network samples the video from the video sensor and extracts B batches of sampled image data for batch training;

[0045] (S2) Extract semantic information and compress features of the input image through the semantic encoder:

[0046] ;

[0047] in, is the output feature vector of the semantic encoder, is the input feature of the semantic encoder, R is the compression ratio of feature compression, are the parameters of the semantic encoder network, is the tracking matrix generated by the feature compression layer, is a semantic encoding function;

[0048] (S3) The compressed feature vector It is sent to the decoder through the channel and mapped to the feature dimension corresponding to the decoder through an embedding layer at the front end of the decoder to obtain Y;

[0049] (S4) performing spatial feature recovery processing on Y;

[0050] (S5) Yes Perform semantic decoding to restore it to the original feature space:

[0051] ;

[0052] in, is the feature vector after semantic decoding, are the parameters of the semantic decoding layer, is the semantic decoding function;

[0053] (S6) Calculate the reconstructed eigenvector The mean square loss between the original feature vector X;

[0054] (S7) Repeat (S1) to (S6) and use the adaptive moment estimation method to update the model parameters until the model converges and an approximate optimal solution for the parameters is obtained.

[0055] Furthermore, the loss function in step (6) adopts the minimum mean square loss of the original image and the restored image, which is defined as follows:

[0056] ;

[0057] in, is the feature vector mapped from the original image to the feature space, is the feature vector recovered by the decoder, and N is the number of feature vectors.

[0058] Furthermore, the implementation process of updating the parameters of the model composed of the semantic encoder and the semantic decoder using the adaptive estimation method in step (6) is as follows:

[0059] The network parameters composed of semantic encoder and semantic decoder are ,in , are the parameters of the semantic encoder and semantic decoder respectively, then the first-order moment and second-order moment of the mean square loss with respect to the model parameters are expressed as:

[0060] ;

[0061] in, They represent the first-order moments of the minimum mean square loss of the t-th and t-1-th rounds of training with respect to the model parameters, They represent the second-order moments of the minimum mean square loss of the t-th and t-1-th rounds of training with respect to the model parameters, are the decay rates corresponding to the first-order moment and the second-order moment, represents the minimum mean square loss;

[0062] Model parameters The update rule is obtained as follows:

[0063] ;

[0064] in, , is the correction value, is a constant used to avoid division by 0. is the learning rate of the model.

[0065] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention realizes a communication method that fits the actual scenario through the semantic encoder and semantic decoder designed by joint training; by adopting specific task loss for specific semantic communication tasks, it achieves closeness to the optimal parameters; the semantic communication technology based on feature compression, on the one hand, focuses on the characteristics of the success or failure of the semantic task, which is in line with the needs of mine mining for semantic understanding of key data. When bit errors do not affect the execution performance of semantic tasks, it can effectively cope with the strong interference environment of mines and ensure that data transmission can support tasks such as safety monitoring and equipment control; on the other hand, the ability to reduce the amount of data through feature compression can alleviate the pressure of limited mine communication bandwidth, extract and compress the necessary semantic features in the mining data for transmission, which not only reduces the resources occupied by redundant data, but also allows key semantic information (such as equipment failure characteristics, environmental abnormality semantics, etc.) to be accurately and efficiently transmitted, helping coal mine mining to achieve safer and more efficient intelligent operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a framework diagram of the present invention;

[0067] Figure 2 The comparison diagram of the effects of image transmission using the present invention; (a) is the original image, and (b) is the reconstructed image;

[0068] Figure 3 The comparison of the effects of using MAE to transmit images; (a) is the original image, and (b) is the reconstructed image;

[0069] Figure 4 This is a comparison chart of the training loss of the present invention and MAE as the training rounds change. DETAILED DESCRIPTION

[0070] The present invention is further described in detail below with reference to the accompanying drawings.

[0071] like Figure 1 As shown in the figure, the present invention proposes a mine video semantic communication transmission method based on feature compression, constructs a semantic communication system model under the mining working face, and includes a semantic encoder and a semantic decoder implemented based on a deep neural network; the semantic encoder is located at the transmitting end and is used to extract and compress the semantic features of the observable data; the semantic decoder is located at the receiving end and is used to recover the original semantic information from the received semantic features and perform subsequent semantic tasks; the specific implementation process is as follows:

[0072] Step 1: The sensors inside the mine collect video, sample the video frames into images, and then package the images and send them to the semantic encoder.

[0073] Step 2: The semantic encoder passes the received image through an embedding layer to map the image vector to a feature vector in the feature space, and then divides the feature vector into blocks.

[0074] The semantic encoder processes the feature vector as follows: Assume that the image received by the semantic encoder is , B is the batch size of the image, C is the number of channels of the image, H is the length of the image, and W is the width of the image. The image X is input into an embedding layer to obtain the feature vector of the feature space , D is the dimension of the eigenvector, N is the number of eigenvectors; assuming the size of the eigenvector is b, the number of eigenvectors N can be obtained by the following formula:

[0075] .

[0076] Here we assume that the feature vectors and the image are square, that is, the length and width of the image are the same, which can be easily achieved by image scaling.

[0077] Step 3: The semantic encoder performs feature-aware compression on the feature vector and records the compression path using the tracking matrix. The compressed features and tracking matrix are then sent to the channel through the transmitter.

[0078] The structure of the semantic encoder is as follows Figure 1 As shown in the figure, it is mainly composed of self-attention layer, feature compression layer and multi-layer perceptron.

[0079] Among them, the processing of the self-attention layer can be determined by the following formula:

[0080] ;

[0081] in, It is an input image that generates a feature vector. are called query, key and value respectively, is the trainable weight matrix, and D is the dimension of the feature vector. X is obtained by After processing, it is called the attention score, which is an important indicator to measure the relevance of features.

[0082] The feature compression layer essentially groups and fuses the feature vectors processed by the self-attention layer.

[0083] Specifically, let the semantic encoder input be , after processing by the self-attention layer, we get ,in is the i-th eigenvector, , N is the number of eigenvectors, and D is the dimension of the eigenvectors.

[0084] The feature grouping of the present invention adopts a dichotomy method, that is, N feature vectors are divided into two groups, each group contains Experiments have shown that the effect of alternating odd and even grouping is better, that is, the eigenvectors at odd positions are divided into one group, and the eigenvectors at even positions are divided into another group.

[0085] Unlike the commonly used cosine similarity standard for measuring feature importance, this paper uses an aggregation standard involving multiple scales to guide the compression of feature vectors. The process can be expressed as follows:

[0086] ;

[0087] in, is the compression weight of the i-th eigenvector and the j-th eigenvector, is the compression weight of the i-th eigenvector and the j-th eigenvector under the k-th compression standard, M is the number of compression standards, is the scaling factor of the kth compression criterion; the larger the compression weight, the more likely the two features are to be compressed together.

[0088] The compression criteria of the present invention involve three metrics: cosine similarity, semantic importance and size.

[0089] Among them, cosine similarity is more about describing the relationship between features at the data level, which can guide the fusion of similar features. The compression weight of cosine similarity is defined as follows:

[0090] ;

[0091] in, It means finding the modulus of the j-th eigenvector, that is, the two-norm. The compression weight of the cosine similarity obtained by this formula will be limited to As mentioned above, cosine similarity describes the relationship between feature vectors at the data level. Specifically for image input, the cosine similarity compression standard tends to compress blocks of similar colors together. In fact, blocks of similar colors in an image often have some correlation, that is, their features are repetitive. However, not all blocks of similar colors have similar semantic information. In other words, the cosine similarity standard tends to over-compress, resulting in feature loss.

[0092] Semantic importance can be expressed by the query and key obtained by the attention layer as shown in the following formula:

[0093] ;

[0094] in, is the compression weight under the semantic importance standard of the i-th eigenvector and the j-th eigenvector, ,in , the variable is defined as above, which represents the influence of the jth eigenvector on other eigenvectors. From this, we can see that the semantic importance criterion tends to compress eigenvectors that have a relatively small impact on other eigenvectors together.

[0095] The above two criteria guide feature compression at the data and semantic levels, respectively. The size criterion is based on the fact that, regardless of whether cosine similarity or semantic importance is used, after feature compression, most redundant features are inevitably compressed into a single feature. This results in some of the remaining features being compressed from many features, while others are compressed from fewer features, or even no compression at all. The compression process itself leads to information loss, and excessive compression can impair semantic communication performance. Therefore, the size criterion is introduced.

[0096] Initially, the size of each feature vector is set to 1, which means that this feature vector is "compressed" by 1 feature vector. Each time this feature vector is compressed with other features, the size value is increased by 1. Let the size of the feature before compression be , where N is the number of feature vectors. The compression weight under the size standard is obtained as follows:

[0097] ;

[0098] Obviously, the smaller the size of two features, the more likely they are to be compressed together, which can make the final feature distribution smoother.

[0099] Based on the above aggregation criteria, the feature vectors divided into two groups can be compressed. First, the two groups of feature vectors above are respectively recorded as Group A and Group B, where Group A should be the compressed group. First, for each feature vector in Group A, its compression weight with each feature vector in Group B should be calculated according to the above definition. Assuming that the number of feature vectors that need to be compressed in a certain layer of the encoder is r, it is necessary to select the feature vectors corresponding to the r largest compression weights in Group A and compress them with the feature vector corresponding to the compression weight in Group B. Here, the compression is performed by averaging.

[0100] It should be noted that in the self-attention layer, the amount of computation is directly related to the number of feature vectors, that is, the amount of computation in the self-attention layer is In addition, after each feature compression layer is processed, the number of feature vectors will be reduced by r, where r can vary with the number of layers in the network. Based on the above facts, in order to further reduce the amount of computation, the present invention adopts a decreasing r distribution, that is, while ensuring that the total number of reduced feature vectors remains unchanged, r is made larger in the front layers and smaller in the back layers, that is, more features are compressed as early as possible.

[0101] In addition, the pre-training of the present invention uses an image transmission task, so the compressed features need to be restored at the receiving end. To achieve the above purpose, a tracking matrix is ​​used to track the compression process. Specifically, the tracking matrix is ​​initially a unit matrix, which is expressed as: , where B is the batch size and N is the number of feature vectors. After each compression, the same compression operation is performed on the tracking matrix. Obviously, if the number of compressed vectors in the first layer is r, then the feature shape after compression will become , accordingly, the shape of the tracking matrix will become After the compression process is completed, the remaining feature vectors in group A are concatenated with the feature vectors in group B and input into a multilayer perceptron for further feature extraction.

[0102] The entire semantic encoder processing process can be expressed as follows:

[0103] ;

[0104] in, is the output feature vector of the semantic encoder, is the input feature of the semantic encoder, R is the compression ratio of feature compression, are the parameters of the semantic encoder network, is the tracking matrix generated by the feature compression layer, is a semantic encoding function.

[0105] It should be noted that the self-attention layer, feature compression layer, and multi-layer perceptron process described above are only one layer of the semantic encoder, and a complete semantic compressor is composed of multiple such layers. In this embodiment, the semantic encoder adopts a 12-layer structure.

[0106] Step 4: The semantic decoder first maps the received feature vector to a low-dimensional space through an embedding layer, and then recovers the spatial position information of the feature based on the tracking matrix.

[0107] like Figure 1 As shown in Figure 2, the semantic decoder mainly consists of a spatial feature recovery layer and a semantic decoding layer. The main function of the spatial feature recovery layer is to complete the spatial feature recovery of the compressed feature vector. The processing process is as follows:

[0108] Compared with the semantic encoder, the semantic decoder has smaller layers and dimensions. Therefore, after receiving the feature vector (where B is the batch size, K is the number of feature vectors received by the decoder, and D is the output dimension of the encoder) and the tracking matrix S, we need to pass a linear layer to map X to the corresponding dimensions of the decoder: , is the dimension corresponding to the decoder). Note that the feature vector contained in Y obtained here is compressed from multiple original feature vectors, which cannot directly correspond to the spatial position in the original image. Therefore, the spatial feature of Y must be restored first. The specific process is as follows:

[0109] The tracking matrix received by the semantic decoder is , K is the number of eigenvectors in Y, and N is the number of original eigenvectors. To make the image of the tracking matrix clearer, let's analyze the compression process in the semantic encoder with a simple example:

[0110] Assume that the number of original eigenvectors is 3, that is, the tracking matrix is ​​initially a third-order unit matrix , now assuming that the first feature is compressed into the second feature, then the tracking matrix will become after the corresponding compression operation: , where the first row corresponds to the first eigenvector after compression, and the second row corresponds to the second eigenvector after compression. It can be obtained that the compressed first eigenvector is compressed by the original first eigenvector and the second eigenvector. Therefore, it is only necessary to assign the value of the compressed first eigenvector to the positions corresponding to the original first and second eigenvectors to complete the recovery of the spatial features.

[0111] The entire spatial feature recovery process can be expressed as:

[0112] ;

[0113] in, is the eigenvector after spatial feature recovery, is the feature vector received by the semantic decoder, is the tracking matrix, is the spatial feature recovery function.

[0114] Step 5: The semantic decoder performs semantic decoding on the feature vector that restores the spatial position information.

[0115] It's important to note that while the spatial characteristics of the features have been restored, the feature vectors at each location are still compressed. However, these compressed feature vectors contain the content of the original feature vectors. Therefore, the core function of the decoder is to "search" these original feature vectors from these compressed feature vectors.

[0116] The vector after spatial feature recovery in step 4 is ,,N is the number of original eigenvectors, is the dimension corresponding to the semantic decoder. The input semantic decoding layer performs feature decoding, which "searches" the original feature vector for each spatial position. The semantic decoding process can be expressed as:

[0117] ;

[0118] in, is the feature vector after semantic decoding, are the parameters of the semantic decoding layer, is the semantic decoding function.

[0119] Step 6: Jointly train the proposed semantic encoder and semantic decoder with the goal of minimizing the loss function of the target task.

[0120] The semantic encoder and semantic decoder are pre-trained using the image restoration semantic communication task. The loss is the minimum mean square loss of the original image and the restored image, which is defined as follows:

[0121] ;

[0122] in, is the feature vector mapped from the original image to the feature space, is the feature vector recovered by the decoder, and N is the number of feature vectors.

[0123] After defining the above loss function, the adaptive matrix moment estimation method (ADMA) is used to update the parameters of the model. The specific process is as follows:

[0124] Assume that the network parameters of the semantic encoder and semantic decoder are ,in , are the parameters of the semantic encoder and semantic decoder respectively, then the first-order moment and second-order moment of the mean square loss with respect to the model parameters are expressed as:

[0125] ;

[0126] in, They represent the first-order moments of the minimum mean square loss of the t-th and t-1-th rounds of training with respect to the model parameters, They represent the second-order moments of the minimum mean square loss of the t-th and t-1-th rounds of training with respect to the model parameters, are the decay rates corresponding to the first-order moment and the second-order moment, represents the least mean square loss.

[0127] On this basis, the model parameters The update rule can be obtained as follows:

[0128] ;

[0129] in, , is the correction value, is a constant used to avoid division by 0. is the learning rate of the model.

[0130] The joint training in step 6 specifically includes the following processes:

[0131] S1: For each training round t, the network samples the video from the video sensor and extracts B batches from the sampled image data for batch training.

[0132] S2: Extract semantic information and compress features of the input image through the semantic encoder:

[0133] .

[0134] S3: The compressed feature vector It is sent to the decoder through the channel and mapped to the feature dimension corresponding to the decoder through an embedding layer at the front end of the decoder to obtain Y.

[0135] S4: Perform spatial feature recovery processing on Y:

[0136] .

[0137] S5: Yes Perform semantic decoding to restore it to the original feature space:

[0138] .

[0139] S6: Calculate the reconstructed eigenvector The mean squared loss between the original feature vector X.

[0140] S7: Repeat S1 to S6, and use the adaptive moment estimation method to update the model parameters until the model converges and obtains an approximate optimal solution for the parameters.

[0141] The present invention effectively compresses semantic features at the transmitter and also incorporates a tracking matrix-based feature recovery method to effectively recover the compressed features at the receiver. This method significantly reduces data transmission while ensuring communication quality, thereby greatly increasing communication effectiveness.

[0142] In the simulation experiment, 200 rounds of training were performed with a compression ratio of 0.75. At the same time, in order to generalize and improve the stability of the model, the compression ratio was allowed to fluctuate around 0.75, and the signal-to-noise ratio was within the range of -5 dB to 15 dB. Figure 2 and Figure 3 The following are comparison diagrams of the experimental results of the present invention and the MAE architecture based on the above methods and parameters. The diagram is taken from a coal mine mining operation diagram captured by a high-altitude camera. It can be seen that the reconstruction of the diagram by the present invention has better visual effects than that of MAE, and the reconstructed details are clearer. Figure 4 The figure below compares the changes in training loss of the present invention and MAE with respect to training rounds based on the above parameter settings; the training loss is the minimum mean square loss (MSE) of the reconstruction loss and the original image. It can be seen that both the present invention and MAE tend to converge after about 100 rounds, but the present invention has lower loss. Therefore, the data proves that the present invention has greater advantages than the MAE architecture in image reconstruction tasks.

[0143] The above description merely illustrates the embodiments of the present invention, and while the description is relatively specific and detailed, it should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the scope of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A mine video semantic communication transmission method based on feature compression, characterized in that: The following steps are involved: (1) The video captured by the sensors inside the mine is sampled into images by frame, and then the images are packaged and sent to the semantic encoder; (2) The semantic encoder maps the received image vector to a feature vector in the feature space through an embedding layer, and then divides the feature vector into blocks; (3) The semantic encoder performs feature-aware compression on the feature vector and uses a tracking matrix to record the compression path. The compressed features and tracking matrix are sent to the channel through the transmitter. (4) The semantic decoder first maps the received feature vector to a low-dimensional space through an embedding layer, and then recovers the spatial position information of the feature according to the tracking matrix; (5) The semantic decoder performs semantic decoding on the feature vector that restores the spatial position information; (6) With the goal of minimizing the loss function of the target task, the proposed semantic encoder and semantic decoder are jointly trained, and the parameters of the model composed of the semantic encoder and semantic decoder are updated using an adaptive estimation method; The semantic encoder in step (3) performs feature-aware compression on the feature vector as follows: The processing of the semantic encoder self-attention layer is determined by the following formula: Where X∈R N×D It is an input image that generates a feature vector, Q, K, V = XW Q ,XW K ,XW V They are called query, key and value respectively, W Q ,W K ,W V is a trainable weight matrix, D is the dimension of the feature vector; X is processed by SA(·) and is called the attention score, which is an important indicator for measuring feature relevance; The feature compression layer groups and fuses the feature vectors processed by the self-attention layer; the semantic encoder input is X∈R N ×D After processing by the self-attention layer, we get where x i ; is the i-th eigenvector, i∈[1,N], N is the number of eigenvectors, D is the dimension of the eigenvector; the N eigenvectors are divided into two groups in an odd-even alternating manner, that is, the eigenvectors at odd positions are divided into one group, and the eigenvectors at even positions are divided into another group; each group contains eigenvectors, respectively recorded as group A and group B; A multi-scale aggregation criterion is used to guide the compression of feature vectors. The process is as follows: Among them, W(x i ,x j ) is the compression weight of the i-th eigenvector and the j-th eigenvector, W k (x i ,x j ) is the compression weight of the i-th eigenvector and the j-th eigenvector under the k-th compression standard, M is the number of compression standards, σ k is the scaling factor of the kth compression standard; the larger the compression weight, the greater the probability that the two features are compressed together; For each feature vector in group A, the compression weight is calculated for each feature vector in group B. The number of feature vectors that need to be compressed in each layer of the semantic encoder is r. Then, the feature vectors corresponding to the r largest compression weights are selected in group A and compressed with the feature vectors corresponding to the compression weights in group B.

2. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The implementation process of step (2) is as follows: The image received by the semantic encoder is X∈R B×C×H×W , B is the batch size of the image, C is the number of channels of the image, H is the length of the image, and W is the width of the image; the image X is input into an embedding layer to obtain the feature vector of the feature space D is the dimension of the eigenvector, N is the number of eigenvectors; Assuming the size of the eigenvector is b, the number of eigenvectors N can be obtained by the following formula:

3. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The compression criteria include cosine similarity, semantic importance, and size; Cosine similarity describes the relationship between features at the data level and fuses similar features. The compression weight of cosine similarity is defined as follows: Among them, ||x j || means finding the modulus of the j-th eigenvector; Semantic importance is represented by the query and key obtained by the attention layer: Among them, W info (x i ,x j ) is the compression weight under the semantic importance standard of the i-th eigenvector and the j-th eigenvector, Indicates the influence of the jth eigenvector on other eigenvectors; The size of each eigenvector is set to 1, which means that this eigenvector is "compressed" from another eigenvector. Each time this eigenvector is compressed with other features, the size is increased by 1. The size of the feature before compression is S = [s1, s2, ..., s N ], N is the number of feature vectors; the compression weight under the size standard is obtained as follows: The smaller the size of two features, the greater the probability that they will be compressed together, making the final feature distribution smoother.

4. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The feature-aware compression processing of the feature vector in step (3) adopts a layer-by-layer decreasing feature compression strategy, which is as follows: r is the number of feature vectors that need to be compressed in each layer of the semantic encoder. A decreasing r distribution is adopted, that is, while ensuring that the total number of reduced feature vectors remains unchanged, the r of the previous layer is made larger and the r of the later layer is made smaller, that is, more features are compressed as early as possible.

5. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The implementation process of using the tracking matrix to record the compression path in step (3) is as follows: The tracking matrix is ​​initially an identity matrix, expressed as: S∈R B×N×N , where B is the batch size and N is the number of feature vectors; After each compression, the same compression operation is performed on the tracking matrix. Obviously, if the number of compression vectors in the first layer is r, then the feature shape after compression will become X1∈R B×(N-r)×D , accordingly, the shape of the tracking matrix will become S1∈R B ×(N-r)×N .

6. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The process of recovering the spatial position information of the feature according to the tracking matrix in step (4) is as follows: The spatial features are restored at the semantic decoder. Assume that the number of original feature vectors is 3, that is, the tracking matrix is ​​initially a third-order unit matrix Assuming that the first feature is compressed into the second feature, the tracking matrix will become: Among them, the first row corresponds to the compressed first eigenvector, and the second row corresponds to the compressed second eigenvector. The compressed first eigenvector is compressed by the original first and second eigenvectors. The value of the compressed first eigenvector is assigned to the corresponding positions of the original first and second eigenvectors to complete the recovery of spatial features. The entire spatial feature recovery process is expressed as: Y′=P(Y;S) Where Y′ is the feature vector after spatial feature recovery, Y is the feature vector received by the semantic decoder, S is the tracking matrix, and P(·) is the spatial feature recovery function.

7. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The implementation process of the proposed semantic encoder and semantic decoder for joint training in step (6) is as follows: (S1) For each training round t, the network samples the video from the video sensor and extracts B batches of sampled image data for batch training; (S2) Semantic information extraction and feature compression of the input image are performed through the semantic encoder: in, is the output feature vector of the semantic encoder, X∈R B×N×D is the input feature of the semantic encoder, R is the compression rate of feature compression, θ1 is the parameter of the semantic encoder network, S∈R B×K×N is the tracking matrix generated by the feature compression layer, S e (·) is the semantic encoding function; (S3) The compressed feature vector It is sent to the decoder through the channel and mapped to the feature dimension corresponding to the decoder through an embedding layer at the front end of the decoder to obtain Y; (S4) performing spatial feature recovery processing on Y; (S5) Perform semantic decoding on Y′ and restore it to the original feature space: in, is the feature vector after semantic decoding, θ2 is the parameter of the semantic decoding layer, S d (·) is the semantic decoding function; (S6) Calculate the reconstructed feature vector The mean square loss between the original feature vector X; (S7) Repeat (S1) to (S6), and use the adaptive moment estimation method to update the model parameters until the model converges and an approximate optimal solution for the parameters is obtained.

8. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The loss function in step (6) uses the minimum mean square loss of the original image and the restored image, which is defined as follows: in, is the feature vector mapped from the original image to the feature space, is the feature vector recovered by the decoder, and N is the number of feature vectors.

9. The mine video semantic communication transmission method based on feature compression according to claim 1 is characterized in that: The implementation process of updating the parameters of the model composed of the semantic encoder and the semantic decoder using the adaptive estimation method in step (6) is as follows: The network parameters of the semantic encoder and semantic decoder are θ = {θ1, θ2}, where θ1 and θ2 are the parameters of the semantic encoder and semantic decoder respectively. The first-order moment and second-order moment of the mean square loss with respect to the model parameters are expressed as: in, They represent the first-order moments of the minimum mean square loss of the t-th and t-1-th rounds of training with respect to the model parameters, They represent the second-order moments of the minimum mean square loss of the t-th round and the t-1th round of training with respect to the model parameters, β1 and β2 are the decay rates corresponding to the first-order moment and the second-order moment, respectively. mse (·) represents the minimum mean square loss; The update rule of model parameter θ is obtained as follows: in, is the correction value, γ is a constant used to avoid division by 0, and η is the learning rate of the model.

Images