Video semantic transmission method based on priority division

Abstract

The application provides a video semantic transmission method based on priority division, which comprises the following steps: acquiring an image group corresponding to a to-be-transmitted video, and performing multi-task-based feature extraction to obtain action vector features, background features and object features of the to-be-transmitted video; next, performing feature priority division, and screening the divided features according to a dynamic feature screening threshold to obtain fused features; mapping the priority weight corresponding to the fused features to a probability distribution; performing arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution to obtain corresponding video semantic features; finally, performing semantic decoding processing, and reconstructing the decoded image group to obtain a transmitted video. Through the application, the technical problem of loss of important data feature information and low quality of subsequent video recovery and reconstruction in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video semantic processing, and particularly relates to a video semantic transmission method based on priority division. BACKGROUND

[0002] Semantic communication systems aim to reconstruct the original information or perform target-oriented intelligent tasks at the receiver. Semantic communication transmits more relevant data within limited bandwidth, thereby improving transmission efficiency and robustness. Semantic communication no longer focuses on the accurate transmission of symbols, but on the transmission accuracy of semantic information behind the bits.

[0003] In the prior art, the research focus of the semantic communication system for video sources is on the compression and transmission of source information. In order to save channel bandwidth and cope with complex channel conditions, the semantic communication system will segment the transmission of source content at the sending end. For example, a key point-based video recovery method only transmits key point information in real time, and the background and other non-key information are shared at the beginning. For example, a group of pictures is directly segmented into people and background, and the background information is transmitted only once, and all the person information is transmitted. These methods achieve the purpose of saving channel bandwidth, but the background, object and motion features will maintain a certain continuity in adjacent frames. Ignoring these features will affect the visual consistency, and the data features are not considered to be divided, and the correlation between the semantic features in the background information and the motion object features is not considered, but the features are divided from the whole class (such as background, object, etc.). As a result, on the one hand, important data feature information is lost, and on the other hand, the channel bandwidth resources are not fully utilized, affecting the recovery of moving objects, the reconstruction quality of the video and the user experience. SUMMARY

[0004] The present application provides a video semantic transmission method based on priority division, which solves the technical problems of losing important data feature information and low quality of subsequent video recovery and reconstruction in the prior art when using segmented transmission of source information in semantic communication.

[0005] The present application provides a video semantic transmission method based on priority division, which solves the technical problems of losing important data feature information and low quality of subsequent video recovery and reconstruction in the prior art when using segmented transmission of source information in semantic communication.

[0006] An image group corresponding to a to-be-transmitted video is obtained, and multi-task-based feature extraction is performed on the image group to obtain action vector features, background features and object features of the to-be-transmitted video.

[0007] The action vector feature, the background feature, and the object feature are classified according to feature priorities, and the classified features are filtered according to a dynamic feature filtering threshold to obtain a fusion feature, wherein the feature filtering threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature filtering threshold complies with a negative correlation relationship.

[0008] The priority weight corresponding to the fusion feature is mapped to a probability distribution, and the fusion feature is arithmetically encoded and arithmetically decoded according to the probability distribution to obtain a corresponding video semantic feature, wherein the priority weight is used to represent the correlation degree of actions and background objects in the video to be transmitted.

[0009] The video semantic feature is subjected to semantic decoding processing, and the decoded image group is reconstructed to obtain a transmitted video.

[0010] In some embodiments, the image group is subjected to multi-task-based feature extraction to obtain the action vector feature, the background feature, and the object feature of the video to be transmitted, including:

[0011] A feature extraction model trained based on multiple tasks is obtained, wherein the multiple tasks selected during training of the feature extraction model include an action feature extraction task, an action classification task, and a background extraction task.

[0012] The feature extraction model is called to extract features from each frame of the image group to obtain the action vector feature, the background feature, and the object feature of the video to be transmitted.

[0013] In some embodiments, the action vector feature, the background feature, and the object feature are classified according to feature priorities, including:

[0014] The background feature and the background feature are combined to form a background object feature sequence.

[0015] The action vector feature and the background object feature sequence are subjected to multi-head attention calculation to obtain a group of attention scores corresponding to the background object feature sequence.

[0016] The attention scores are used as distribution weights to perform weighted summation on feature elements in the background object feature sequence, and the result of the weighted summation is fused with the action vector feature.

[0017] In some embodiments, the action vector feature, the background object feature sequence are subjected to multi-head attention calculation to obtain a group of attention scores corresponding to the background object feature sequence, including:

[0018] The sequence of the action vector features is taken as a query vector, and the sequence of the background object features is taken as a key vector and a value vector respectively;

[0019] The query vector, the key vector, and the value vector are input into a plurality of attention heads for attention calculation, and a set of attention scores is output, the attention scores being attention calculation values between each feature element in the action vector features and feature elements in the sequence of the background object features.

[0020] In some embodiments, the transmitted video is constructed after semantic transmission of the video to be transmitted by calling a video semantic transmission model, and a training method of the video semantic transmission model includes:

[0021] Obtaining a video sample;

[0022] Inputting the video sample into the video semantic transmission model for forward propagation to obtain a reconstructed video, and determining an action prediction label of the video sample during the forward propagation;

[0023] Constructing a final loss function of the video semantic transmission model based on the reconstructed video and the action prediction label;

[0024] Updating parameters of the video semantic transmission model through the final loss function for backward propagation in the video semantic transmission model.

[0025] In some embodiments, the video sample carries a real classification label of an action category, and the final loss function of the video semantic transmission model is constructed based on the reconstructed video and the action prediction label, including:

[0026] Constructing a cross-entropy loss function as an action classification loss function of the video semantic transmission model according to the real classification label and the action prediction label;

[0027] Constructing a mean square error loss function as a video reconstruction loss function of the video semantic transmission model according to real video frames in the video sample and reconstructed video frames in the reconstructed video;

[0028] A preset action classification loss weight and a video reconstruction loss weight are obtained, the action classification loss function is weighted by the action classification loss weight to obtain an action classification weighted loss, and the video reconstruction loss function is weighted by the video reconstruction loss weight to obtain a video reconstruction weighted loss;

[0029] The action classification weighted loss and the video reconstruction weighted loss are summed to obtain the final loss function of the video semantic transmission model.

[0030] The application further provides a video semantic transmission device based on priority division.

[0031] The video feature extraction module is configured to acquire an image group corresponding to the to-be-transmitted video, and perform multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the to-be-transmitted video.

[0032] The feature priority division module is configured to perform feature priority division on the action vector features, the background features and the object features, and perform screening on the divided features according to a dynamic feature screening threshold to obtain fused features, wherein the feature screening threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship.

[0033] The feature dynamic transmission module is configured to map a priority weight corresponding to the fused features into a probability distribution, perform arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution, and obtain corresponding video semantic features, wherein the priority weight is used to represent a correlation degree of actions and background objects in the to-be-transmitted video.

[0034] The video feature reconstruction module is configured to perform semantic decoding processing on the video semantic features, and reconstruct the decoded image group to obtain a transmitted video.

[0035] The application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the video semantic transmission method based on priority division according to any one of the above when executing the computer program.

[0036] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the video semantic transmission method based on priority division according to any one of the above.

[0037] The application further provides a computer program product, which comprises a computer program, and the computer program is executable on a processor to implement the video semantic transmission method based on priority division according to any one of the above.

[0038] The application provides a video semantic transmission method based on priority division. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description one by one. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0040] Figure 1 Fig. 1 is a flowchart of the video semantic transmission method based on priority division provided by the application.

[0041] Figure 2 Fig. 2 is a schematic diagram of the framework of the video semantic transmission method based on priority division provided by the application.

[0042] Figure 3 Fig. 3 is a schematic diagram of the feature screening process provided by the application.

[0043] Figure 4 Fig. 4 is a structural schematic diagram of the video semantic transmission device based on priority division provided by the application.

[0044] Figure 5 Fig. 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all the other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the application.

[0046] The priority-based video semantic transmission method of the present application can be applied to electronic devices capable of video playing or video transmission, such as servers or terminals. Therefore, the execution subject of the method is a terminal or a server. When the electronic device performs a video transmission task, the image group corresponding to the video to be transmitted is obtained by the method, and the video after transmission is finally obtained to complete the video transmission task of the electronic device.

[0047] The priority-based video semantic transmission method and device of the present application will be described below in conjunction with the accompanying drawings, Figure 1 is a flowchart of the priority-based video semantic transmission method provided by the present application, as Figure 1 shown, the method comprises the following steps 101 to 104.

[0048] Step 101, obtaining an image group corresponding to a video to be transmitted, and performing multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the video to be transmitted.

[0049] First, in the application scenario requiring video semantic transmission, the video to be transmitted is obtained, and then the video to be transmitted is divided into images frame by frame as the corresponding image group.

[0050] When transmitting, the image group needs to be subjected to multi-task-based feature extraction to obtain the action vector features, background features and object features of the video to be transmitted.

[0051] Here, the present application embodiment pre-trains a feature extraction model, which is trained based on multiple tasks, that is, the training method of multi-task learning is adopted during training. The multiple tasks selected during training of the feature extraction model include action feature extraction task, action classification task and background extraction task. The feature extraction model can be a shared feature extractor based on 3D-CNN network.

[0052] During multi-task learning, the shared video features required by several tasks are first extracted by the shared feature extractor, and then the shared video features are respectively input into three branches for executing multiple tasks. The three branches are action extraction branch, action classification branch and background extraction branch, which are respectively used to execute action feature extraction task, action classification task and background extraction task.

[0053] During multi-task learning, the performance of the model on each task can be improved by learning multiple related tasks. Moreover, the parameters can be shared while executing multiple tasks, the parameters of all tasks are learned, the joint optimization of parameters is realized, and the feature extraction capability of the model is improved.

[0054] Therefore, when feature extraction is performed on the image group, a feature extraction model trained based on multi-task learning is first obtained, and then the feature extraction model is called to perform feature extraction on each frame of image in the image group to obtain the action vector feature, the background feature and the object feature of the video to be transmitted.

[0055] Here, each frame of image in the image group is sequentially input into the feature extraction model, and the model directly outputs the action vector feature, the background feature and the object feature corresponding to the image frame. After all the images are extracted, the action vector feature, the background feature and the object feature of the video to be transmitted are obtained.

[0056] As shown in Figure 2 When video feature extraction is performed, a group of frame pictures (i.e. the image group above) is input to the shared feature extractor of the shared feature layer, and then enters the action extraction branch, the action classification branch and the background extraction branch respectively for feature extraction, and finally outputs the action vector feature, the background feature and the object feature.

[0057] In the embodiment of the application, when video feature extraction is performed, the feature extraction model is trained by a multi-task learning method. During training, the feature extraction model can improve the performance on each task by learning multiple related tasks, and can perform parameter sharing while performing multiple tasks, learn the parameters of all tasks, realize joint optimization of the parameters, and enhance the feature extraction effect of the model.

[0058] In step 102, the action vector feature, the background feature and the object feature are divided into feature priority, and the divided features are filtered according to the dynamic feature filtering threshold to obtain the fusion feature.

[0059] After feature extraction by step 101, the action vector feature, the background feature and the object feature are then divided into feature priority. The purpose of priority division is to focus on the correlation degree of action and background object. Because the more relevant action and background object is, the more important the key information belonging to video transmission is, and the higher the priority is. These key information determines the quality of subsequent video reconstruction. To calculate the correlation degree of action and background object, the embodiment of the application is calculated by joint multi-head attention mechanism, and then the feature priority division is realized.

[0060] As shown in Figure 3 The background feature physical feature sequence is formed, and then the action feature is input into the multi-head attention module for attention calculation to realize the priority division of the video feature. Then, the features obtained by division are filtered to finally obtain the fusion feature, which will be described below.

[0061] First, the background features and the background feature sequence are formed, and then the action vector features and the background object feature sequence are calculated by multi-head attention to obtain a group of attention scores corresponding to the background object feature sequence. Here, the attention score is used to measure the correlation between the action and the background object. The higher the attention score, the higher the correlation between the action and the background object, and vice versa.

[0062] Specifically, the sequence formed by the action vector features is taken as a query vector (Query), and the background object feature sequence is taken as a key vector (Key) and a value vector (Value). The joint multi-head attention mechanism realizes through multiple attention heads, and inputs the query vector, the key vector and the value vector into multiple attention heads for attention calculation, and outputs the attention score corresponding to each attention head. The attention score is the attention calculation value between each feature element in the action vector feature and each feature element in the background object feature sequence.

[0063] Here, when the attention head is calculated, the attention calculation value between each feature element in the action vector feature and all feature elements in the background object feature sequence is calculated to obtain a group of attention calculation values, for example, if there are 10 feature elements in the action vector feature, 10 attention calculation values will be calculated. Then, the group of attention calculation values calculated by each attention head is averaged to obtain a group of average attention scores, for example, if there are 3 attention heads, 3 groups of attention calculation values are obtained, and then the corresponding 3 groups of attention calculation values are averaged to obtain 10 average attention calculation values.

[0064] Finally, the attention score is taken as a distribution weight to weight and sum the feature elements in the background object feature sequence, and the result obtained by the weighted sum is fused with the action vector feature. The attention score can be a group of distribution weight values, which is used to represent the correlation between the action feature elements and the background object feature elements. Then, the background object feature elements are weighted by the distribution weight values, and then the weighted results are summed, and finally the summed background object features are fused with the action vector features.

[0065] As Figure 2As shown, the action vector feature, the background feature and the object feature are input into a joint multi-head attention weighting module for priority division, and the features after priority division are obtained by multi-head attention mechanism calculation on these features. Then, the divided features are screened according to a dynamic feature screening threshold to obtain fusion features. In the divided features, the feature elements greater than the feature screening threshold are retained for channel transmission, and the feature elements less than or equal to the feature screening threshold are screened out. Finally, the screened feature elements form corresponding fusion features for information transmission. The feature screening threshold is obtained by mapping the signal-to-noise ratio of the transmission channel.

[0066] As shown in FIG. 1, Figure 2 As shown, the divided features are input into a feature screening fusion module, and the divided features are screened in the feature screening fusion module. However, the feature screening threshold is set according to the signal-to-noise ratio of the current channel transmission. The feature screening threshold is obtained by mapping the signal-to-noise ratio of the transmission channel.

[0067] Specifically, the signal-to-noise ratio of the current channel transmission is obtained first, and then the sigmoid function is used to map the signal-to-noise ratio to the feature screening threshold, so as to dynamically adjust the feature screening threshold according to the signal-to-noise ratio. The divided features are screened by the feature screening threshold to obtain fusion features, thereby realizing the control of the information amount of channel transmission. It should be noted that the mapping of the signal-to-noise ratio and the feature screening threshold is in accordance with a negative correlation relationship, that is, if the quality of the current transmission channel is worse, the signal-to-noise ratio is lower, and the feature screening threshold obtained by mapping is higher, so that the information amount of channel transmission can be reduced by screening. If the quality of the current channel is better, the signal-to-noise ratio is higher, and the feature screening threshold obtained by mapping is lower, so that more information amount can be transmitted.

[0068] In the embodiment of the present application, the feature screening threshold is determined by the signal-to-noise ratio of channel transmission, and the divided features are reasonably screened, so that the information amount of channel transmission can be reasonably controlled, and the bandwidth resources of channel transmission can be utilized to the maximum extent.

[0069] In step 103, the priority weight corresponding to the fusion feature is mapped to a probability distribution, and the fusion feature is arithmetically encoded and arithmetically decoded according to the probability distribution to obtain corresponding video semantic features.

[0070] After information transmission is completed in step 102, video reconstruction is performed based on the fused features obtained after transmission. The priority weights corresponding to the fused features are mapped as a probability distribution. This mapping can be calculated using the sigmoid function. These priority weights indicate the key features in the video to be transmitted, namely, features that are more relevant to the action and background objects. Because background objects may remain unchanged across different video frames, but some actions may change, video reconstruction requires a greater focus on the correlation between action features and video frame features, which helps improve the quality of video reconstruction.

[0071] Priority weights are represented by a probability distribution, where probability values ​​characterize the different degrees of correlation between different actions and background objects. Next, arithmetic encoding and decoding are performed on the fused features based on the probability distribution to obtain the corresponding video semantic features.

[0072] like Figure 2 As shown, during dynamic feature transmission, a traditional video semantic transmission method is used. The probability values ​​of the probability distribution and the fused features are input together into an arithmetic encoder for arithmetic coding. Arithmetic coding is a type of joint entropy coding, which can compress video image features. The encoded result is input into the channel and then into the subsequent arithmetic decoder. Through arithmetic decoding, the arithmetic coding result is mapped back to the corresponding feature sequence, thereby recovering the corresponding video semantic features. This enables effective compression of feature data.

[0073] Step 104: Perform semantic decoding on the video semantic features, and reconstruct the decoded image group to obtain the transmitted video.

[0074] After obtaining the video semantic features through step 103, the video semantic features are then semantically decoded, and the decoded image groups are reconstructed to obtain the transmitted video. For example... Figure 2 As shown, semantic decoding can be implemented using a semantic decoder, which can be a generator network model. This model restores the semantic feature sequence of the video to the corresponding image. Finally, in the video reconstruction layer, the restored multiple images are reconstructed into the corresponding video, which is then transmitted as the video to be transmitted. This process of semantic transmission of video is thus realized.

[0075] In the embodiment of the present application, the action vector feature, the background feature and the object feature and other video features are obtained by performing multi-task based feature extraction on the image group of the video to be transmitted, and then the video features are prioritized, and then the features after prioritization are filtered through the signal-to-noise ratio to obtain the fusion features for video reconstruction. Thus, the video features are prioritized during video transmission, and the prioritized features are filtered under the condition of signal-to-noise ratio, so that the correlation between action and background object is paid more attention to during video transmission, thereby improving the transmission effect of video key information and the reconstruction quality, and further improving the user experience. In addition, the feature filtering threshold is obtained by negative correlation mapping according to the signal-to-noise ratio of the transmission channel, and the bandwidth resources during transmission channel can be fully utilized.

[0076] In order to further improve the transmission quality and reconstruction accuracy of the video, a video semantic transmission model is designed in the embodiment of the present application, and the video after transmission is constructed by calling the video semantic transmission model to perform semantic transmission on the video to be transmitted. An end-to-end training method is used to train the video semantic transmission model, and the training process of the video semantic transmission model is introduced below.

[0077] First, the video samples are obtained by collecting a large amount of video resources as samples for model training. Then, the video samples are input into the video semantic transmission model for forward propagation to obtain the reconstructed video. During the forward transmission process after the video samples are input into the video semantic transmission model, the video semantic transmission model performs a series of processing processes such as feature extraction, feature filtering, arithmetic encoding, arithmetic decoding and the like on the video samples as the video to be transmitted, and finally obtains the reconstructed video as the video after transmission. These processing processes can refer to steps 101 to 105 in the above Figure 1 , which will not be described here.

[0078] Further, the action prediction label of the video sample is determined during the forward propagation, because during the feature extraction process of the forward propagation, a multi-task needs to be performed, in which the action classification task needs to classify the action features extracted by the action feature extraction task to realize the category prediction of the corresponding action. During this period, the corresponding classification prediction probability will be output, according to which the prediction label of the action of the video sample by the video semantic transmission model can be determined, such as 0 or 1, 0 representing prediction error and 1 representing prediction correct.

[0079] Next, based on the reconstructed video and the action prediction label, a final loss function of the video semantic transmission model is constructed. The final loss function of the video semantic transmission model is composed of two parts, the first part is the action classification loss, and the other part is the video reconstruction loss. The action classification loss is used to train the video semantic transmission model to improve the accuracy of identifying the action in the video, and the video reconstruction loss is used to train the video semantic transmission model to improve the accuracy of reconstructing the video according to the video semantic features. The process of constructing the final loss function is described below.

[0080] The video sample carries a real classification label of the action category, and the real classification label labels which category the action in the video sample belongs to, so as to provide the model for classification learning. When constructing the loss function, a cross-entropy loss function is constructed according to the real classification label and the action prediction label, which is used as the action classification loss function of the video semantic transmission model, denoted as , which is expressed as formula (1) as follows:

[0081] (1)

[0082] In the above formula (1), N represents the total number of video samples, C represents the number of categories of actions in the video sample, represents the real classification label of the i-th video sample in the category c, and takes the value of 0 or 1, represents the prediction probability of the i-th video sample predicted by the video semantic transmission model as the category c.

[0083] When constructing the video reconstruction loss, a mean square error loss function is constructed according to the real video frame in the video sample and the reconstructed video frame in the reconstructed video, which is used as the video reconstruction loss function of the video semantic transmission model, denoted as , which is expressed as formula (2) as follows:

[0084] (2)

[0085] In the above formula (2), represents the real video frame of the i-th video sample, represents the reconstructed video frame of the i-th reconstructed video, and N represents the total number of video samples.

[0086] In order to balance the importance of action classification and video reconstruction in the video semantic transmission model, the action classification loss weight and the video reconstruction loss weight are preset in the embodiment of the application. When setting the two weights, the different combinations of weight coefficients can be systematically explored by grid search. The grid search automatically tries different parameter combinations and evaluates the performance of each combination, and finally selects the best weight parameter combination , ).

[0087] Next, we will use action classification to determine the loss weights. Action classification loss function We perform weighted analysis to obtain the action classification weighted loss, and then reconstruct the loss weights from the video. Video reconstruction loss function Weighting is applied to obtain the video reconstruction weighted loss. Finally, the action classification weighted loss and the video reconstruction weighted loss are summed to obtain the final loss function of the video semantic transmission model. It is expressed as the following formula (3):

[0088] (3)

[0089] During training, the parameters of the video semantic transmission model are updated by backpropagating through the final loss function.

[0090] Here, training is performed according to a preset number of iterations. In each training iteration, the input video samples are forward-propagated through the video semantic transmission model to calculate the final loss value. Then, the final loss value is back-propagated through the video semantic transmission model. During back-propagation, the gradient of the video semantic transmission model is calculated and optimized using a gradient optimization algorithm, thereby updating the parameters of the video semantic transmission model. When the final loss value begins to converge or the preset number of training iterations is reached in a certain training iteration, the training process ends. The trained video semantic transmission model can then perform video semantic transmission based on the input video to be transmitted, and finally output the reconstructed video as the result of video semantic transmission.

[0091] In some embodiments, during the training of the video semantic transmission model, situations may arise where the final loss function fails to converge, gradients vanish, or gradients explode, increasing the training difficulty of the model. Therefore, this embodiment of the invention proposes a two-stage training strategy, dividing the video semantic transmission model into a video feature extraction model and a video transmission reconstruction model. The video feature extraction model is specifically designed to extract features from video samples, obtaining corresponding action vector features, background features, and object features. The video transmission reconstruction model, on the other hand, is specifically designed to perform feature filtering, arithmetic encoding, and subsequent video reconstruction based on the action vector features, background features, and object features.

[0092] According to the two-stage training strategy, in the first stage, the video feature extraction model is first specially pre-trained through part of the video samples, so as to ensure that the video semantic transmission model has excellent feature extraction capability, and in the training process, the parameters of the video transmission reconstruction model are fixed. In the second stage, end-to-end training is performed, and the video feature extraction model and the video transmission reconstruction model are trained through the remaining part of the video samples. In this way, not only can the final loss function convergence speed of the second stage end-to-end training be accelerated, but also the risk of gradient explosion or disappearance can be reduced, thereby reducing the training difficulty of the model.

[0093] The video semantic transmission device based on priority division provided by the application is described below. The video semantic transmission device based on priority division described below can be correspondingly referred to the video semantic transmission method based on priority division described above.

[0094] As shown in Figure 4 The video semantic transmission device based on priority division includes a video feature extraction module 401, a feature priority division module 402, a feature dynamic transmission module 403, and a video feature reconstruction module 404. Specifically, the video feature extraction module 401 is configured to obtain an image group corresponding to a to-be-transmitted video, and perform multi-task-based feature extraction on the image group to obtain action vector features, background features, and object features of the to-be-transmitted video. The feature priority division module 402 is configured to divide the action vector features, the background features, and the object features according to feature priority, and filter the divided features according to a dynamic feature filtering threshold to obtain fused features. The feature filtering threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature filtering threshold follows a negative correlation relationship. The feature dynamic transmission module 403 is configured to map a priority weight corresponding to the fused features to a probability distribution, perform arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution, and obtain corresponding video semantic features. The priority weight is used to represent the correlation degree of actions and background objects in the to-be-transmitted video. The video feature reconstruction module 404 is configured to perform semantic decoding on the video semantic features, reconstruct the decoded image group, and obtain a transmitted video.

[0095] It should be noted that the beneficial effects of the video semantic transmission device based on priority division described herein can correspond to the beneficial effects of the video semantic transmission method based on priority division described above. Therefore, the beneficial effects of the video semantic transmission device based on priority division will not be described again.

[0096] Figure 5 An example of an entity structure diagram of an electronic device is shown in Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute a video semantic transmission method based on priority division, which includes: acquiring an image group corresponding to a to-be-transmitted video, and performing multi-task-based feature extraction on the image group to obtain action vector features, background features, and object features of the to-be-transmitted video; performing feature priority division on the action vector features, the background features, and the object features, and screening the divided features according to a dynamic feature screening threshold to obtain fused features, wherein the feature screening threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping of the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship;

[0097] mapping a priority weight corresponding to the fused features into a probability distribution, performing arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution to obtain corresponding video semantic features, wherein the priority weight is used to represent a correlation degree of actions and background objects in the to-be-transmitted video; performing semantic decoding processing on the video semantic features, and reconstructing the decoded image group to obtain a transmitted video.

[0098] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0099] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the priority-based video semantic transmission method provided by the above-mentioned methods, which comprises: obtaining an image group corresponding to a to-be-transmitted video, and performing multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the to-be-transmitted video; performing feature priority division on the action vector features, the background features and the object features, and screening the divided features according to a dynamic feature screening threshold to obtain fused features, wherein the feature screening threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship; mapping a priority weight corresponding to the fused features into a probability distribution, performing arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution to obtain corresponding video semantic features, wherein the priority weight is used to represent the correlation degree of actions and background objects in the to-be-transmitted video; performing semantic decoding processing on the video semantic features, and reconstructing the decoded image group to obtain a transmitted video.

[0100] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the priority-based video semantic transmission method provided by the above-mentioned methods, which comprises: obtaining an image group corresponding to a to-be-transmitted video, and performing multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the to-be-transmitted video; performing feature priority division on the action vector features, the background features and the object features, and screening the divided features according to a dynamic feature screening threshold to obtain fused features, wherein the feature screening threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship; mapping a priority weight corresponding to the fused features into a probability distribution, performing arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution to obtain corresponding video semantic features, wherein the priority weight is used to represent the correlation degree of actions and background objects in the to-be-transmitted video; performing semantic decoding processing on the video semantic features, and reconstructing the decoded image group to obtain a transmitted video.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0103] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for video semantic transmission based on priority division, characterized in that, The method comprises: acquiring an image group corresponding to a to-be-transmitted video, and performing multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the to-be-transmitted video; performing feature priority division on the action vector features, the background features and the object features, and performing screening on the divided features according to a dynamic feature screening threshold to obtain fused features, wherein the feature screening threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship; mapping a priority weight corresponding to the fused features into a probability distribution, performing arithmetic encoding and arithmetic decoding on the fused features according to the probability distribution to obtain corresponding video semantic features, wherein the priority weight is used to represent the correlation degree of actions and background objects in the to-be-transmitted video; performing semantic decoding processing on the video semantic features, and reconstructing the decoded image group to obtain a transmitted video; the transmitted video is obtained by calling a video semantic transmission model to perform semantic transmission on the to-be-transmitted video, and a training method of the video semantic transmission model comprises: acquiring a video sample; inputting the video sample into the video semantic transmission model for forward propagation to obtain a reconstructed video, and determining an action prediction label of the video sample during the forward propagation; constructing a final loss function of the video semantic transmission model based on the reconstructed video and the action prediction label; updating parameters of the video semantic transmission model through backward propagation of the final loss function in the video semantic transmission model; the video sample carries a real classification label of an action category, and the final loss function of the video semantic transmission model is constructed based on the reconstructed video and the action prediction label, comprising: constructing a cross-entropy loss function according to the real classification label and the action prediction label as an action classification loss function of the video semantic transmission model; constructing a mean square error loss function according to real video frames in the video sample and reconstructed video frames in the reconstructed video as a video reconstruction loss function of the video semantic transmission model; presetting an action classification loss weight and a video reconstruction loss weight, weighting the action classification loss function by the action classification loss weight to obtain an action classification weighted loss, and weighting the video reconstruction loss function by the video reconstruction loss weight to obtain a video reconstruction weighted loss; summing the action classification weighted loss and the video reconstruction weighted loss to obtain the final loss function of the video semantic transmission model. 2.The method of claim 1, wherein, the multi-task-based feature extraction on the image group to obtain the action vector features, the background features and the object features of the to-be-transmitted video comprises: acquiring a feature extraction model trained based on multi-task, wherein the multi-task selected during training of the feature extraction model comprises an action feature extraction task, an action classification task and a background extraction task; Call the feature extraction model to perform feature extraction on each frame of image in the image group, to obtain action vector features, background features and object features of the video to be transmitted. 3.The priority-based video semantic transmission method of claim 1, wherein, The feature priority division on the action vector features, the background features and the object features comprises: The background features and the background features are combined into a background object feature sequence; Multi-head attention calculation is performed on the action vector features and the background object feature sequence to obtain a group of attention scores corresponding to the background object feature sequence; The attention scores are used as assignment weights to perform weighted summation on feature elements in the background object feature sequence, and the result of the weighted summation is fused with the action vector features.

4. The method of claim 3, wherein, The multi-head attention calculation on the action vector features and the background object feature sequence to obtain a group of attention scores corresponding to the background object feature sequence comprises: The sequence composed of the action vector features is used as a query vector, and the background object feature sequence is used as a key vector and a value vector respectively; The query vector, the key vector and the value vector are input into a plurality of attention heads for attention calculation, and a group of attention scores are output, which are attention calculation values between each feature element in the action vector features and feature elements in the background object feature sequence.

5. A video semantic transmission apparatus based on priority division, characterized in that, The device comprises: A video feature extraction module is configured to acquire an image group corresponding to a video to be transmitted, and perform multi-task-based feature extraction on the image group to obtain action vector features, background features and object features of the video to be transmitted. A feature priority division module is configured to perform feature priority division on the action vector features, the background features and the object features, and perform filtering on the divided features according to a dynamic feature filtering threshold to obtain fusion features, wherein the feature filtering threshold is obtained according to a signal-to-noise ratio of a transmission channel, and the mapping between the signal-to-noise ratio and the feature filtering threshold complies with a negative correlation relationship. A feature dynamic transmission module is configured to map a priority weight corresponding to the fusion features into a probability distribution, perform arithmetic encoding and arithmetic decoding on the fusion features according to the probability distribution, and obtain corresponding video semantic features, wherein the priority weight is used to represent a correlation degree of actions and background objects in the video to be transmitted. A video feature reconstruction module is configured to perform semantic decoding processing on the video semantic features, and reconstruct the decoded image group to obtain a transmitted video. The transmitted video is constructed after semantic transmission of the video to be transmitted by calling a video semantic transmission model, and a training method of the video semantic transmission model comprises: Acquiring a video sample; Inputting the video sample into the video semantic transmission model for forward propagation to obtain a reconstructed video, and determining an action prediction label of the video sample during the forward propagation; Constructing a final loss function of the video semantic transmission model based on the reconstructed video and the action prediction label; The final loss function is used for back propagation in the video semantic transmission model to update parameters of the video semantic transmission model; The video samples carry real classification labels of action categories, and the final loss function of the video semantic transmission model is constructed based on the reconstructed video and the action prediction labels, including: A cross-entropy loss function is constructed according to the real classification labels and the action prediction labels, as an action classification loss function of the video semantic transmission model; A mean square error loss function is constructed according to real video frames in the video samples and reconstructed video frames in the reconstructed video, as a video reconstruction loss function of the video semantic transmission model; A preset action classification loss weight and a video reconstruction loss weight are used to weight the action classification loss function to obtain an action classification weighted loss, and to weight the video reconstruction loss function to obtain a video reconstruction weighted loss; The action classification weighted loss and the video reconstruction weighted loss are summed to obtain the final loss function of the video semantic transmission model.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the video semantic transmission method based on priority division according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the video semantic transmission method based on priority division according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the video semantic transmission method based on priority division according to any one of claims 1 to 4.

Images