Video semantic transmission method based on priority division

Abstract

The invention provides a video semantic transmission method based on priority division, and the method comprises the steps: obtaining an image group corresponding to a to-be-transmitted video, carrying out the feature extraction based on multiple tasks, and obtaining the motion vector feature, background feature and object feature of the to-be-transmitted video; next, feature priority division is carried out, the divided features are screened according to a dynamic feature screening threshold to obtain fusion features, priority weights corresponding to the fusion features are subjected to mapping processing to obtain probability distribution, arithmetic coding and arithmetic decoding are carried out on the fusion features according to the probability distribution to obtain corresponding video semantic features; and finally, performing semantic decoding processing, and reconstructing an image group obtained by decoding to obtain a transmitted video. Through the method and the device, the technical problems of loss of important data feature information and low quality of subsequent video recovery and reconstruction due to adoption of segmentation transmission of information source information during semantic communication in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of video semantic processing, and in particular to a video semantic transmission method based on priority division. Background Art

[0002] Semantic communication systems aim to reconstruct the original information or perform goal-oriented intelligent tasks at the receiver. Semantic communication transmits more relevant data within a limited bandwidth, thereby improving transmission efficiency and robustness. Semantic communication no longer focuses on the accurate transmission of symbols, but on the accuracy of the transmission of the semantic information behind the bits.

[0003] In the prior art, the semantic communication system for video sources focuses on the compressed transmission of source information. In order to save channel bandwidth and cope with complex channel conditions, the semantic communication system will segment and transmit the source content at the sending end. For example, the video recovery method based on key points only encodes and transmits the key point information in real time, while the background and other non-critical information are shared at the beginning. Another example is the direct segmentation of the characters and background in a group of frame pictures. A group of pictures only transmits the background information once, and transmits all the character information. Although these methods have achieved the purpose of saving channel bandwidth, the background, objects and motion features will maintain a certain continuity in adjacent frames. Ignoring these features will affect visual consistency, and the data features themselves are not considered to be divided. The correlation between the semantic features in the background information and the features of the moving objects is not considered, but the classes to which the features belong (such as background, objects, etc.) are divided as a whole. In this way, on the one hand, important data feature information will be lost, and on the other hand, the channel bandwidth resources are not fully utilized, which affects the recovery of moving objects, the reconstruction quality of the video and the user experience. Summary of the invention

[0004] The present invention provides a video semantic transmission method based on priority division, which is used to solve the technical problems in the prior art of losing important data feature information and low quality of subsequent video recovery and reconstruction when segmented transmission of source information is used in semantic communication.

[0005] The present invention provides a video semantic transmission method based on priority division, the method comprising the following steps: Acquire an image group corresponding to the video to be transmitted, and perform multi-task-based feature extraction on the image group to obtain motion vector features, background features, and object features of the video to be transmitted; The motion vector feature, the background feature and the object feature are divided into feature priorities, and the divided features are screened according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to a signal-to-noise ratio mapping of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation relationship; Mapping the priority weight corresponding to the fusion feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fusion feature according to the probability distribution, and obtaining a corresponding video semantic feature, wherein the priority weight is used to characterize the degree of correlation between the action and the background object in the video to be transmitted; The video semantic features are semantically decoded, and the decoded image group is reconstructed to obtain a transmitted video.

[0006] In some embodiments, the performing multi-task based feature extraction on the image group to obtain the motion vector features, background features, and object features of the video to be transmitted includes: Acquire a feature extraction model based on multi-task training, wherein the multi-tasks selected during the feature extraction model training include an action feature extraction task, an action classification task, and a background extraction task; The feature extraction model is called to perform feature extraction on each frame of the image group to obtain motion vector features, background features, and object features of the video to be transmitted.

[0007] In some embodiments, the step of performing feature priority classification on the motion vector feature, the background feature, and the object feature includes: Combining the background features and the background features into a background object feature sequence; Performing multi-head attention calculation on the motion vector feature and the background object feature sequence to obtain a set of attention scores corresponding to the background object feature sequence; The attention score is used as the distribution weight to perform weighted summation on the feature elements in the background object feature sequence, and the result of the weighted summation is fused with the action vector feature.

[0008] In some embodiments, performing multi-head attention calculation on the motion vector feature and the background object feature sequence to obtain a set of attention scores corresponding to the background object feature sequence includes: Using the sequence composed of the action vector features as a query vector, and using the background object feature sequence as a key vector and a value vector respectively; The query vector, the key vector and the value vector are input into multiple attention heads for attention calculation, and a set of attention scores are output, where the attention scores are the attention calculation values ​​between each feature element in the action vector feature and the feature element in the background object feature sequence.

[0009] In some embodiments, the transmitted video is constructed by calling a video semantic transmission model to perform semantic transmission on the video to be transmitted, and the training method of the video semantic transmission model includes: Get video samples; Inputting the video sample into the video semantic transfer model for forward propagation to obtain a reconstructed video, and determining the action prediction label of the video sample during forward propagation; Constructing a final loss function of the video semantic transfer model based on the reconstructed video and the action prediction label; The final loss function is used to perform back propagation in the video semantic transfer model to update the parameters of the video semantic transfer model.

[0010] In some embodiments, the video sample carries a real classification label of an action category, and the final loss function of the video semantic transfer model is constructed based on the reconstructed video and the action prediction label, including: Constructing a cross entropy loss function according to the true classification label and the action prediction label as the action classification loss function of the video semantic transfer model; According to the real video frames in the video sample and the reconstructed video frames in the reconstructed video, a mean square error loss function is constructed as a video reconstruction loss function of the video semantic transfer model; Preset an action classification loss weight and a video reconstruction loss weight, and weight the action classification loss function by the action classification loss weight to obtain an action classification weighted loss, and weight the video reconstruction loss function by the video reconstruction loss weight to obtain a video reconstruction weighted loss; The action classification weighted loss and the video reconstruction weighted loss are summed to obtain a final loss function of the video semantic transfer model.

[0011] The present invention also provides a video semantic transmission device based on priority division, the device comprising the following modules: A video feature extraction module is used to obtain an image group corresponding to the video to be transmitted, and perform multi-task feature extraction on the image group to obtain motion vector features, background features, and object features of the video to be transmitted; A feature priority division module, used for performing feature priority division on the motion vector feature, the background feature and the object feature, and screening the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to a signal-to-noise ratio mapping of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship; A feature dynamic transmission module, used for mapping the priority weight corresponding to the fused feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fused feature according to the probability distribution, and obtaining a corresponding video semantic feature, wherein the priority weight is used to characterize the correlation between the action and the background object in the video to be transmitted; The video feature reconstruction module is used to perform semantic decoding processing on the video semantic features and reconstruct the decoded image group to obtain the transmitted video.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video semantic transmission method based on priority division as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video semantic transmission method based on priority division as described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned priority-based video semantic transmission methods.

[0015] The video semantic transmission method based on priority division provided by the present invention obtains video features such as motion vector features, background features and object features by performing multi-task feature extraction on the image group of the video to be transmitted, and then prioritizes these video features themselves, and then screens the divided features by signal-to-noise ratio to obtain fusion features for video reconstruction. In this way, the video features themselves are prioritized during video transmission, and the divided features can be screened under the condition of signal-to-noise ratio, so that the correlation between action and background objects is paid more attention during video transmission, thereby improving the transmission effect and reconstruction quality of key video information, and further improving user experience. In addition, the feature screening threshold is obtained by negative correlation mapping based on the signal-to-noise ratio of the transmission channel, which can make full use of the bandwidth resources of the transmission channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the video semantic transmission method based on priority division provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the framework of the video semantic transmission method based on priority division provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the feature screening process provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of a video semantic transmission device based on priority division provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The video semantic transmission method based on priority division of the present invention can be applied to electronic devices that can play or transmit videos, such as servers or terminals. Therefore, the execution subject of the method is a terminal or a server. When the electronic device performs a video transmission task, the image group corresponding to the video to be transmitted is obtained through the method, and the transmitted video is finally obtained, which is used to complete the video transmission task of the electronic device.

[0024] The following describes the video semantic transmission method and device based on priority division of the present invention in conjunction with the accompanying drawings. Figure 1 is a flow chart of the video semantic transmission method based on priority division provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0025] Step 101: Obtain an image group corresponding to the video to be transmitted, and perform multi-task based feature extraction on the image group to obtain motion vector features, background features, and object features of the video to be transmitted.

[0026] First, in an application scenario where video semantic transmission is required, a video to be transmitted is obtained, and then the video to be transmitted is divided into frames of images as corresponding image groups.

[0027] During transmission, it is necessary to perform multi-task based feature extraction on the image group to obtain the motion vector features, background features and object features of the video to be transmitted.

[0028] Here, the embodiment of the present invention pre-trains a feature extraction model, and this feature extraction model is based on multi-task training, that is, a multi-task learning training method is adopted during training. Among them, the multi-tasks selected during feature extraction model training include action feature extraction tasks, action classification tasks, and background extraction tasks. The feature extraction model can be a shared feature extractor based on a 3D-CNN network.

[0029] In the multi-task learning process, the shared video features required for several tasks are first extracted through the shared feature extractor, and then the shared video features are input into three branches respectively for performing multiple tasks. The three branches are action extraction branch, action classification branch and background extraction branch, which are used to perform action feature extraction task, action classification task and background extraction task respectively.

[0030] In the process of multi-task learning, the performance of the model on each task can be improved by learning multiple related tasks. It can also share parameters while executing multiple tasks, learn the parameters of all tasks, achieve joint optimization of parameters, and improve the feature extraction ability of the model.

[0031] Therefore, when extracting features from an image group, first obtain a feature extraction model based on multi-task training, and then call the feature extraction model to extract features from each frame of the image group to obtain the motion vector features, background features, and object features of the video to be transmitted.

[0032] Here, each frame of the image group is input into the feature extraction model in turn, and the model directly outputs the motion vector features, background features and object features corresponding to the image frame. After all images are extracted, the motion vector features, background features and object features of the video to be transmitted are obtained.

[0033] like Figure 2As shown in the figure, when performing video feature extraction, a group of frame images (i.e., the image group mentioned above) are input into the shared feature extractor of the shared feature layer, and then enter the action extraction branch, action classification branch, and background extraction branch respectively for feature extraction, and finally output action vector features, background features, and object features.

[0034] In the embodiment of the present invention, when performing video feature extraction, a feature extraction model is trained through a multi-task learning method. During training, the feature extraction model can improve the performance on each task by learning multiple related tasks, and can also share parameters while executing multiple tasks, learn the parameters of all tasks, achieve joint optimization of parameters, and enhance the feature extraction effect of the model.

[0035] Step 102: perform feature priority classification on the motion vector features, background features, and object features, and filter the classified features according to a dynamic feature filtering threshold to obtain fused features.

[0036] After feature extraction is performed in step 101, the action vector features, background features, and object features are then prioritized. The purpose of priority division is to focus on the correlation between the action and the background objects, because the more relevant the action and the background objects are, the more important the key information belonging to the video transmission is, and the higher the priority. These key information determines the quality of subsequent video reconstruction. The correlation between the action and the background objects is calculated by the embodiment of the present invention through the joint multi-head attention mechanism, thereby realizing feature priority division.

[0037] like Figure 3 As shown in the figure, the physical features of the background features will form the corresponding background physical feature sequence, and then be input into the multi-head attention module together with the action features for attention calculation to realize the priority division of the video features, and then the divided features will be screened to finally obtain the fusion features, which will be explained in detail below.

[0038] First, the background features and background features are combined into a background object feature sequence, and then multi-head attention calculation is performed on the action vector features and the background object feature sequence to obtain a set of attention scores corresponding to the background object feature sequence. Here, the attention score is used to measure the correlation between the action and the background object. The higher the attention score, the higher the correlation between the action and the background object, and vice versa.

[0039] Specifically, the sequence composed of action vector features is used as the query vector (Query), and the background object feature sequence is used as the key vector (Key) and the value vector (Value). The joint multi-head attention mechanism is implemented through multiple attention heads. The query vector, key vector and value vector are input into multiple attention heads for attention calculation, and the attention score corresponding to each attention head is output. The attention score is the attention calculation value between each feature element in the action vector feature and each feature element in the background object feature sequence.

[0040] Here, when the attention head is calculated, the attention calculation value between each feature element in the motion vector feature and all feature elements in the background object feature sequence is calculated to obtain a set of attention calculation values. For example, if there are 10 feature elements in the motion vector feature, 10 attention calculation values ​​will be calculated. Then the set of attention calculation values ​​calculated by each attention head are averaged to obtain a set of average attention scores. For example, if there are 3 attention heads, 3 sets of attention calculation values ​​are obtained, and then the corresponding 3 attention calculation values ​​are averaged to obtain 10 average attention calculation values.

[0041] Finally, the attention score is used as the assigned weight to perform weighted summation on the feature elements in the background object feature sequence, and the result of the weighted summation is fused with the action vector feature. The attention score can be used as a set of assigned weight values ​​to indicate the degree of correlation between the action feature elements and the background object feature elements. The background object feature elements are then weighted separately by the assigned weight values, and the weighted results are summed, and finally the background object features obtained by the summation are fused with the action vector features.

[0042] like Figure 2 As shown, the motion vector features, background features, and object features are input into the joint multi-head attention weighted module for priority division. The features after feature priority division are obtained by calculating the multi-head attention mechanism for these features. Then, the divided features are screened according to the dynamic feature screening threshold to obtain the fused features. Among the divided features, the feature elements greater than the feature screening threshold are retained for channel transmission, and those less than or equal to the feature screening threshold are screened out. The feature elements finally screened constitute the corresponding fused features for information transmission. Among them, the feature screening threshold is obtained according to the signal-to-noise ratio mapping of the transmission channel.

[0043] Here Figure 2 As shown, the divided features will be input into the feature screening and fusion module, where the divided features will be screened. However, the feature screening threshold will be set according to the signal-to-noise ratio of the current channel transmission. The feature screening threshold is obtained by mapping the signal-to-noise ratio of the transmission channel.

[0044] Specifically, the signal-to-noise ratio of the current channel transmission is first obtained, and then the signal-to-noise ratio is mapped to the feature screening threshold using the sigmoid function, so as to dynamically adjust the feature screening threshold according to the signal-to-noise ratio. The divided features are screened by the feature screening threshold to obtain fusion features, thereby controlling the amount of information transmitted by the channel. It should be noted that the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation, that is, if the quality of the current transmission channel is worse, the signal-to-noise ratio is lower, and the mapped feature screening threshold is higher, so that the amount of information transmitted by the channel can be reduced through screening. When the quality of the current channel is better, the signal-to-noise ratio is higher, and the mapped feature screening threshold is lower, so that more information can be transmitted.

[0045] The embodiment of the present invention determines the feature screening threshold by the signal-to-noise ratio of the channel transmission, and reasonably screens the divided features, which can reasonably control the amount of information transmitted by the channel and ensure that the bandwidth resources of the channel transmission are utilized to the maximum extent.

[0046] Step 103: Map the priority weights corresponding to the fusion features into probability distributions, perform arithmetic coding and arithmetic decoding on the fusion features according to the probability distributions, and obtain corresponding video semantic features.

[0047] After completing the information transmission through step 102, the video is reconstructed according to the fusion features obtained after the transmission, and the priority weight mapping corresponding to the fusion features is processed into a probability distribution. The mapping process can be calculated by the sigmoid function. This priority weight can indicate the key features in the video to be transmitted, that is, the features that are more related to the action and background objects. Because the background objects may be unchanged in different video frames, but some actions may change in different video frames. Therefore, when reconstructing the video, it is actually necessary to pay more attention to the correlation between the action features and the video frame features, which helps to improve the quality of video reconstruction.

[0048] The priority weight is reflected by probability distribution, in which the probability values ​​represent the different correlations between different actions and background objects. Next, the fused features are arithmetically encoded and decoded according to the probability distribution to obtain the corresponding video semantic features.

[0049] like Figure 2As shown in the figure, when the features are dynamically transmitted, the traditional video semantic transmission method is used to input the probability value of the probability distribution together with the fusion features into the arithmetic encoder for arithmetic coding. Arithmetic coding is a joint entropy coding that can achieve the compression of video image features. The encoded result will be input into the channel and then into the subsequent arithmetic decoder. The arithmetic coding result will be mapped back to the corresponding feature sequence through arithmetic decoding, thereby restoring the corresponding video semantic features, thereby achieving effective compression of feature data.

[0050] Step 104: semantically decode the video semantic features, and reconstruct the decoded image group to obtain the transmitted video.

[0051] After the video semantic features are obtained through mapping in step 103, the video semantic features are finally semantically decoded, and the decoded image group is reconstructed to obtain the transmitted video. Figure 2 As shown, the semantic decoding process can be implemented by a semantic decoder. The semantic decoder can be a generator network model, which is equivalent to restoring the video semantic feature sequence into the corresponding image. Finally, the restored multiple images are reconstructed into the corresponding video at the reconstructed video layer as the video after the video is transmitted, thereby realizing the whole process of semantic transmission of the video.

[0052] In the embodiment of the present invention, a multi-task-based feature extraction is performed on the image group of the video to be transmitted to obtain video features such as motion vector features, background features, and object features, and then the video features themselves are prioritized, and then the divided features are screened by the signal-to-noise ratio to obtain fusion features for video reconstruction. In this way, the video features themselves are prioritized during the video transmission process, and the divided features can be screened under the condition of the signal-to-noise ratio, so that the correlation between the action and the background objects is paid more attention during the video transmission, thereby improving the transmission effect and reconstruction quality of the key information of the video, and further improving the user experience. In addition, the feature screening threshold is obtained by negative correlation mapping according to the signal-to-noise ratio of the transmission channel, which can make full use of the bandwidth resources of the transmission channel.

[0053] In order to further improve the transmission quality and reconstruction accuracy of the video, a video semantic transmission model is designed in the embodiment of the present invention. After the video to be transmitted is semantically transmitted by calling the video semantic transmission model, the transmitted video is constructed. An end-to-end training method is used to train the video semantic transmission model. The training process of the video semantic transmission model is described below.

[0054] First, obtain video samples by collecting a large amount of video resources as samples for model training. Then input the video samples into the video semantic transmission model for forward propagation to obtain the reconstructed video. In the forward transmission process after inputting the video samples into the video semantic transmission model, the video semantic transmission model uses the video samples as the video to be transmitted for a series of processing processes such as feature extraction, feature screening, arithmetic coding, and arithmetic decoding, and finally obtains the reconstructed video as the transmitted video. These processing processes can be referred to Figure 1 Steps 101 to 105 in the above are not described in detail here.

[0055] Furthermore, the action prediction label of the video sample is determined during forward propagation, because multiple tasks need to be performed during the feature extraction process of forward propagation. Among them, the action classification task needs to classify the action features extracted by the action feature extraction task to achieve the category prediction of the corresponding action. During this period, the corresponding classification prediction probability will be output. According to this classification prediction probability, the video semantic transfer model can determine the prediction label of the action in the video sample, such as 0 or 1, 0 represents a wrong prediction, and 1 represents a correct prediction.

[0056] Next, the final loss function of the video semantic transfer model is constructed based on the reconstructed video and the action prediction label. The final loss function of the video semantic transfer model consists of two parts: the first part is the action classification loss, and the other part is the video reconstruction loss. The action classification loss is used to train the video semantic transfer model to recognize the action in the video, and the video reconstruction loss is used to train the video semantic transfer model to reconstruct the video according to the video semantic features. The process of constructing the final loss function is described below.

[0057] The video sample carries the real classification label of the action category. The real classification label marks which category the action in the video sample belongs to for the model to classify and learn. When constructing the loss function, a cross entropy loss function is constructed based on the real classification label and the action prediction label as the action classification loss function of the video semantic transfer model, denoted as , expressed as the following formula (1): (1) In the above formula (1), N represents the total number of video samples, C represents the number of action categories in the video samples, represents the true classification label of the i-th video sample in category c, with a value of 0 or 1. It represents the predicted probability that the video semantic transfer model predicts the i-th video sample as category c.

[0058] When constructing the video reconstruction loss, a mean square error loss function is constructed based on the real video frames in the video sample and the reconstructed video frames in the reconstructed video, which is used as the video reconstruction loss function of the video semantic transfer model, denoted as , expressed as the following formula (2): (2) In the above formula (2), represents the real video frame of the i-th video sample, represents the reconstructed video frame of the i-th reconstructed video, and N represents the total number of video samples.

[0059] In order to balance the importance of action classification and video reconstruction in the video semantic transmission model, the embodiment of the present invention presets the action classification loss weight And the video reconstruction loss weight When setting these two weights, grid search can be used to systematically explore different combinations of weight coefficients. Grid search automatically tries different parameter combinations, evaluates the performance of each combination, and finally selects the best weight parameter combination. , ).

[0060] Next, the action classification loss weight Loss function for action classification Weighted, get the weighted loss of action classification, and reconstruct the loss weight through video Reconstruct the loss function for the video The weighted loss is obtained by weighting the video reconstruction. Finally, the weighted loss of action classification and the weighted loss of video reconstruction are summed to obtain the final loss function of the video semantic transfer model. , expressed as the following formula (3): (3) During training, the final loss function is used to back-propagate in the video semantic transfer model to update the parameters of the video semantic transfer model.

[0061] Here, the training is carried out according to the preset iterative rounds. In each training round, the input video sample is forward propagated in the video semantic transmission model, and the final loss value is calculated. Then, the final loss value is back-propagated in the video semantic transmission model. The gradient of the video semantic transmission model is calculated during the back-propagation process, and the gradient is optimized using the gradient optimization algorithm, thereby updating the parameters of the video semantic transmission model. When the final loss value begins to converge or reaches the preset training round in a certain training round, the training process ends. The trained video semantic transmission model can perform video semantic transmission according to the input video to be transmitted, and finally output the reconstructed video as the result of video semantic transmission.

[0062] In some embodiments, the final loss function may be difficult to converge, the gradient may disappear or explode during the training of the video semantic transmission model, which increases the difficulty of model training. Therefore, the embodiment of the present invention proposes a two-stage training strategy, which divides the video semantic transmission model into a video feature extraction model and a video transmission reconstruction model. The video feature extraction model specifically extracts features from video samples to obtain corresponding motion vector features, background features, and object features, while the video transmission reconstruction model specifically performs feature screening, arithmetic coding, and subsequent video reconstruction based on motion vector features, background features, and object features.

[0063] According to the two-stage training strategy, in the first stage, the video feature extraction model is pre-trained with some video samples to ensure that the video semantic transmission model has excellent feature extraction capabilities. During the training process, the parameters of the video transmission reconstruction model are fixed. In the second stage, end-to-end training is performed, and the video feature extraction model and the video transmission reconstruction model are trained simultaneously with the remaining video samples. In this way, not only can the convergence speed of the final loss function in the second stage of end-to-end training be accelerated, but also the risk of gradient explosion or disappearance can be reduced, thereby reducing the difficulty of model training.

[0064] The following is a description of a video semantic transmission device based on priority division provided by the present invention. The video semantic transmission device based on priority division described below and the video semantic transmission method based on priority division described above can correspond to each other.

[0065] like Figure 4As shown, the video semantic transmission device based on priority division includes a video feature extraction module 401, a feature priority division module 402, a feature dynamic transmission module 403, and a video feature reconstruction module 404. Specifically, the video feature extraction module 401 is used to obtain the image group corresponding to the video to be transmitted, and perform multi-task feature extraction on the image group to obtain the action vector feature, background feature and object feature of the video to be transmitted; the feature priority division module 402 is used to perform feature priority division on the action vector feature, the background feature and the object feature, and filter the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to the signal-to-noise ratio mapping of the transmission channel, and the mapping of the signal-to-noise ratio and the feature screening threshold follows a negative correlation; the feature dynamic transmission module 403 is used to map the priority weight corresponding to the fusion feature into a probability distribution, perform arithmetic coding and arithmetic decoding on the fusion feature according to the probability distribution, and obtain the corresponding video semantic feature, wherein the priority weight is used to characterize the correlation between the action and the background object in the video to be transmitted; the video feature reconstruction module 404 is used to perform semantic decoding on the video semantic feature, and reconstruct the decoded image group to obtain the transmitted video.

[0066] It should be noted that the beneficial effects of the video semantic transmission device based on priority division here and the video semantic transmission method based on priority division mentioned above can correspond to each other, so the beneficial effects of the video semantic transmission device based on priority division will not be repeated here.

[0067] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a video semantic transmission method based on priority division, the method comprising: obtaining an image group corresponding to the video to be transmitted, and performing multi-task-based feature extraction on the image group to obtain the motion vector feature, background feature and object feature of the video to be transmitted; performing feature priority division on the motion vector feature, the background feature and the object feature, and screening the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to the signal-to-noise ratio mapping of the transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation relationship; The priority weight mapping corresponding to the fusion feature is processed into a probability distribution, and the fusion feature is arithmetically encoded and arithmetically decoded according to the probability distribution to obtain the corresponding video semantic feature, wherein the priority weight is used to characterize the degree of correlation between the action and the background object in the video to be transmitted; the video semantic feature is semantically decoded, and the decoded image group is reconstructed to obtain the video after transmission.

[0068] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0069] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video semantic transmission method based on priority division provided by the above methods, the method including: obtaining an image group corresponding to the video to be transmitted, and performing multi-task feature extraction on the image group to obtain the motion vector feature, background feature and object feature of the video to be transmitted; performing feature priority division on the motion vector feature, the background feature and the object feature, and filtering the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to the signal-to-noise ratio mapping of the transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation; mapping the priority weight corresponding to the fusion feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fusion feature according to the probability distribution to obtain the corresponding video semantic feature, wherein the priority weight is used to characterize the degree of correlation between the action and the background object in the video to be transmitted; performing semantic decoding on the video semantic feature, and reconstructing the decoded image group to obtain the transmitted video.

[0070] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the video semantic transmission method based on priority division provided by the above methods, the method comprising: obtaining an image group corresponding to the video to be transmitted, and performing multi-task-based feature extraction on the image group to obtain the motion vector feature, background feature and object feature of the video to be transmitted; performing feature priority division on the motion vector feature, the background feature and the object feature, and filtering the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to the signal-to-noise ratio mapping of the transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation; mapping the priority weight corresponding to the fusion feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fusion feature according to the probability distribution to obtain the corresponding video semantic feature, wherein the priority weight is used to characterize the degree of correlation between the action and the background object in the video to be transmitted; performing semantic decoding on the video semantic feature, and reconstructing the decoded image group to obtain the transmitted video.

[0071] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video semantic transmission method based on priority division, characterized in that: The method comprises: Acquire an image group corresponding to the video to be transmitted, and perform multi-task-based feature extraction on the image group to obtain motion vector features, background features, and object features of the video to be transmitted; The motion vector feature, the background feature and the object feature are divided into feature priorities, and the divided features are screened according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to a signal-to-noise ratio mapping of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold follows a negative correlation relationship; Mapping the priority weight corresponding to the fusion feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fusion feature according to the probability distribution, and obtaining a corresponding video semantic feature, wherein the priority weight is used to characterize the degree of correlation between the action and the background object in the video to be transmitted; The video semantic features are semantically decoded, and the decoded image group is reconstructed to obtain a transmitted video.

2. The video semantic transmission method based on priority division according to claim 1 is characterized in that: The performing multi-task based feature extraction on the image group to obtain motion vector features, background features and object features of the video to be transmitted includes: Acquire a feature extraction model based on multi-task training, wherein the multi-tasks selected during the feature extraction model training include an action feature extraction task, an action classification task, and a background extraction task; The feature extraction model is called to perform feature extraction on each frame of the image group to obtain motion vector features, background features, and object features of the video to be transmitted.

3. The video semantic transmission method based on priority division according to claim 1 is characterized in that: The step of performing feature priority classification on the action vector feature, the background feature, and the object feature includes: Combining the background features and the background features into a background object feature sequence; Performing multi-head attention calculation on the motion vector feature and the background object feature sequence to obtain a set of attention scores corresponding to the background object feature sequence; The attention score is used as the distribution weight to perform weighted summation on the feature elements in the background object feature sequence, and the result of the weighted summation is fused with the action vector feature.

4. The video semantic transmission method based on priority division according to claim 3 is characterized in that: The performing multi-head attention calculation on the motion vector feature and the background object feature sequence to obtain a set of attention scores corresponding to the background object feature sequence includes: Using the sequence composed of the action vector features as a query vector, and using the background object feature sequence as a key vector and a value vector respectively; The query vector, the key vector and the value vector are input into multiple attention heads for attention calculation, and a set of attention scores are output, where the attention scores are the attention calculation values ​​between each feature element in the action vector feature and the feature element in the background object feature sequence.

5. The video semantic transmission method based on priority division according to claim 1 is characterized in that: The transmitted video is constructed by calling a video semantic transmission model to perform semantic transmission on the video to be transmitted, and the training method of the video semantic transmission model includes: Get video samples; Inputting the video sample into the video semantic transfer model for forward propagation to obtain a reconstructed video, and determining the action prediction label of the video sample during forward propagation; Constructing a final loss function of the video semantic transfer model based on the reconstructed video and the action prediction label; The final loss function is used to perform back propagation in the video semantic transfer model to update the parameters of the video semantic transfer model.

6. The video semantic transmission method based on priority division according to claim 5 is characterized in that: The video sample carries a real classification label of the action category, and the final loss function of the video semantic transfer model is constructed based on the reconstructed video and the action prediction label, including: Constructing a cross entropy loss function according to the true classification label and the action prediction label as the action classification loss function of the video semantic transfer model; According to the real video frames in the video sample and the reconstructed video frames in the reconstructed video, a mean square error loss function is constructed as a video reconstruction loss function of the video semantic transfer model; Preset an action classification loss weight and a video reconstruction loss weight, and weight the action classification loss function by the action classification loss weight to obtain an action classification weighted loss, and weight the video reconstruction loss function by the video reconstruction loss weight to obtain a video reconstruction weighted loss; The action classification weighted loss and the video reconstruction weighted loss are summed to obtain a final loss function of the video semantic transfer model.

7. A video semantic transmission device based on priority division, characterized in that: The device comprises: A video feature extraction module is used to obtain an image group corresponding to the video to be transmitted, and perform multi-task feature extraction on the image group to obtain motion vector features, background features, and object features of the video to be transmitted; A feature priority division module, used for performing feature priority division on the motion vector feature, the background feature and the object feature, and screening the divided features according to a dynamic feature screening threshold to obtain a fusion feature, wherein the feature screening threshold is obtained according to a signal-to-noise ratio mapping of a transmission channel, and the mapping between the signal-to-noise ratio and the feature screening threshold complies with a negative correlation relationship; A feature dynamic transmission module, used for mapping the priority weight corresponding to the fused feature into a probability distribution, performing arithmetic coding and arithmetic decoding on the fused feature according to the probability distribution, and obtaining a corresponding video semantic feature, wherein the priority weight is used to characterize the correlation between the action and the background object in the video to be transmitted; The video feature reconstruction module is used to perform semantic decoding processing on the video semantic features and reconstruct the decoded image group to obtain the transmitted video.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for video semantic transmission based on priority division as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for video semantic transmission based on priority division as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for video semantic transmission based on priority division as claimed in any one of claims 1 to 6 is implemented.

Images