Video semantic communication method and device based on target-attribute-relation representation

Abstract

The invention provides a video semantic communication method and device based on target-attribute-relation representation, and relates to the technical field of semantic communication, a video source end obtains OAR features of a key frame by using an OAR feature extraction network; inputting the key frame and the OAR feature of the key frame into a source-channel joint encoder to obtain an encoding result of the key frame; transmitting the coding result of the key frame and the OAR representation diagrams of the T continuous frames through an AWGN channel; the video reconstruction end uses the OAR feature extraction network to obtain the OAR feature of the key frame; inputting the coding result of the key frame, the OAR feature of the key frame and the signal-to-noise ratio into a generator of the key frame to obtain a reconstruction result of the key frame; and inputting the reconstruction result of the key frame and the OAR representation diagram of each non-key frame into a generator of the non-key frame to obtain a reconstruction result of each non-key frame. According to the method, the OAR data and the video transmission information source channel joint codec are used for assistance, and good foreground reconstruction quality can still be kept under the conditions of low bandwidth and signal-to-noise ratio.

Description

Technical Field

[0001] The present application relates to the field of semantic communication technology, and in particular to a method and device for video semantic communication based on target-attribute-relation representation. Background Art

[0002] With the rapid growth of multimedia data, the need for low-bitrate video compression and efficient transmission has become increasingly critical in applications such as virtual reality and future video streaming services, and the demand for efficient video transmission is increasing. Currently, semantic communication is becoming an important technology to ensure efficient and reliable transmission in low-bandwidth and high-noise environments, enhancing the communication performance of specific tasks by focusing on the semantic level of data representation and transmission.

[0003] Most video transmission methods usually use separate joint source-channel coding strategies to ensure pixel and symbol accuracy. However, with the diminishing returns of standard video encoders, the video quality at low bit rates cannot meet the requirements of user perception and specific tasks. In addition, the clear separation of source and channel coding may jeopardize the reliable transmission of video, especially over channels with low signal-to-noise ratio, resulting in increased instances of decoding failures.

[0004] Based on this, a new video semantic communication method is urgently needed. Summary of the invention

[0005] The present application provides a video semantic communication method and device based on target-attribute-relation representation to solve the above-mentioned problems existing in the existing video transmission methods.

[0006] In a first aspect of an embodiment of the present application, a video semantic communication method based on target-attribute-relation representation is proposed, the method comprising: The video source takes the first frame of T consecutive frames in each GOP as a key frame, and uses an OAR feature extraction network to extract OAR features from the OAR representation graph of the key frame to obtain the OAR features of the key frame; The video source end inputs the key frame into a source encoder to obtain an encoding result of the key frame, and inputs the encoding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame; The video source end transmits the source-channel joint coding result of the key frame to the video reconstruction end through the AWGN channel, and losslessly transmits the OAR representation graphs of T consecutive frames through the AWGN channel; The video reconstruction end uses the OAR feature extraction network to extract OAR features from the OAR representation graph of the received key frame to obtain the OAR features of the key frame; The video reconstruction end inputs the received source-channel joint encoding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; The video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

[0007] In an optional embodiment of the present application, the channel encoder includes two branches, each branch including: a first convolution module, an OAR modulation module and an SNR modulation module; Inputting the encoding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame, including: Inputting the encoding result of the key frame into the first convolution module to obtain a first result; Inputting the first result and the OAR feature of the key frame into the OAR modulation module to obtain a first OAR modulation result; The first OAR modulation result and the signal-to-noise ratio are input into the SNR modulation module to obtain a source-channel joint coding result of the key frame.

[0008] In an optional embodiment of the present application, the key frame generator includes two branches, each branch including: a second convolution module, an OAR modulation module and an SNR modulation module; The video reconstruction end inputs the received source-channel joint encoding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame, including: Inputting the received source-channel joint coding result of the key frame into the second convolution module to obtain a second result; Inputting the second result and the OAR feature of the key frame into the OAR modulation module to obtain a second OAR modulation result; The second OAR modulation result and the signal-to-noise ratio are input into the SNR modulation module to obtain a reconstruction result of the key frame.

[0009] In an optional embodiment of the present application, the OAR modulation module includes: a splicing layer and a first branch and a second branch, the first branch includes: at least one convolutional layer and at least one activation layer, and the second branch includes: at least one convolutional layer and at least one activation layer; Inputting the result output by the previous module of the OAR modulation module and the OAR feature of the key frame into the splicing layer to obtain a splicing result; Inputting the splicing results into the first branch and the second branch respectively, to obtain a first OAR result and a second OAR result respectively; Multiplying the first OAR result by the result output by the previous module to obtain an OAR intermediate result; The OAR intermediate result and the second OAR result are added to obtain an OAR modulation result.

[0010] In an optional embodiment of the present application, the SNR modulation module includes: an average layer, a splicing layer, and a third branch and a fourth branch, the third branch includes: at least one linear layer and at least one activation layer, and the fourth branch includes: at least one linear layer and at least one activation layer; Inputting the result output by the previous module of the SNR modulation module into the averaging layer to obtain an average result; Inputting the average result and the signal-to-noise ratio into the splicing layer to obtain an intermediate splicing result; Inputting the intermediate splicing result into the third branch and the fourth branch respectively, to obtain a third channel result and a fourth channel result respectively; Multiplying the result output by the previous module of the SNR modulation module by the third channel result to obtain a channel intermediate result; The channel intermediate result and the fourth channel result are added together to obtain a channel modulation result.

[0011] In an optional embodiment of the present application, the OAR feature extraction network includes: an embedding module, a layout generation module and a convolution module; Using the OAR feature extraction network, OAR features are extracted from the OAR representation graph of the key frame to obtain the OAR features of the key frame, including: Inputting the OAR representation graph of the key frame into the embedding module to obtain the OAR embedding representation of the key frame; Inputting the OAR embedding representation of the key frame into the layout generation module to obtain the OAR layout representation of the key frame, where the OAR layout representation of the key frame is used to represent all objects contained in the key frame; The OAR layout representation of the key frame is input into the convolution module to obtain the OAR feature of the key frame.

[0012] In an optional embodiment of the present application, the generator of the non-key frame includes an OAR embedding and graph calculation module, an OAR layout generation module, an optical flow estimation and frame prediction module, and an image synthesis and fusion module; The video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame, including: Inputting the OAR representation graph of the non-key frame into the OAR embedding and graph calculation module to obtain the OAR embedding representation; The OAR embedding representation is input into the graph convolutional network in the OAR layout generation module to obtain the deep features of each target in the non-key frame; Inputting the deep features of each target in the non-key frame into the layout generation network in the OAR layout generation module to obtain the OAR layout representation of the non-key frame, wherein the OAR layout representation of the non-key frame is used to characterize all targets contained in the non-key frame; Inputting the OAR layout representation of the non-key frame into the layout feature extraction network in the optical flow estimation and frame prediction module to obtain the OAR layout feature of the non-key frame; Inputting the last reconstructed frame into the image feature extraction network in the optical flow estimation and frame prediction module to obtain the image features of the last reconstructed frame; splicing the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame to obtain intermediate features; Inputting the intermediate features into an optical flow encoder in an optical flow estimation network in the optical flow estimation and frame prediction module to obtain an optical flow encoding result; Inputting the optical flow encoding result into an optical flow decoder in an optical flow estimation network in the optical flow estimation and frame prediction module to obtain an estimated optical flow; Obtaining an optical flow estimation result according to the estimated optical flow and the last reconstructed frame; Inputting the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame into the image synthesis network in the image synthesis and fusion module to obtain an image synthesis result; Inputting the optical flow encoding result into the weight estimation layer of the image fusion network in the image synthesis and fusion module to obtain an estimated weight; Obtaining a reconstruction result of a non-key frame according to the optical flow estimation result, the image synthesis result and the estimation weight; When the non-key frame is the first non-key frame, the last reconstructed frame is the reconstruction result of the key frame, and the OAR layout feature of the last frame is the OAR layout feature of the key frame.

[0013] In a second aspect of an embodiment of the present application, a video semantic communication device based on target-attribute-relation representation is proposed, the device comprising: A first key frame feature extraction module is used to take the first frame of T consecutive frames in each GOP as a key frame, and use an OAR feature extraction network to perform OAR feature extraction on an OAR representation graph of the key frame to obtain an OAR feature of the key frame; A joint encoding module, used for inputting the key frame into a source encoder to obtain an encoding result of the key frame, and inputting the encoding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame; A transmission module, configured to transmit the source-channel joint encoding result of the key frame to a video reconstruction end through an AWGN channel, and to losslessly transmit the OAR representation graphs of T consecutive frames through the AWGN channel; A second key frame feature extraction module is used to extract OAR features from the received OAR representation graph of the key frame using an OAR feature extraction network to obtain the OAR features of the key frame; A key frame reconstruction module, used for inputting the received source-channel joint coding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; The non-key frame reconstruction module is used to input the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

[0014] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the video semantic communication method based on target-attribute-relationship representation as described in any one of the first aspects above.

[0015] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the video semantic communication method based on target-attribute-relationship representation as described in any one of the first aspects above is implemented.

[0016] The present application includes the following advantages: the present application provides a method and device for video semantic communication based on target-attribute-relation representation, the video source end uses the first frame of T consecutive frames in each GOP as a key frame, uses the OAR feature extraction network to extract the OAR feature of the OAR representation graph of the key frame, and obtains the OAR feature of the key frame; the video source end inputs the key frame into a source encoder to obtain the encoding result of the key frame, inputs the encoding result of the key frame, the OAR feature of the key frame and the signal-to-noise ratio into a channel encoder to obtain the source-channel joint encoding result of the key frame; the video source end transmits the key frame through an AWGN channel The source channel joint encoding result is given to the video reconstruction end, and the OAR representation map of T consecutive frames is losslessly transmitted through the AWGN channel; the video reconstruction end uses the OAR feature extraction network to extract the OAR feature of the received key frame OAR representation map to obtain the OAR feature of the key frame; the video reconstruction end inputs the received source channel joint encoding result of the key frame, the OAR feature of the key frame and the signal-to-noise ratio into the generator of the key frame to obtain the reconstruction result of the key frame; the video reconstruction end inputs the reconstruction result of the key frame and the OAR representation map of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame. This application utilizes the compactness of OAR data-related downstream tasks, and assists with the source channel joint codec of video transmission, so as to pay more attention to the key objects in the scene. This method can still maintain good foreground reconstruction quality under low bandwidth and signal-to-noise ratio conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a schematic diagram of describing an image using OAR proposed in the related art; Figure 2 It is a schematic diagram of the steps of a video semantic communication method based on target-attribute-relation representation proposed in an embodiment of the present application; Figure 3 It is a flowchart of a video semantic communication method based on target-attribute-relation representation proposed in an embodiment of the present application; Figure 4 It is a schematic diagram of a key frame reconstruction process in a video semantic communication method based on target-attribute-relation representation provided by an embodiment of the present application; Figure 5It is a schematic diagram of a non-key frame reconstruction process in a video semantic communication method based on target-attribute-relation representation provided by an embodiment of the present application; Figure 6 It is an architecture diagram of a video semantic communication device based on target-attribute-relation representation proposed in an embodiment of the present application; Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] With the rapid growth of multimedia data, the need for low-bitrate video compression and efficient transmission has become increasingly critical in applications such as virtual reality and future video streaming services, and the demand for efficient video transmission is increasing. Currently, semantic communication is becoming an important technology to ensure efficient and reliable transmission in low-bandwidth and high-noise environments, enhancing the communication performance of specific tasks by focusing on the semantic level of data representation and transmission.

[0021] Most video transmission methods usually use separate joint source-channel coding strategies to ensure pixel and symbol accuracy. However, with the diminishing returns of standard video encoders, the video quality at low bit rates cannot meet the requirements of user perception and specific tasks. In addition, the clear separation of source and channel coding may jeopardize the reliable transmission of video, especially over channels with low signal-to-noise ratio, resulting in increased instances of decoding failures.

[0022] Based on this, an embodiment of the present application proposes a video semantic communication method based on a target-attribute-relationship model. The target-attribute-relationship model captures the key objects of the video frame and their attributes to represent the scene semantics according to the needs of downstream applications, combines semantic information extraction and joint source channel coding, and prioritizes key semantic information through effective semantic representation to improve transmission efficiency.

[0023] The following technical terms appearing in the present invention are explained: OAR model: Object-Attribute-Relation model, also called target-attribute-relation model, is a set of semantic computing methods for formal knowledge modeling and representation based on cognitive models, which is proposed by imitating the characteristics of human brain using neurons for cognition and information storage. The OAR model considers the information of three parts more comprehensively: Object (the object itself and extension), Attribute (the connotation of the object) and Relation (relationship). The three elements of information together constitute a semantic unit to represent a certain object or scene. This is a relatively common semantic representation method for knowledge, common sense, etc.

[0024] Usually, the OAR model can be defined by a triple, where O is a finite object set consisting of several different objects, A is a finite attribute set consisting of the unique attributes that describe and characterize the object. The attribute of the object can be all the information used to describe and distinguish the characteristics of the object; R is a finite relationship set consisting of several relationships. This set of relationships is the key and core that distinguishes the OAR model from general concept representation. R also includes three relationships, namely: the relationship between Object and Attribute, the relationship between Attributes, and the relationship between Object and other Objects.

[0025] When the OAR model is actually used to represent multimedia data, the attribute set A and the relationship set R are the main data carriers. The background information can be described by a vector, while the information in the target set О only contains the target number to distinguish different targets. In the attribute set A, the attributes corresponding to each target are divided into two domains, namely dynamic attributes and static attributes. The dynamic attributes represent the information of the target that changes over time, such as the position, pixel value, posture, action, etc. of the target, while the static attributes represent the information of the target that does not change over time, such as the category, age, gender, etc. of the target. In the relationship set, each relationship needs to be described by three pieces of information, namely the starting target number of the relationship, the ending target number and the characteristics of the relationship. The starting target and the ending target respectively indicate which two targets the relationship connects. If the relationship is regarded as a predicate, the starting target is equivalent to the subject and the ending target is equivalent to the object.

[0026] For example, if Figure 1 As shown, Figure 1 It is a schematic diagram of using OAR to describe an image proposed in the related art. Figure 1In the figure, the image below the arrow is the OAR image of the image above the arrow, which is an OAR label graph. The cyclist, bike, person, tire, helmet, bag, etc. are all targets in the figure, and tire (starting target)-composition (relation)-bike (end target) constitutes a "subject-verb-object" triple form of (OA)-(RA)-(OA), and helmet (starting target)-wear (relation)-person (end target) constitutes another "subject-verb-object" triple form of (OA)-(RA)-(OA).

[0027] In a first aspect, the present application provides a method for video semantic communication based on target-attribute-relation representation, referring to Figure 2 , Figure 2 1 is a schematic diagram of the steps of a video semantic communication method based on target-attribute-relation representation proposed in an embodiment of the present application, the method comprising the following steps: Step 201: The video source takes the first frame of T consecutive frames in each GOP as a key frame, and uses an OAR feature extraction network to extract OAR features from an OAR representation graph of the key frame to obtain the OAR features of the key frame.

[0028] Step 202: The video source inputs the key frame into a source encoder to obtain an encoding result of the key frame, and inputs the encoding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame; Step 203: the video source end transmits the source-channel joint coding result of the key frame to the video reconstruction end through the AWGN channel, and losslessly transmits the OAR representation graphs of T consecutive frames through the AWGN channel; Step 204: The video reconstruction end uses the OAR feature extraction network to perform OAR feature extraction on the received OAR representation graph of the key frame to obtain the OAR feature of the key frame; Step 205: the video reconstruction end inputs the received source-channel joint coding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; Step 206: the video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

[0029] To clearly illustrate the video semantic communication method based on target-attribute-relation representation proposed in this application, Figure 3 and Figure 4 To explain, Figure 3is a flowchart of a method for video semantic communication based on target-attribute-relation representation proposed in an embodiment of the present application, Figure 4 It is a schematic diagram of the reconstruction process of key frames in a video semantic communication method based on target-attribute-relationship representation provided in an embodiment of the present application.

[0030] When implementing step 201, in the video transmission method proposed in the present application, T consecutive video frames are taken as a group of images, hereinafter referred to as GOP, and the first frame of T consecutive frames in each GOP is taken as a key frame, and the remaining frames are taken as non-key frames. OAR captures more structured semantic associations, and represents these associations with a connected graph, called an OAR representation graph, which contains richer attributes and can better reconstruct the video. At the video source end, an OAR feature extraction network is used to extract OAR features from the OAR representation graph of the key frame to obtain the OAR features of the key frame.

[0031] In an optional embodiment of the present application, the above-mentioned OAR feature extraction network includes: an embedding module, a layout generation module and a convolution module. Using the OAR feature extraction network, the OAR feature extraction is performed on the OAR representation map of the key frame to obtain the OAR feature of the above-mentioned key frame, specifically including: first, the OAR representation map of the above-mentioned key frame is input into the embedding module to obtain the OAR embedding representation of the key frame; then, the OAR embedding representation of the key frame is input into the layout generation module to obtain the OAR layout representation of the key frame, and the OAR layout representation of the above-mentioned key frame is used to represent all targets contained in the key frame; finally, the OAR layout representation of the key frame is input into the convolution module to obtain the OAR feature of the above-mentioned key frame.

[0032] When step 202 is specifically implemented, after the OAR feature of the key frame is obtained, the key frame is internally encoded and transmitted to provide contour and texture information about the background and key objects for image reconstruction. Specifically, the video source inputs the key frame into a source encoder to obtain the encoding result of the key frame with respect to the source, and inputs the encoding result of the key frame and the OAR feature and signal-to-noise ratio (such as SNR) of the key frame into a channel encoder to obtain the source-channel joint encoding result of the key frame.

[0033] In an optional embodiment of the present application, the channel encoder used above includes two branches, each branch includes: a first convolution module, an OAR modulation module and an SNR modulation module; the encoding result of the above key frame and the OAR feature and the signal-to-noise ratio of the above key frame are input into the channel encoder to obtain the source-channel joint encoding result of the above key frame, including: first, inputting the encoding result of the above key frame into the first convolution module to obtain a first result; then, inputting the first result and the OAR feature of the key frame into the OAR modulation module to obtain a first OAR modulation result; finally, inputting the first OAR modulation result and the above signal-to-noise ratio (such as SNR) into the SNR modulation module to obtain the source-channel joint encoding result of the above key frame.

[0034] When step 203 is specifically implemented, the video source end transmits the source-channel joint coding result of the key frame to the video reconstruction end through the AWGN channel, and the OAR representation graph of T consecutive frames is losslessly transmitted through the AWGN channel. Specifically, the OAR representation graph is also transmitted through the AWGN channel after entropy coding, LDPC coding and BPSK modulation, such as Figure 3 As shown in , under given LDPC coding parameters, the probability of OAR transmission error within the tested channel signal-to-noise ratio range is always 1, thus achieving lossless transmission.

[0035] When step 204 is specifically implemented, the video reconstruction end uses the OAR feature extraction network to extract OAR features from the OAR representation graph of the received key frame to obtain the OAR features of the key frame; the parameters and network structure used by the OAR feature extraction network at the video reconstruction end and the OAR feature extraction network at the video source end are the same, and the input object is the OAR representation graph of the key frame transmitted from the AWGN channel. The process of extracting OAR features from the OAR representation graph of the received key frame using the OAR feature extraction network at the video reconstruction end is the same as the process of extracting features from the OAR representation graph of the key frame using the OAR feature extraction network at the video source end, and will not be repeated here.

[0036] When step 205 is specifically implemented, after obtaining the source channel coding result of the key frame, the video reconstruction end inputs the received source channel joint coding result of the key frame, the OAR feature of the key frame, and the signal-to-noise ratio into the generator of the key frame to obtain the reconstruction result of the key frame. By combining the source channel coding result containing the source channel information and the OAR feature containing the background information and the contour information of the key object in the key frame, and combining the signal-to-noise ratio corresponding to the channel, the key frame is reconstructed, and the key frame reconstruction result containing the background information and the contour information of the key object can be reconstructed while reducing the influence of the signal-to-noise ratio, so as to ensure that the reconstruction result of the key frame has good fidelity.

[0037] In an optional embodiment of the present application, the generator of the key frame includes two branches, each of which includes: a second convolution module, an OAR modulation module and an SNR modulation module. Specifically, the video reconstruction end inputs the received source channel joint coding result of the key frame, the OAR feature of the key frame and the signal-to-noise ratio into the generator of the key frame to obtain the reconstruction result of the key frame, including the following process: first, input the received source channel joint coding result of the key frame into the second convolution module to obtain the second result; then, input the second result and the OAR feature of the key frame into the OAR modulation module to obtain the second OAR modulation result; finally, input the second OAR modulation result and the signal-to-noise ratio (such as SNR) into the SNR modulation module to obtain the reconstruction result of the key frame.

[0038] In an optional embodiment of the present application, the OAR modulation module in the channel encoder and the key frame generator includes: a splicing layer and a first branch and a second branch, the first branch includes: at least one convolution layer and at least one activation layer, and the second branch includes: at least one convolution layer and at least one activation layer. The process of modulating the OAR feature using the OAR modulation module is as follows: the result output by the previous module of the above-mentioned OAR modulation module and the OAR feature of the above-mentioned key frame are input into the above-mentioned splicing layer to obtain a splicing result; then the above-mentioned splicing result is respectively input into the first branch and the second branch to obtain the first OAR result and the second OAR result respectively; the first OAR result and the result output by the above-mentioned previous module are multiplied to obtain an OAR intermediate result; finally, the above-mentioned OAR intermediate result and the second OAR result are added to obtain an OAR modulation result. If the previous module of the above-mentioned OAR modulation module is the first convolution module, the OAR modulation result is the first OAR modulation result, and if the previous module of the above-mentioned OAR modulation module is the second convolution module, the OAR modulation result is the second OAR modulation result.

[0039] In an optional embodiment of the present application, the SNR modulation module in the channel encoder and the key frame generator includes: an average layer, a splicing layer, and a third branch and a fourth branch, the third branch includes: at least one linear layer and at least one activation layer, and the fourth branch includes: at least one linear layer and at least one activation layer. The process of using the SNR modulation module to modulate the signal-to-noise ratio is as follows: first, the result output by the previous module of the SNR modulation module is input into the average layer to obtain an average result; then the average result and the signal-to-noise ratio (such as SNR) are input into the splicing layer to obtain an intermediate splicing result; further, the intermediate splicing result is input into the third branch and the fourth branch respectively to obtain a third channel result and a fourth channel result respectively; the result output by the previous module of the SNR modulation module is multiplied by the third channel result to obtain a channel intermediate result; finally, the channel intermediate result and the fourth channel result are added to obtain a channel modulation result. If the previous module of the SNR modulation module is the OAR modulation module in the channel encoder, the channel modulation result is the source-channel joint encoding result of the key frame; if the previous module of the SNR modulation module is the OAR modulation module in the key frame generator, the channel modulation result is the reconstruction result of the key frame.

[0040] When step 206 is specifically implemented, after obtaining the reconstruction result of the key frame, the video reconstruction end inputs the reconstruction result of the key frame and the OAR representation map of each non-key frame into the generator of the non-key frame to obtain the reconstruction results of each non-key frame. By using the contour and texture information of the background and key objects in the reconstruction result of the key frame, the semantic fidelity of the reconstruction result of the subsequent non-key frame can be guaranteed.

[0041] In an optional embodiment of the present application, see Figure 5 , Figure 5 This is a schematic diagram of the non-key frame reconstruction process in a video semantic communication method based on target-attribute-relation representation provided by an embodiment of the present application. The generator of the above non-key frame includes an OAR embedding and graph calculation module, an OAR layout generation module, an optical flow estimation and frame prediction module, and an image synthesis and fusion module; specifically, the video reconstruction end inputs the reconstruction results of the above key frames and the OAR representation graphs of each non-key frame into the generator of the non-key frame to obtain the reconstruction results of each non-key frame, including the following process: The OAR representation graph of the non-key frame is input into the above-mentioned OAR embedding and graph calculation module to obtain the OAR embedding representation; the above-mentioned OAR embedding representation is input into the graph convolution network in the OAR layout generation module to obtain the deep features of each target in the non-key frame; the deep features of each target in the above-mentioned non-key frame are input into the layout generation network in the OAR layout generation module to obtain the OAR layout representation of the non-key frame, and the OAR layout representation of the above-mentioned non-key frame is used to represent all targets contained in the non-key frame; the OAR layout representation of the above-mentioned non-key frame is input into the layout feature extraction network in the optical flow estimation and frame prediction module to obtain the OAR layout features of the non-key frame; the previous reconstructed frame is input into the image feature extraction network in the optical flow estimation and frame prediction module to obtain the image features of the previous reconstructed frame; the OAR layout features of the above-mentioned non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame are extracted into the image feature extraction network in the optical flow estimation and frame prediction module to obtain the image features of the previous reconstructed frame; The image features of the frames are spliced ​​to obtain intermediate features; the intermediate features are input into the optical flow encoder in the optical flow estimation network in the optical flow estimation and frame prediction module to obtain the optical flow encoding result; the optical flow encoding result is input into the optical flow decoder in the optical flow estimation network in the optical flow estimation and frame prediction module to obtain the estimated optical flow; according to the estimated optical flow and the previous reconstructed frame, the optical flow estimation result is obtained through affine transformation; the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame are input into the image synthesis network in the image synthesis and fusion module to obtain the image synthesis result; the optical flow encoding result is input into the weight estimation layer of the image fusion network in the image synthesis and fusion module to obtain the estimated weight; according to the optical flow estimation result, the image synthesis result and the estimated weight, the reconstruction result of the non-key frame is obtained.

[0042] Among them, when the above-mentioned non-key frame is the first non-key frame, the previous reconstructed frame is the reconstruction result of the key frame, and the OAR layout features of the previous frame are the OAR layout features of the key frame; when the above-mentioned non-key frame is any frame between the second non-key frame and the last non-key frame in the GOP, the previous reconstructed frame is the reconstruction result of the previous non-key frame of the non-reconstructed frame, and the OAR layout features of the previous frame are the OAR layout features of the previous non-key frame of the non-reconstructed frame.

[0043] In an optional embodiment of the present application, the OAR embedding and graph calculation module and the OAR layout generation module in the above-mentioned non-key frame generator have the same network structure as the embedding module and layout generation module in the OAR feature extraction network, and use the same parameters.

[0044] In a video semantic communication method based on target-attribute-relation representation provided in an embodiment of the present application, a video source end uses the first frame of T consecutive frames in each GOP as a key frame, and uses an OAR feature extraction network to extract OAR features from the OAR representation graph of the key frame to obtain the OAR features of the key frame; the video source end inputs the key frame into a source encoder to obtain an encoding result of the key frame, and inputs the encoding result of the key frame, the OAR features of the key frame, and the signal-to-noise ratio into a channel encoder to obtain a source-channel joint encoding result of the key frame; the video source end transmits the source channel of the key frame through an AWGN channel The joint coding result is given to the video reconstruction end, and the OAR representation graphs of T consecutive frames are losslessly transmitted through the AWGN channel; the video reconstruction end uses the OAR feature extraction network to extract the OAR features of the received key frame OAR representation graph to obtain the OAR features of the key frame; the video reconstruction end inputs the received source channel joint coding result of the key frame, the OAR features of the key frame and the signal-to-noise ratio into the generator of the key frame to obtain the reconstruction result of the key frame; the video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graphs of each non-key frame into the generator of the non-key frame to obtain the reconstruction results of each non-key frame. This application utilizes the compactness of OAR data-related downstream tasks, and assists with the source channel joint codec of video transmission, so as to pay more attention to the key objects in the scene. This method can still maintain good foreground reconstruction quality under low bandwidth and signal-to-noise ratio conditions.

[0045] In a second aspect of the embodiment of the present application, a video semantic communication device based on target-attribute-relation representation is provided, referring to Figure 6 , Figure 6 : is an architecture diagram of a video semantic communication device based on target-attribute-relation representation proposed in an embodiment of the present application, the device comprising: A first key frame feature extraction module 601 is used to take the first frame of T consecutive frames in each GOP as a key frame, and use an OAR feature extraction network to perform OAR feature extraction on the OAR representation graph of the key frame to obtain the OAR feature of the key frame; The joint coding module 602 is used to input the key frame into a source encoder to obtain a coding result of the key frame, and input the coding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint coding result of the key frame; The transmission module 603 is used to transmit the source-channel joint coding result of the key frame to the video reconstruction end through the AWGN channel, and to losslessly transmit the OAR representation graphs of T consecutive frames through the AWGN channel; The second key frame feature extraction module 604 is used to extract OAR features from the received OAR representation graph of the key frame using the OAR feature extraction network to obtain the OAR features of the key frame; The key frame reconstruction module 605 is used to input the received source-channel joint coding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; The non-key frame reconstruction module 606 is used to input the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

[0046] Optionally, the channel encoder in the joint encoding module includes two branches, each branch including: a first convolution module, an OAR modulation module and an SNR modulation module; The joint encoding module also includes: A first convolution submodule, used for inputting the encoding result of the key frame into the first convolution module to obtain a first result; A first OAR modulation submodule, configured to input the first result and the OAR feature of the key frame into the OAR modulation module to obtain a first OAR modulation result; The first SNR modulation submodule is used to input the first OAR modulation result and the signal-to-noise ratio into the SNR modulation module to obtain the source-channel joint coding result of the key frame.

[0047] Optionally, the key frame generator of the key frame reconstruction module includes two branches, each branch including: a second convolution module, an OAR modulation module and an SNR modulation module; The key frame reconstruction module also includes: A second convolution submodule, used for inputting the received source-channel joint coding result of the key frame into the second convolution module to obtain a second result; A second OAR modulation submodule, configured to input the second result and the OAR feature of the key frame into the OAR modulation module to obtain a second OAR modulation result; The second SNR modulation submodule is used to input the second OAR modulation result and the signal-to-noise ratio into the SNR modulation module to obtain the reconstruction result of the key frame.

[0048] Optionally, the OAR modulation module in the first OAR modulation submodule or the second OAR modulation submodule includes: a splicing layer and a first branch and a second branch, the first branch includes: at least one convolution layer and at least one activation layer, and the second branch includes: at least one convolution layer and at least one activation layer; The first OAR modulation submodule or the second OAR modulation submodule further includes: A first splicing unit, configured to input the result output by the previous module of the OAR modulation module and the OAR feature of the key frame into the splicing layer to obtain a splicing result; A first OAR processing unit is used to input the splicing result into the first branch and the second branch respectively, to obtain a first OAR result and a second OAR result respectively; A second OAR processing unit, configured to multiply the first OAR result by the result output by the previous module to obtain an OAR intermediate result; The third OAR processing unit is used to add the OAR intermediate result and the second OAR result to obtain an OAR modulation result.

[0049] Optionally, the SNR modulation module in the first SNR modulation submodule or the second SNR modulation submodule includes: an average layer, a splicing layer, and a third branch and a fourth branch, the third branch includes: at least one linear layer and at least one activation layer, and the fourth branch includes: at least one linear layer and at least one activation layer; The first SNR modulation submodule or the second SNR modulation submodule further includes: An averaging unit, used to input the result output by the previous module of the SNR modulation module into the averaging layer to obtain an average result; A second splicing unit, used for inputting the average result and the signal-to-noise ratio into the splicing layer to obtain an intermediate splicing result; A first channel processing unit, configured to input the intermediate splicing result into the third branch and the fourth branch respectively, to obtain a third channel result and a fourth channel result respectively; A second channel processing unit, configured to multiply the result output by the previous module of the SNR modulation module by the third channel result to obtain a channel intermediate result; The third channel processing unit is used to add the channel intermediate result and the fourth channel result to obtain a channel modulation result.

[0050] Optionally, the OAR feature extraction network in the first key frame feature extraction module or the second key frame feature extraction module includes: an embedding module, a layout generation module and a convolution module; The first key frame feature extraction module or the second key frame feature extraction module further includes: A first embedding submodule, configured to input the OAR representation graph of the key frame into the embedding module to obtain the OAR embedding representation of the key frame; A first layout generation submodule, configured to input the OAR embedding representation of the key frame into the layout generation module to obtain the OAR layout representation of the key frame, wherein the OAR layout representation of the key frame is used to represent all objects contained in the key frame; The convolution submodule is used to input the OAR layout representation of the key frame into the convolution module to obtain the OAR feature of the key frame.

[0051] Optionally, the generator of the non-key frame includes an OAR embedding and graph calculation module, an OAR layout generation module, an optical flow estimation and frame prediction module, and an image synthesis and fusion module; The non-key frame reconstruction module comprises: A second embedding submodule, configured to input the OAR representation graph of the non-keyframe into the OAR embedding and graph calculation module to obtain an OAR embedding representation; A graph convolution submodule, used for inputting the OAR embedding representation into the graph convolution network in the OAR layout generation module to obtain deep features of each target in a non-key frame; A second layout generation submodule is used to input the deep features of each target in the non-key frame into the layout generation network in the OAR layout generation module to obtain the OAR layout representation of the non-key frame, wherein the OAR layout representation of the non-key frame is used to characterize all targets contained in the non-key frame; A layout feature extraction submodule, used for inputting the OAR layout representation of the non-key frame into the layout feature extraction network in the optical flow estimation and frame prediction module to obtain the OAR layout feature of the non-key frame; An image feature extraction submodule, used for inputting the last reconstructed frame into the image feature extraction network in the optical flow estimation and frame prediction module to obtain the image features of the last reconstructed frame; A feature stitching submodule, used for stitching the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame to obtain an intermediate feature; An optical flow encoding submodule, used for inputting the intermediate features into an optical flow encoder in an optical flow estimation network in the optical flow estimation and frame prediction module to obtain an optical flow encoding result; An optical flow decoding submodule, used for inputting the optical flow encoding result into an optical flow decoder in the optical flow estimation network in the optical flow estimation and frame prediction module to obtain an estimated optical flow; An optical flow estimation submodule, configured to obtain an optical flow estimation result according to the estimated optical flow and the last reconstructed frame; An image synthesis submodule, used for inputting the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame into the image synthesis network in the image synthesis and fusion module to obtain an image synthesis result; A weight estimation submodule, used for inputting the optical flow encoding result into the weight estimation layer of the image fusion network in the image synthesis and fusion module to obtain an estimated weight; The non-key reconstruction submodule obtains the reconstruction result of the non-key frame according to the optical flow estimation result, the image synthesis result and the estimation weight; when the non-key frame is the first non-key frame, the previous reconstructed frame is the reconstruction result of the key frame, and the OAR layout feature of the previous frame is the OAR layout feature of the key frame.

[0052] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 7 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 7 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a video semantic communication method based on target-attribute-relationship representation disclosed in an embodiment of the present application.

[0053] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, a video semantic communication method based on target-attribute-relationship representation disclosed in the embodiment of the present application is implemented.

[0054] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0055] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0058] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0059] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0060] The above is a detailed introduction to a video semantic communication method and device based on target-attribute-relationship representation provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video semantic communication method based on target-attribute-relation representation, characterized in that: The method comprises: The video source takes the first frame of T consecutive frames in each GOP as a key frame, and uses an OAR feature extraction network to extract OAR features from the OAR representation graph of the key frame to obtain the OAR features of the key frame; The video source end inputs the key frame into a source encoder to obtain an encoding result of the key frame, and inputs the encoding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame; The video source end transmits the source-channel joint coding result of the key frame to the video reconstruction end through the AWGN channel, and losslessly transmits the OAR representation graphs of T consecutive frames through the AWGN channel; The video reconstruction end uses the OAR feature extraction network to extract OAR features from the OAR representation graph of the received key frame to obtain the OAR features of the key frame; The video reconstruction end inputs the received source-channel joint coding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; The video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

2. The video semantic communication method based on target-attribute-relation representation according to claim 1 is characterized in that: The channel encoder comprises two branches, each branch comprising: a first convolution module, an OAR modulation module and an SNR modulation module; Inputting the encoding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame, including: Inputting the encoding result of the key frame into the first convolution module to obtain a first result; Inputting the first result and the OAR feature of the key frame into the OAR modulation module to obtain a first OAR modulation result; The first OAR modulation result and the signal-to-noise ratio are input into the SNR modulation module to obtain a source-channel joint coding result of the key frame.

3. The video semantic communication method based on target-attribute-relation representation according to claim 1 is characterized in that: The key frame generator includes two branches, each branch includes: a second convolution module, an OAR modulation module and an SNR modulation module; The video reconstruction end inputs the received source-channel joint encoding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame, including: Inputting the received source-channel joint coding result of the key frame into the second convolution module to obtain a second result; Inputting the second result and the OAR feature of the key frame into the OAR modulation module to obtain a second OAR modulation result; The second OAR modulation result and the signal-to-noise ratio are input into the SNR modulation module to obtain a reconstruction result of the key frame.

4. The video semantic communication method based on target-attribute-relation representation according to claim 2 or 3, characterized in that: The OAR modulation module comprises: a concatenation layer and a first branch and a second branch, the first branch comprises: at least one convolution layer and at least one activation layer, the second branch comprises: at least one convolution layer and at least one activation layer; Inputting the result output by the previous module of the OAR modulation module and the OAR feature of the key frame into the splicing layer to obtain a splicing result; Inputting the splicing results into the first branch and the second branch respectively, to obtain a first OAR result and a second OAR result respectively; Multiplying the first OAR result by the result output by the previous module to obtain an OAR intermediate result; The OAR intermediate result and the second OAR result are added to obtain an OAR modulation result.

5. The video semantic communication method based on target-attribute-relation representation according to claim 2 or 3, characterized in that: The SNR modulation module includes: an average layer, a splicing layer, and a third branch and a fourth branch, the third branch includes: at least one linear layer and at least one activation layer, and the fourth branch includes: at least one linear layer and at least one activation layer; Inputting the result output by the previous module of the SNR modulation module into the averaging layer to obtain an average result; Inputting the average result and the signal-to-noise ratio into the splicing layer to obtain an intermediate splicing result; Inputting the intermediate splicing result into the third branch and the fourth branch respectively, to obtain a third channel result and a fourth channel result respectively; Multiplying the result output by the previous module of the SNR modulation module by the third channel result to obtain a channel intermediate result; The channel intermediate result and the fourth channel result are added together to obtain a channel modulation result.

6. The video semantic communication method based on target-attribute-relation representation according to claim 1, characterized in that: The OAR feature extraction network includes: an embedding module, a layout generation module and a convolution module; Using the OAR feature extraction network, OAR features are extracted from the OAR representation graph of the key frame to obtain the OAR features of the key frame, including: Inputting the OAR representation graph of the key frame into the embedding module to obtain the OAR embedding representation of the key frame; Inputting the OAR embedding representation of the key frame into the layout generation module to obtain the OAR layout representation of the key frame, where the OAR layout representation of the key frame is used to represent all objects contained in the key frame; The OAR layout representation of the key frame is input into the convolution module to obtain the OAR feature of the key frame.

7. The video semantic communication method based on target-attribute-relation representation according to claim 1 is characterized in that: The non-key frame generator includes an OAR embedding and graph calculation module, an OAR layout generation module, an optical flow estimation and frame prediction module, and an image synthesis and fusion module; The video reconstruction end inputs the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame, including: Inputting the OAR representation graph of the non-key frame into the OAR embedding and graph calculation module to obtain the OAR embedding representation; The OAR embedding representation is input into the graph convolutional network in the OAR layout generation module to obtain the deep features of each target in the non-key frame; Inputting the deep features of each target in the non-key frame into the layout generation network in the OAR layout generation module to obtain the OAR layout representation of the non-key frame, wherein the OAR layout representation of the non-key frame is used to characterize all targets contained in the non-key frame; Inputting the OAR layout representation of the non-key frame into the layout feature extraction network in the optical flow estimation and frame prediction module to obtain the OAR layout feature of the non-key frame; Inputting the last reconstructed frame into the image feature extraction network in the optical flow estimation and frame prediction module to obtain the image features of the last reconstructed frame; splicing the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame to obtain intermediate features; Inputting the intermediate features into an optical flow encoder in an optical flow estimation network in the optical flow estimation and frame prediction module to obtain an optical flow encoding result; Inputting the optical flow encoding result into an optical flow decoder in an optical flow estimation network in the optical flow estimation and frame prediction module to obtain an estimated optical flow; Obtaining an optical flow estimation result according to the estimated optical flow and the last reconstructed frame; Inputting the OAR layout features of the non-key frame, the OAR layout features of the previous frame, and the image features of the previous reconstructed frame into the image synthesis network in the image synthesis and fusion module to obtain an image synthesis result; Inputting the optical flow encoding result into the weight estimation layer of the image fusion network in the image synthesis and fusion module to obtain an estimated weight; Obtaining a reconstruction result of a non-key frame according to the optical flow estimation result, the image synthesis result and the estimation weight; When the non-key frame is the first non-key frame, the last reconstructed frame is the reconstruction result of the key frame, and the OAR layout feature of the last frame is the OAR layout feature of the key frame.

8. A video semantic communication device based on target-attribute-relation representation, characterized in that: The device comprises: A first key frame feature extraction module is used to take the first frame of T consecutive frames in each GOP as a key frame, and use an OAR feature extraction network to perform OAR feature extraction on an OAR representation graph of the key frame to obtain an OAR feature of the key frame; A joint encoding module, used for inputting the key frame into a source encoder to obtain an encoding result of the key frame, and inputting the encoding result of the key frame and the OAR feature and signal-to-noise ratio of the key frame into a channel encoder to obtain a source-channel joint encoding result of the key frame; A transmission module, configured to transmit the source-channel joint encoding result of the key frame to a video reconstruction end through an AWGN channel, and to losslessly transmit the OAR representation graphs of T consecutive frames through the AWGN channel; A second key frame feature extraction module is used to extract OAR features from the received OAR representation graph of the key frame using an OAR feature extraction network to obtain the OAR features of the key frame; A key frame reconstruction module, used for inputting the received source-channel joint coding result of the key frame, the OAR feature and the signal-to-noise ratio of the key frame into the key frame generator to obtain the reconstruction result of the key frame; The non-key frame reconstruction module is used to input the reconstruction result of the key frame and the OAR representation graph of each non-key frame into the generator of the non-key frame to obtain the reconstruction result of each non-key frame.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the video semantic communication method based on target-attribute-relation representation according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the video semantic communication method based on target-attribute-relation representation described in any one of claims 1 to 7 is implemented.

Images