Video transmission method and device based on key frame semantics, electronic equipment, storage medium and program product

Abstract

The application provides a video transmission method and device based on key frame semantics, electronic equipment, a storage medium and a program product, and relates to the technical field of video transmission. In the sending end, the semantic information of an IDR frame is extracted by using a graph-to-text model, the foreground target object and the position area information of the foreground target object are determined from a normal I frame, and the prediction error and motion vector information contained in a prediction frame are extracted. The background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained, and transmission is performed. In the receiving end, the IDR frame, the normal I frame and the prediction frame are generated in sequence by using the obtained video semantic information, and the original H.26X video is recovered in order according to the H.26X frame structure configuration characteristics. On the premise of ensuring the basic reliability of video transmission, the demand of effectively reducing the transmission bandwidth pressure is met.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video transmission technology, and in particular to a video transmission method, device, electronic device, storage medium and program product based on key frame semantics. Background Art

[0002] The H.26X digital video compression format is one of the most common video compression standards, widely used for video streaming and storage. Video encoded using this format is referred to as "H.26X video." H.26X video consists of a series of digital image frames, which can be divided into two types: key frames (I-frames) and predicted frames (P-frames and B-frames). I-frames contain more information and can be independently encoded and decoded. Any IDR frame (a special type of I-frame) can be randomly decoded to play the video stream. P / B frames, on the other hand, contain less information and cannot be independently encoded or decoded. Because P / B frames use inter-frame prediction coding using motion prediction, only the prediction error and motion vectors are transmitted, significantly reducing transmission redundancy. Although H.26X video compression is highly efficient, the current rapid growth of video traffic is placing significant strain on limited bandwidth as massive data transmission places a significant strain on limited transmission bandwidth. The successful transmission of H.26X video streams in bandwidth-constrained scenarios remains uncertain.

[0003] Semantic communication technology represents a new paradigm in communications. Unlike traditional bit-level transmission methods based on Shannon's theory, semantic communication extracts valid semantic information from the source signal, compresses it, and transmits it. At the receiving end, this information is used to restore the original signal, thereby achieving "semantic" communication. Because semantic information represents an extreme compression of the original signal, semantic communication can significantly reduce transmission bandwidth usage. Currently, semantic communication technology based on large models has achieved promising results in fields such as text and image transmission, making it a highly efficient method for digital signal transmission.

[0004] Although H.26X video streams can be decomposed into a series of digital image frames and directly transmitted using semantic communication technology to achieve video transmission, the number of image frames that make up H.26X video is large (for example, 24 frames per second), and the extraction and recovery of semantic information for each frame requires a large amount of computing resources, making it difficult to ensure the timely and effective transmission of all image frames. Therefore, existing image semantic transmission methods are no longer applicable in the H.26X video stream transmission process. It is necessary to redesign an efficient and applicable semantic communication method based on the characteristics of H.26X video. Summary of the Invention

[0005] Embodiments of the present invention provide a video transmission method, apparatus, electronic device, storage medium, and program product based on key frame semantics. Based on the extraction and recovery of image semantic information, semantic transmission steps are designed for the different image characteristics of key frames and predicted frames in H.26X series standard format video streams, so as to be suitable for extracting and recovering key semantic information of H.26X video streams.

[0006] A first aspect of an embodiment of the present invention provides a video transmission method based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream, wherein the semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver. The method includes:

[0007] The H.26X video stream transmitter decodes the H.26X video stream through the H.26X video decoder and extracts all image frames in the H.26X video stream;

[0008] The H.26X video stream transmitter classifies all extracted image frames into IDR frames, normal I frames, and predicted frames.

[0009] The H.26X video stream transmitter extracts semantic information of the IDR frame using a fine-tuned graph-to-text model based on the transmitter's local semantic knowledge base, and uses this information as background semantic information for the H.26X video stream.

[0010] The H.26X video stream transmitter combines the semantic information of the IDR frame to determine the foreground target object and its location area information from the ordinary I frame;

[0011] The H.26X video stream transmitter extracts the prediction error and motion vector information from the predicted frame and combines it with the location area information of the foreground target object to obtain the dynamic behavior information of the foreground target object.

[0012] The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0013] The H.26X video stream receiver uses the received background semantic information to recover the IDR frame through a fine-tuned text-based graph model based on the local semantic knowledge base of the receiver.

[0014] The H.26X video stream receiver recovers the normal I frame and the predicted frame based on the recovered IDR frame and the received dynamic behavior information;

[0015] The H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame, and the restored prediction frame through the H.26X video encoder to obtain a restored H.26X video stream.

[0016] Optionally, the H.26X video stream receiver recovers a normal I frame and a predicted frame based on the recovered IDR frame and the received dynamic behavior information, including:

[0017] Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame;

[0018] Based on the prediction error and motion vector information in the dynamic behavior information and the restored normal I frame, a predicted frame is restored.

[0019] Optionally, the H.26X video stream transmitter, in addition to extracting all image frames from the H.26X video stream, further determines the frame rate and GOP structure information of the H.26X video stream;

[0020] The H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream;

[0021] The H.26X video stream receiver recovers normal I frames and predicted frames based on the recovered IDR frames and received dynamic behavior information, including:

[0022] The H.26X video stream receiver determines the total number of restored image frames per second based on the received frame rate information;

[0023] The H.26X video stream receiver determines the intervals between I-frames, P-frames, and B-frames in a group of pictures based on the received GOP structure information. Combined with the total number of restored frames per second, the receiver determines the number and permutation of I-frames, P-frames, and B-frames that need to be restored per second.

[0024] According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

[0025] Optionally, the H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in text form, including:

[0026] The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object, obtains a semantic encoding result, and transmits the semantic encoding result in the form of text.

[0027] Optionally, the method further includes:

[0028] The H.26X video stream receiving end combines the receiving end's local semantic knowledge base to perform semantic decoding on the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

[0029] Optionally, the semantic communication system further comprises: a cloud-based shared knowledge base;

[0030] The H.26X video stream sender builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base.

[0031] The H.26X video stream receiver builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base.

[0032] A second aspect of an embodiment of the present invention provides a video transmission device based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream. The semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver. The device includes:

[0033] The video decoding module is used by the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream;

[0034] The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, ordinary I frames, and predicted frames;

[0035] An extraction module is used at the H.26X video stream transmitter to extract semantic information of the IDR frame using a fine-tuned graph-to-text model based on a local semantic knowledge base at the transmitter, and use the extracted semantic information as background semantic information of the H.26X video stream;

[0036] A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame;

[0037] The second determination module is used by the H.26X video stream transmitter to extract the prediction error and motion vector information from the predicted frame, and obtain the dynamic behavior information of the foreground target object by combining the position area information of the foreground target object;

[0038] A transmission module, configured for the H.26X video stream transmitter to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0039] The first recovery module is used at the H.26X video stream receiving end to recover the IDR frame using the received background semantic information through a large model of the text graph fine-tuned based on the receiving end's local semantic knowledge base;

[0040] The second recovery module is used for the H.26X video stream receiving end to recover the normal I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information;

[0041] The video encoding module is used for encoding the restored IDR frame, the restored common I frame and the restored prediction frame by the H.26X video encoder at the H.26X video stream receiving end to obtain the restored H.26X video stream.

[0042] Optionally, the second recovery module is specifically configured to:

[0043] Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame;

[0044] Based on the prediction error and motion vector information in the dynamic behavior information and the restored normal I frame, a predicted frame is restored.

[0045] Optionally, the video decoding module is further configured to determine the frame rate and GOP structure information of the H.26X video stream; the transmission module is further configured to transmit the frame rate information and GOP structure information of the H.26X video stream from the H.26X video stream transmitter;

[0046] The second recovery module is specifically configured to:

[0047] The H.26X video stream receiver determines the total number of restored image frames per second based on the received frame rate information;

[0048] The H.26X video stream receiver determines the intervals between I-frames, P-frames, and B-frames in a group of pictures based on the received GOP structure information. Combined with the total number of restored frames per second, the receiver determines the number and permutation of I-frames, P-frames, and B-frames that need to be restored per second.

[0049] According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

[0050] Optionally, the transmission module is specifically configured to:

[0051] The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object, obtains a semantic encoding result, and transmits the semantic encoding result in the form of text.

[0052] Optionally, the device further comprises:

[0053] The receiving module is used for the H.26X video stream receiving end to combine with the receiving end local semantic knowledge base to perform semantic decoding on the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

[0054] Optionally, the semantic communication system further comprises: a cloud-based shared knowledge base;

[0055] The H.26X video stream sender builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base.

[0056] The H.26X video stream receiver builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base.

[0057] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the method described in the first aspect of the present invention.

[0058] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method described in the first aspect of the present invention.

[0059] A fifth aspect of an embodiment of the present invention provides a computer program product, comprising a computer program / instruction, which is used by a processor to implement the steps in the method described in the first aspect of the present invention.

[0060] To further compress H.26X video and reduce transmission bandwidth, an embodiment of the present invention, based on image semantic information extraction and recovery, designs a semantic transmission process tailored to the different image characteristics of key frames and predicted frames in H.26X standard format video streams. Specifically, at the H.26X video stream transmitter, an image-to-text macro model (image-to-text macro model) is used to extract semantic information (including key background information) from IDR frames. Combined with this semantic information, foreground objects and their location information are determined from regular I frames. The prediction error and motion vector information contained in the predicted frames are also extracted. The background semantic information and dynamic behavior information of the foreground objects in the H.26X video stream are then transmitted as video semantic information. At the H.26X video stream receiver, the acquired video semantic information is used to sequentially generate IDR frames, regular I frames, and predicted frames. The original H.26X video is then restored in an orderly manner based on the H.26X frame structure. This approach effectively reduces transmission bandwidth pressure while ensuring basic video transmission reliability. Therefore, in the embodiment of the present invention, the semantic communication technology is used for H.26X video stream transmission in combination with the characteristics of the image frame structure of the H.26X video stream itself, so as to effectively reduce the pressure on the H.26X video stream transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A flowchart showing the steps of a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown;

[0063] Figure 2 A schematic diagram of a semantic information extraction process at a transmitting end in a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown;

[0064] Figure 3 The diagram shows a flowchart of semantic information recovery at a receiving end in a video transmission method based on key frame semantics provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] In an embodiment of the present invention, a video transmission method based on key frame semantics is provided. Figure 1 As shown, Figure 1 A flowchart of a keyframe semantics-based video transmission method provided by an embodiment of the present invention is shown. Specifically, the method is applied to a semantic communication system for transmitting an H.26X video stream, the semantic communication system including an H.26X video stream transmitter and an H.26X video stream receiver. The method includes the following steps:

[0067] S101, the H.26X video stream transmitter decodes the H.26X video stream through an H.26X video decoder and extracts all image frames from the H.26X video stream;

[0068] S102, the H.26X video stream transmitter classifies all extracted image frames to determine IDR frames, normal I frames, and predicted frames;

[0069] S103, the H.26X video stream transmitter extracts semantic information of the IDR frame using a fine-tuned graph-to-text model based on a local semantic knowledge base of the transmitter, and uses the extracted semantic information as background semantic information of the H.26X video stream;

[0070] S104, the H.26X video stream transmitter determines the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame;

[0071] S105, the H.26X video stream transmitter extracts the prediction error and motion vector information from the predicted frame, and combines the position area information of the foreground target object to obtain the dynamic behavior information of the foreground target object;

[0072] S106, the H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0073] S107, the H.26X video stream receiver recovers the IDR frame using the received background semantic information by using the fine-tuned text-based graph model based on the local semantic knowledge base of the receiver;

[0074] S108, the H.26X video stream receiver recovers a normal I frame and a predicted frame based on the recovered IDR frame and the received dynamic behavior information;

[0075] S109 , the H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame, and the restored prediction frame through an H.26X video encoder to obtain a restored H.26X video stream.

[0076] To further compress H.26X video and reduce transmission bandwidth, an embodiment of the present invention proposes: At the H.26X video stream transmitter, a large image-to-text model (large image-to-text model) is used to extract semantic information (including key background information) from IDR frames. Combined with this semantic information, foreground objects and their location information are determined from regular I frames. The prediction error and motion vector information contained in the predicted frames are also extracted. This combined process yields background semantic information and dynamic behavior information for the H.26X video stream, which is then transmitted as video semantic information. At the H.26X video stream receiver, the acquired video semantic information is used to sequentially generate IDR frames, regular I frames, and predicted frames. The original H.26X video is then restored in an orderly fashion based on the H.26X frame structure. This approach effectively reduces transmission bandwidth pressure while ensuring basic video transmission reliability. Therefore, in this embodiment, semantic communication technology is applied to H.26X video stream transmission, taking into account the unique image frame structure of the H.26X video stream itself, effectively reducing transmission bandwidth pressure.

[0077] Specifically, H.26X video coding primarily utilizes a predictive coding architecture. Predictive coding includes intra-frame prediction and inter-frame prediction, which leverage the spatial or temporal correlations of adjacent pixels. The currently encoded pixel is first predicted using previously transmitted pixels. The residual between the predicted value and the true value (i.e., the prediction error) is then encoded and transmitted. The key frames (I-frames) that comprise H.26X video utilize spatial correlation for intra-frame prediction, while the predicted frames (P / B-frames) utilize temporal correlation for inter-frame prediction. For example, an H.26X video with a frame rate of 24 frames per second, when the group of pictures (GOP) structure is M=3 and N=12, requires the transmission of two I-frames, six P-frames, and 16 B-frames per second.

[0078] Based on the characteristics of the H.26X video stream GOP structure, an embodiment of the present invention proposes extracting semantic information (including key background information) from IDR frames. Combined with the semantic information of the IDR frames, the foreground target object and its location area information are determined from ordinary I frames. The prediction error and motion vector information contained in the predicted frame are also extracted. In summary, the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained as video semantic information and transmitted. Based on this video semantic information, the original H.26X video can be restored in an orderly manner at the receiving end.

[0079] In an embodiment of the present invention, in step S101, all image frames in the H.26X video stream can be extracted using the FFmpeg gadget and grouped according to the GOP structure. Typically, the first I-frame in a GOP structure is an IDR frame. Furthermore, the type of each I-frame can be determined by examining the type field of the Network Abstraction Layer Unit (NALU) of each I-frame to determine whether it is an IDR frame or a regular I-frame.

[0080] In an embodiment of the present invention, the above-mentioned step S108 includes: restoring the ordinary I frame based on the position area information of the foreground target object in the received dynamic behavior information and the restored IDR frame; restoring the predicted frame based on the prediction error and motion vector information in the dynamic behavior information and the restored ordinary I frame.

[0081] In an embodiment of the present invention, dynamic behavior information is obtained based on the foreground target object and its position area information determined from the ordinary I-frame, as well as the prediction error and motion vector information extracted from the predicted frame. This dynamic behavior information can characterize the trajectory of the position area changes of the foreground target object, so that the ordinary I-frame can be restored based on this dynamic behavior information. The position area information of the foreground target object in the restored ordinary I-frame is the same as the information carried in the dynamic behavior information. After obtaining the ordinary I-frame, the prediction error and motion vector information of the predicted frame can also be determined based on the dynamic behavior information, so that the predicted frame (P / B frame) can be restored based on the ordinary I-frame.

[0082] In the embodiment of the present invention, the dynamic behavior information may be represented as a coordinate point change trajectory or a contour point change angle of the foreground target object.

[0083] In this embodiment of the present invention, in step S107, the IDR frame is restored at the receiving end based on the text-generated image model configured at the receiving end and the received background semantic information. In this embodiment of the present invention, the background semantic information sent by the transmitting end is obtained by semantically extracting the IDR frame based on the image-generated text model configured at the transmitting end.

[0084] In an embodiment of the present invention, the image-to-text model configured at the sending end is fine-tuned based on the sending end's local semantic knowledge base, and the text-to-image model configured at the receiving end is fine-tuned based on the receiving end's local semantic knowledge base. In this embodiment of the present invention, the sending end's local semantic knowledge base and the receiving end's local semantic knowledge base are synchronized, so that the image-to-text model at the sending end and the text-to-image model at the receiving end are mutually adapted, thereby ensuring that the background knowledge mastered by the models at both ends is consistent. The image-to-text model is used at the sending end to extract the semantic information of the IDR frame, which is then transmitted to the receiving end. At the receiving end, the text-to-image model is used to restore the IDR frame based on the semantic information, making the restored image more realistic and reliable.

[0085] Specifically, in an embodiment of the present invention, the semantic communication system further includes: a cloud-based shared knowledge base; an H.26X video stream transmitter constructs a local semantic knowledge base at the transmitter based on the cloud-based shared knowledge base and maintains synchronization with the cloud-based shared knowledge base; and an H.26X video stream receiver constructs a local semantic knowledge base at the receiver based on the cloud-based shared knowledge base and maintains synchronization with the cloud-based shared knowledge base. In this embodiment of the present invention, synchronization between the local semantic knowledge base at the transmitter and the local semantic knowledge base at the receiver can be achieved based on the cloud-based shared knowledge base.

[0086] In an embodiment of the present invention, the cloud-based shared knowledge base can be configured by technical personnel, specifically according to the field to which the image content of the H.26X video stream to be transmitted belongs. After constructing the local semantic knowledge base of the originating end and the local semantic knowledge base of the receiving end based on the cloud-based shared knowledge base, the text description extracted by the image-generated text model fine-tuned based on the local semantic knowledge base of the originating end is more accurate and reliable, and the image generated by the text-generated image model fine-tuned based on the local semantic knowledge base of the receiving end is more realistic and reliable.

[0087] In an embodiment of the present invention, the background semantic information and dynamic behavior information corresponding to the video stream can also be combined with the local semantic knowledge base of the originating end to obtain a semantic encoding result corresponding to the video stream, and the result can be transmitted in the form of text. Specifically, step S106 includes: the H.26X video stream transmitter combines the local semantic knowledge base of the originating end to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object, obtains a semantic encoding result, and transmits the semantic encoding result in the form of text.

[0088] Correspondingly, the received semantic encoding result can be semantically decoded at the receiving end based on the local semantic knowledge base of the receiving end. Specifically, the method also includes: S106.5: The H.26X video stream receiving end combines the local semantic knowledge base of the receiving end to semantically decode the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

[0089] In the embodiment of the present invention, the local semantic knowledge base at the sending end and the local semantic knowledge base at the receiving end are synchronized, thereby further ensuring the credibility of semantic transmission.

[0090] In an embodiment of the present invention, the frame rate and GOP structure of the H.26X video stream need to be referenced during the process of extrapolating and generating the remaining ordinary I frames and interpolating and generating P / B frames. Specifically, in an optional implementation, the frame rate and GOP structure information of the H.26X video stream also need to be transmitted between the sending end and the receiving end so that the receiving end can restore the original H.26X video in an orderly manner.

[0091] Specifically, the H.26X video stream transmitter extracts all image frames from the H.26X video stream and also determines the frame rate and GOP structure information of the H.26X video stream. During the transmission process, the H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream.

[0092] In this case, the step S108 includes:

[0093] S1081, the H.26X video stream receiver determines the total number of restored image frames per second based on the received frame rate information;

[0094] S1082: The H.26X video stream receiver determines the intervals between I frames, P frames, and B frames in a group of pictures based on the received GOP structure information. Combined with the total number of restored frames per second, the receiver determines the number and permutation of I frames, P frames, and B frames required to be restored per second.

[0095] S1083 , according to the number and arrangement of I frames, P frames and B frames that need to be restored per second, and based on the restored IDR frames and the received dynamic behavior information, restore the normal I frames and restore the predicted frames.

[0096] In this embodiment of the present invention, the process of extrapolating the remaining normal I frames and interpolating the P / B frames can refer to the frame rate and GOP structure of the H.26X video stream. The video frame rate can determine the total number of image frames generated per second, and the GOP structure can determine the intervals between I frames, P frames, and B frames in a group of frames. Combining these two, the number of I frames, P frames, and B frames required per second and their specific arrangement and combination can be determined. After the receiving and transmitting ends are synchronized, the specific time points for generating normal I frames and P / B frames can be determined based on this information.

[0097] In the embodiment of the present invention, during H.26X video transmission, the transmitting end extracts semantic information and transmits it through a semantic channel, while the receiving end generates image frames using the semantic information obtained from the semantic channel and restores the original video.

[0098] For ease of understanding, the semantic information extraction process of the sending end in the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below. Figure 2 As shown, Figure 2 The diagram shows a flow chart of semantic information extraction at a sending end in a video transmission method based on key frame semantics provided by an embodiment of the present invention.

[0099] First, the H.26X video stream is decoded to obtain image frames. A large model for generating text from images is then used to extract the image semantic information of the instantly decoded refresh frames (i.e., IDR frames) within the H.26X video stream. This IDR frame image semantic information not only captures the critical background information that is updated instantly within the video, but also identifies the primary target objects within the video and their location within the IDR frames. The foreground target objects within the I-frames are then identified based on the IDR frames and combined with the remaining I-frames. These foreground target objects are then identified as dynamic target objects, and their location within the I-frames is determined. The prediction error and motion vectors contained within the P / B frames are then extracted. By combining the dynamic target objects and their location with the prediction error and motion vectors, dynamic behavior recognition of the target objects can be achieved.

[0100] The image semantic information extracted from the IDR frame is used as the background semantic information of the video stream, and the dynamic behavior of the identified target object is used as the dynamic information of the video stream object. By combining the video stream background semantic information and the target object dynamic behavior information with the local semantic knowledge base at the originating end, the complete semantic information of the video transmitted in text form can be obtained. This information enters the semantic channel after semantic encoding processing.

[0101] In an embodiment of the present invention, by decomposing an H.26X video stream into a series of image frames and then compressing the video data using semantic communication technology to obtain complete video semantic information transmitted in text form, the transmission bandwidth pressure of the H.26X video stream can be effectively alleviated.

[0102] In an embodiment of the present invention, when the channel quality is good, it can support more information transmission, and can directly use images (for example, sketches, contours, etc.) to transmit video background semantic information or directly transmit IDR image frames; when the channel quality is poor, it can only support a small amount of information transmission, and can use text to transmit background semantic information, thereby reducing channel transmission pressure.

[0103] For ease of understanding, the semantic information recovery process of the receiving end in the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below. Figure 3 As shown, Figure 3 The diagram shows a flow chart of semantic information recovery at a receiving end in a video transmission method based on key frame semantics provided by an embodiment of the present invention.

[0104] In an embodiment of the present invention, the receiving end first receives semantic information of the video stream through a semantic channel. Then, semantic decoding is used to obtain background semantic information and object dynamic information in the video stream, combined with the receiving end's local semantic knowledge base. Subsequently, based on the obtained video stream background semantic information and object dynamic information, an IDR frame is generated using a large text-generated image model. The generated IDR frame is then combined with the dynamic behavior information of the foreground object, and the remaining ordinary I frames are extrapolated based on the number of I frames determined in the GOP structure. Similarly, based on the number of P / B frames determined in the GOP structure, all P / B frames are interpolated based on the I frame combined with the object dynamic behavior information. The process of generating image frames based on the object dynamic behavior information can correspond to the regional position of the background image depicted by the IDR frame based on the motion trajectory of the dynamic behavior of the P / B frame. Finally, based on the frame sorting method in the GOP structure, all generated I frames and P / B frames are jointly encoded to restore the original H.26X video stream.

[0105] For ease of understanding, the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below based on an exemplary embodiment:

[0106] For example, a sample H.264 video is recorded in advance. This video shows three white A4 sheets of paper with red, green, and blue sticky notes attached to them, moving in and out of the initial shot. The following uses this video as an example to further illustrate the specific implementation of the keyframe semantics-based video transmission method.

[0107] At the sending end, the FFmpeg tool is first used to extract all image frames from the video and group them according to the GOP structure. The first I-frame in the GOP structure is designated as an IDR frame. This results in six IDR frames in the video: before the A4 paper with a red sticky note is moved in, before the A4 paper with a red sticky note is moved in, before the A4 paper with a green sticky note is moved in, before the A4 paper with a green sticky note is moved in, before the A4 paper with a blue sticky note is moved in, and after the A4 paper with a blue sticky note is moved in. Then, using a large image-to-text model (e.g., CM3Leon) fine-tuned using the local semantic knowledge base at the receiving end, semantic information is extracted from each IDR frame, such as "a keyboard is on the desk," "an A4 paper with a pink sticky note is on the desk," "an A4 paper with a blue sticky note is on the desk," and so on. The semantic information of each IDR frame can be transmitted as independent background semantic information in the form of text over a semantic channel.

[0108] Combining the IDR frame with the remaining ordinary I frames reveals that the foreground target object is "an A4 sheet of paper with sticky notes attached to it." The target object and its location in the I frame, combined with the motion vector information contained in the P / B frame, can reveal the target object's dynamic information, namely, the "trajectory of the A4 sheet of paper with different colored sticky notes moving in and out" in the video. This foreground target object behavior information can be transmitted as video stream object dynamic information in text form over a semantic channel, thereby transmitting the video stream background semantic information and video stream object dynamic information as video stream semantic information to the receiver.

[0109] At the receiving end, semantic information about the video stream transmitted to the receiving end is first obtained from the semantic channel. The semantic decoding module then parses the background semantics and object dynamics of the transmitted video. For background semantics such as "a keyboard is on the desk," "an A4 paper with a pink sticky note on the desk," or "an A4 paper with a blue sticky note on the desk," IDR frames are generated using diffusion models fine-tuned from the local semantic knowledge base. These images consist of six IDR frames, each of which depicts three A4 papers with red, green, and blue sticky notes moving in and out of the background. Based on this information, the number of I-frames and P / B-frames implied by the GOP structure is extrapolated to generate the remaining normal I-frames and interpolated to generate P / B-frames. The generation of I-frames can reference the target object and its location in the object dynamic information, e.g., "an A4 sheet of paper with a sticky note is located on the left side of the frame" or "an A4 sheet of paper with a sticky note is located in the center of the frame." The generation of P / B-frames is based on the dynamic behavior of the foreground target object in the behavior information, e.g., "an A4 sheet of paper with a sticky note moves from the left side of the frame to the center" or "an A4 sheet of paper with a sticky note moves from the center to the left side of the frame." By encoding all generated image frames in the standard H.264 format, an H.264 video with the same semantics as the original video can be restored.

[0110] Based on the same inventive concept, an embodiment of the present invention provides a video transmission device based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream. The semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver. The device includes:

[0111] The video decoding module is used by the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream;

[0112] The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, ordinary I frames, and predicted frames;

[0113] An extraction module is used at the H.26X video stream transmitter to extract semantic information of the IDR frame using a fine-tuned graph-to-text model based on a local semantic knowledge base at the transmitter, and use the extracted semantic information as background semantic information of the H.26X video stream;

[0114] A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame;

[0115] The second determination module is used by the H.26X video stream transmitter to extract the prediction error and motion vector information from the predicted frame, and obtain the dynamic behavior information of the foreground target object by combining the position area information of the foreground target object;

[0116] A transmission module, configured for the H.26X video stream transmitter to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0117] The first recovery module is used at the H.26X video stream receiving end to recover the IDR frame using the received background semantic information through a large model of the text graph fine-tuned based on the receiving end's local semantic knowledge base;

[0118] The second recovery module is used for the H.26X video stream receiving end to recover the normal I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information;

[0119] The video encoding module is used for encoding the restored IDR frame, the restored common I frame and the restored prediction frame by the H.26X video encoder at the H.26X video stream receiving end to obtain the restored H.26X video stream.

[0120] Based on the same inventive concept, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes, the steps in the video transmission method based on key frame semantics described in any of the above embodiments are implemented.

[0121] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements the steps of the video transmission method based on key frame semantics described in any of the above embodiments.

[0122] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the video transmission method based on key frame semantics described in any of the above embodiments.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0125] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0126] The above is a detailed introduction to a video transmission method based on key frame semantics provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video transmission method based on key frame semantics, characterized in that: A semantic communication system for transmitting an H.26X video stream, the semantic communication system comprising an H.26X video stream transmitter and an H.26X video stream receiver; the method comprising: The H.26X video stream transmitter decodes the H.26X video stream through the H.26X video decoder and extracts all image frames in the H.26X video stream; The H.26X video stream transmitter classifies all extracted image frames into IDR frames, normal I frames, and predicted frames. The H.26X video stream transmitter extracts semantic information of the IDR frame using a fine-tuned graph-to-text model based on the transmitter's local semantic knowledge base, and uses this information as background semantic information for the H.26X video stream. The H.26X video stream transmitter combines the semantic information of the IDR frame to determine the foreground target object and its location area information from the ordinary I frame; The H.26X video stream transmitter extracts the prediction error and motion vector information from the predicted frame and combines it with the location area information of the foreground target object to obtain the dynamic behavior information of the foreground target object. The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text; The H.26X video stream receiver uses the received background semantic information to recover the IDR frame through a fine-tuned text-based graph model based on the local semantic knowledge base of the receiver. The H.26X video stream receiver recovers the normal I frame and the predicted frame based on the recovered IDR frame and the received dynamic behavior information; The H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame, and the restored prediction frame through the H.26X video encoder to obtain a restored H.26X video stream.

2. The video transmission method based on key frame semantics according to claim 1, characterized in that The H.26X video stream receiver recovers normal I frames and predicted frames based on the recovered IDR frames and received dynamic behavior information, including: Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame; Based on the prediction error and motion vector information in the dynamic behavior information and the restored normal I frame, a predicted frame is restored.

3. The video transmission method based on key frame semantics according to claim 1 or 2, characterized in that: The H.26X video stream transmitter extracts all image frames from the H.26X video stream and also determines the frame rate and GOP structure information of the H.26X video stream; The H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream; The H.26X video stream receiver recovers normal I frames and predicted frames based on the recovered IDR frames and received dynamic behavior information, including: The H.26X video stream receiver determines the total number of restored image frames per second based on the received frame rate information; The H.26X video stream receiver determines the intervals between I-frames, P-frames, and B-frames in a group of pictures based on the received GOP structure information. Combined with the total number of restored frames per second, the receiver determines the number and permutation of I-frames, P-frames, and B-frames that need to be restored per second. According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

4. The video transmission method based on key frame semantics according to claim 1, characterized in that: The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text, including: The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object, obtains a semantic encoding result, and transmits the semantic encoding result in the form of text.

5. The video transmission method based on key frame semantics according to claim 4, characterized in that: The method further comprises: The H.26X video stream receiving end combines the receiving end's local semantic knowledge base to perform semantic decoding on the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

6. The video transmission method based on key frame semantics according to claim 1, characterized in that: The semantic communication system further includes: a cloud-based shared knowledge base; The H.26X video stream sender builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base. The H.26X video stream receiver builds a local semantic knowledge base based on the cloud-based shared knowledge base and keeps it synchronized with the cloud-based shared knowledge base.

7. A video transmission device based on key frame semantics, characterized in that: A semantic communication system for transmitting an H.26X video stream, comprising an H.26X video stream transmitter and an H.26X video stream receiver; the apparatus comprises: The video decoding module is used by the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream; The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, ordinary I frames, and predicted frames; An extraction module is used at the H.26X video stream transmitter to extract semantic information of the IDR frame using a fine-tuned graph-to-text model based on a local semantic knowledge base at the transmitter, and use the extracted semantic information as background semantic information of the H.26X video stream; A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame; The second determination module is used by the H.26X video stream transmitter to extract the prediction error and motion vector information from the predicted frame, and obtain the dynamic behavior information of the foreground target object by combining the position area information of the foreground target object; A transmission module, configured for the H.26X video stream transmitter to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text; The first recovery module is used at the H.26X video stream receiving end to recover the IDR frame using the received background semantic information through a large model of the text graph fine-tuned based on the receiving end's local semantic knowledge base; The second recovery module is used for the H.26X video stream receiving end to recover the normal I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information; The video encoding module is used for encoding the restored IDR frame, the restored common I frame and the restored prediction frame by the H.26X video encoder at the H.26X video stream receiving end to obtain the restored H.26X video stream.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executed, the processor implements the steps in the video transmission method based on key frame semantics as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video transmission method based on key frame semantics are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the video transmission method based on key frame semantics described in any one of claims 1 to 6 are implemented.

Images