Video transmission method and device based on key frame semantics, electronic equipment, storage medium and program product

Abstract

The invention provides a video transmission method and device based on key frame semantics, electronic equipment, a storage medium and a program product, and relates to the technical field of video transmission. The method comprises the following steps: extracting semantic information of an IDR frame at a sending end by utilizing a graph-to-text model, and determining a foreground target object and position area information of the foreground target object from a common I frame; extracting prediction errors and motion vector information contained in the prediction frame; and the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained and transmitted. According to the method, an IDR frame, a common I frame and a prediction frame are sequentially generated by using acquired video semantic information at a receiving end, and an original H.26X video is orderly recovered according to H.26X frame structure configuration characteristics, so that the requirement of effectively reducing transmission bandwidth pressure is met on the premise of ensuring basic reliability of video transmission.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video transmission, and in particular to a video transmission method, device, electronic device, storage medium and program product based on key frame semantics. Background Art

[0002] H.26X digital video compression format is one of the most common video compression standards currently, and is widely used in video stream transmission and storage. Video encoded in this format can be referred to as "H.26X video". H.26X video consists of a series of digital image frames, which can be divided into two types: key frames (i.e., I frames) and predicted frames (including P frames and B frames). I frames contain more information and can be independently encoded and decoded. Any IDR frame (a special I frame) can be randomly decoded to play the video stream, while P / B frames contain relatively less information and cannot be independently encoded and decoded. Since P / B frames use motion prediction to perform inter-frame prediction coding, only prediction errors and motion vectors are transmitted, which can significantly reduce transmission redundancy. Although H.26X video compression efficiency is high, in the context of the rapid growth of video traffic, massive data transmission puts great pressure on limited transmission bandwidth. In scenarios where channel bandwidth is limited, there is still great uncertainty as to whether H.26X video streams can be smoothly transmitted.

[0003] Semantic communication technology is a new paradigm of communication technology. Different from the traditional bit-level transmission method based on Shannon's theory, semantic communication extracts effective semantic information from the source signal, compresses and transmits it, and uses the acquired semantic information to restore the original signal at the receiving end, thereby realizing communication based on "semantics". Since semantic information is an extreme compression of the original signal, semantic communication can greatly reduce the occupancy of transmission bandwidth. At present, semantic communication technology based on large models has achieved good application results in the fields of text and image transmission. Therefore, semantic communication has become an efficient way of digital signal transmission.

[0004] Although the H.26X video stream can be decomposed into a series of digital image frames, and video transmission can be achieved by directly transmitting image frames using semantic communication technology, the number of image frames that make up the H.26X video is large (for example, 24 frames per second), and the extraction and recovery of semantic information of each frame of the image requires a large amount of computing resources, making it difficult to ensure the timely and effective transmission of all image frames. Therefore, the existing image semantic transmission method will no longer be applicable in the H.26X video stream transmission process, and it is necessary to redesign an efficient and applicable semantic communication method based on the characteristics of H.26X video. Summary of the invention

[0005] The embodiments of the present invention provide a video transmission method, device, electronic device, storage medium and program product based on key frame semantics. On the basis of image semantic information extraction and recovery, semantic transmission steps are designed for different image characteristics of key frames and predicted frames in H.26X series standard format video streams, so as to be suitable for key semantic information extraction and recovery of H.26X video streams.

[0006] A first aspect of an embodiment of the present invention provides a video transmission method based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream, wherein the semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver; the method includes:

[0007] The H.26X video stream sender decodes the H.26X video stream through the H.26X video decoder and extracts all image frames in the H.26X video stream;

[0008] The H.26X video stream transmitter classifies all extracted image frames to determine IDR frames, common I frames, and predicted frames;

[0009] The H.26X video stream transmitter extracts semantic information of the IDR frame through a graph-based text model fine-tuned based on a local semantic knowledge base of the transmitter, and uses the semantic information as background semantic information of the H.26X video stream;

[0010] The H.26X video stream transmitter determines the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame;

[0011] The H.26X video stream transmitter extracts the prediction error and motion vector information from the prediction frame, and combines the location area information of the foreground target object to obtain the dynamic behavior information of the foreground target object;

[0012] The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0013] The H.26X video stream receiver recovers the IDR frame by using the received background semantic information through a fine-tuned large model of the text graph based on the local semantic knowledge base of the receiver;

[0014] The H.26X video stream receiving end recovers the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information;

[0015] The H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain a restored H.26X video stream.

[0016] Optionally, the H.26X video stream receiving end restores a common I frame and a predicted frame according to the restored IDR frame and the received dynamic behavior information, including:

[0017] Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame;

[0018] Based on the prediction error and motion vector information in the dynamic behavior information and the restored ordinary I frame, the predicted frame is restored.

[0019] Optionally, the H.26X video stream transmitter, in addition to extracting all image frames from the H.26X video stream, further determines the frame rate and GOP structure information of the H.26X video stream;

[0020] The H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream;

[0021] The H.26X video stream receiver recovers the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information, including:

[0022] The H.26X video stream receiving end determines the total number of image frames restored per second based on the received frame rate information;

[0023] The H.26X video stream receiver determines the intervals between I frames, P frames and B frames in a picture group according to the received GOP structure information, and obtains the number and arrangement of I frames, P frames and B frames that need to be restored per second in combination with the total number of image frames restored per second;

[0024] According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

[0025] Optionally, the H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text, including:

[0026] The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object to obtain a semantic encoding result, and transmits the semantic encoding result in the form of text.

[0027] Optionally, the method further comprises:

[0028] The H.26X video stream receiving end combines the receiving end local semantic knowledge base to semantically decode the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

[0029] Optionally, the semantic communication system further comprises: a cloud-based shared knowledge base;

[0030] The H.26X video stream sender builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base;

[0031] The H.26X video stream receiver builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base.

[0032] A second aspect of an embodiment of the present invention provides a video transmission device based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream, wherein the semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver; the device includes:

[0033] The video decoding module is used for the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream;

[0034] The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, common I frames and predicted frames;

[0035] An extraction module is used for the H.26X video stream sending end to extract semantic information of the IDR frame through a graph-based text model fine-tuned based on a local semantic knowledge base of the sending end, and use the semantic information as the background semantic information of the H.26X video stream;

[0036] A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the common I frame in combination with the semantic information of the IDR frame;

[0037] The second determination module is used for the H.26X video stream transmitter to extract the prediction error and motion vector information from the prediction frame, and obtain the dynamic behavior information of the foreground target object in combination with the position area information of the foreground target object;

[0038] A transmission module, used for the H.26X video stream sending end to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0039] The first recovery module is used for the H.26X video stream receiving end to recover the IDR frame by using the received background semantic information through the Wensheng graph large model fine-tuned based on the local semantic knowledge base of the receiving end;

[0040] The second recovery module is used for the H.26X video stream receiving end to recover the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information;

[0041] The video encoding module is used for the H.26X video stream receiving end to encode the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain the restored H.26X video stream.

[0042] Optionally, the second recovery module is specifically configured to:

[0043] Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame;

[0044] Based on the prediction error and motion vector information in the dynamic behavior information and the restored ordinary I frame, the predicted frame is restored.

[0045] Optionally, the video decoding module is further used to determine the frame rate and GOP structure information of the H.26X video stream; the transmission module is further used to transmit the frame rate information and GOP structure information of the H.26X video stream to the H.26X video stream transmitter;

[0046] The second recovery module is specifically used for:

[0047] The H.26X video stream receiving end determines the total number of image frames restored per second based on the received frame rate information;

[0048] The H.26X video stream receiver determines the intervals between I frames, P frames and B frames in a picture group according to the received GOP structure information, and obtains the number and arrangement of I frames, P frames and B frames that need to be restored per second in combination with the total number of image frames restored per second;

[0049] According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

[0050] Optionally, the transmission module is specifically used to:

[0051] The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object to obtain a semantic encoding result, and transmits the semantic encoding result in the form of text.

[0052] Optionally, the device further comprises:

[0053] The receiving module is used for the H.26X video stream receiving end to combine with the receiving end local semantic knowledge base, perform semantic decoding on the received semantic encoding result, and obtain the received background semantic information and the received dynamic behavior information.

[0054] Optionally, the semantic communication system further comprises: a cloud-based shared knowledge base;

[0055] The H.26X video stream sender builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base;

[0056] The H.26X video stream receiver builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base.

[0057] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the method described in the first aspect of the present invention.

[0058] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method described in the first aspect of the present invention.

[0059] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the method described in the first aspect of the present invention.

[0060] In the embodiment of the present invention, in order to further "compress" the H.26X video and reduce the transmission bandwidth occupation, based on the extraction and recovery of image semantic information, the semantic transmission step is designed for the different image characteristics of key frames and predicted frames in the H.26X standard format video stream, so as to be suitable for the extraction and recovery of key semantic information of the H.26X video stream. Specifically, it is proposed that: at the H.26X video stream sending end, the image is used to generate a large text model (image-generated text model) to extract the semantic information of the IDR frame (including key background information), and combined with the semantic information of the IDR frame, the foreground target object and the position area information of the foreground target object are determined from the ordinary I frame; and the prediction error and motion vector information contained in the predicted frame are extracted; in summary, the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained as video semantic information and transmitted. At the H.26X video stream receiving end, the acquired video semantic information is used to generate IDR frames, ordinary I frames and predicted frames in turn, and the original H.26X video is restored in order according to the H.26X frame structure configuration characteristics, so as to meet the demand of effectively reducing the transmission bandwidth pressure under the premise of ensuring the basic reliability of video transmission. Therefore, in the embodiment of the present invention, the semantic communication technology is used for H.26X video stream transmission in combination with the characteristics of the image frame structure of the H.26X video stream itself, so as to effectively reduce the pressure on the H.26X video stream transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0062] Figure 1 A flowchart showing the steps of a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown;

[0063] Figure 2 A schematic diagram of a semantic information extraction process at a sending end in a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown;

[0064] Figure 3 A schematic diagram of a semantic information recovery process at a receiving end in a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] In an embodiment of the present invention, a video transmission method based on key frame semantics is provided. Figure 1 As shown, Figure 1 The flowchart of the step of the video transmission method based on key frame semantics provided by an embodiment of the present invention is shown; specifically, the method is applied to a semantic communication system for transmitting an H.26X video stream, and the semantic communication system includes an H.26X video stream sending end and an H.26X video stream receiving end; the method includes the following steps:

[0067] S101, the H.26X video stream transmitter decodes the H.26X video stream through an H.26X video decoder and extracts all image frames in the H.26X video stream;

[0068] S102, the H.26X video stream transmitter classifies all extracted image frames to determine IDR frames, common I frames, and predicted frames;

[0069] S103, the H.26X video stream transmitter extracts semantic information of the IDR frame through a fine-tuned graph-text model based on a local semantic knowledge base of the transmitter, and uses the semantic information as background semantic information of the H.26X video stream;

[0070] S104, the H.26X video stream transmitter determines the foreground target object and the location area information of the foreground target object from the common I frame in combination with the semantic information of the IDR frame;

[0071] S105, the H.26X video stream transmitter extracts the prediction error and motion vector information from the prediction frame, and obtains the dynamic behavior information of the foreground target object by combining the position area information of the foreground target object;

[0072] S106, the H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0073] S107, the H.26X video stream receiving end recovers the IDR frame by using the received background semantic information through the fine-tuned textual graph large model based on the local semantic knowledge base of the receiving end;

[0074] S108, the H.26X video stream receiving end restores a common I frame and a predicted frame according to the restored IDR frame and the received dynamic behavior information;

[0075] S109, the H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain a restored H.26X video stream.

[0076] In the embodiment of the present invention, in order to further "compress" the H.26X video and reduce the transmission bandwidth occupancy, it is proposed that: at the H.26X video stream sending end, the image is used to generate a large text model (image-generated text large model), the semantic information of the IDR frame (including key background information) is extracted, and the foreground target object and the position area information of the foreground target object are determined from the ordinary I frame in combination with the semantic information of the IDR frame; and the prediction error and motion vector information contained in the predicted frame are extracted; in summary, the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained, which are used as video semantic information and transmitted. At the H.26X video stream receiving end, the obtained video semantic information is used to generate IDR frames, ordinary I frames and predicted frames in sequence, and the original H.26X video is restored in order according to the configuration characteristics of the H.26X frame structure, so as to meet the demand of effectively reducing the pressure of the transmission bandwidth while ensuring that the video transmission is basically reliable. Therefore, in the embodiment of the present invention, the semantic communication technology is used for H.26X video stream transmission in combination with the characteristics of the image frame structure of the H.26X video stream itself, so as to effectively reduce the pressure of the H.26X video stream transmission bandwidth.

[0077] Specifically, H.26X video coding mainly adopts the structural design of predictive coding, which includes intra-frame prediction and inter-frame prediction. It uses the spatial or temporal correlation of adjacent pixels respectively. First, the pixel currently being encoded is predicted by the transmitted pixel, and then the residual between the predicted value and the true value (i.e., the prediction error) is encoded and transmitted. The key frames (I frames) that make up the H.26X video use spatial correlation to achieve intra-frame prediction, and the predicted frames (P / B frames) use temporal correlation to achieve inter-frame prediction. For example, an H.26X video frame rate is 24 frames per second. When the group of pictures (GOP) structure is M=3 and N=12, then 2 I frames, 6 P frames, and 16 B frames need to be transmitted per second.

[0078] Based on the characteristics of the above-mentioned H.26X video stream image group structure, an embodiment of the present invention proposes to extract the semantic information (including key background information) of the IDR frame, and determine the foreground target object and the position area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame; and extract the prediction error and motion vector information contained in the predicted frame; in summary, the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object are obtained as video semantic information and transmitted, so that the original H.26X video can be restored in an orderly manner at the receiving end based on the video semantic information.

[0079] In the embodiment of the present invention, in step S101, all image frames in the H.26X video stream can be extracted using the FFmpeg gadget and grouped according to the GOP structure. Usually, the first I frame of the GOP structure is an IDR frame. In addition, the type of the network abstraction layer unit (NALU) of all I frames can be checked to determine whether it is an IDR frame or a normal I frame.

[0080] In an embodiment of the present invention, the above-mentioned step S108 includes: restoring an ordinary I frame based on the position area information of the foreground target object in the received dynamic behavior information and the restored IDR frame; restoring a predicted frame based on the prediction error and motion vector information in the dynamic behavior information and the restored ordinary I frame.

[0081] In the embodiment of the present invention, the dynamic behavior information is obtained based on the foreground target object and its position area information determined from the ordinary I frame and the prediction error and motion vector information extracted from the predicted frame. The dynamic behavior information can characterize the change trajectory of the position area of ​​the foreground target object, so that the ordinary I frame can be restored based on the dynamic behavior information. The position area information of the foreground target object in the restored ordinary I frame is the same as the information carried in the dynamic behavior information. After obtaining the ordinary I frame, the prediction error and motion vector information of the predicted frame can also be determined based on the dynamic behavior information, so that the predicted frame (P / B frame) can be restored based on the ordinary I frame.

[0082] In the embodiment of the present invention, the dynamic behavior information may be represented as a coordinate point change trajectory or a contour point change angle of the foreground target object.

[0083] In the embodiment of the present invention, in step S107, the IDR frame at the receiving end is restored based on the text-generated image model configured at the receiving end and the received background semantic information. In the embodiment of the present invention, the background semantic information sent by the sending end is obtained by semantically extracting the IDR frame based on the image-generated text model configured at the sending end.

[0084] In the embodiment of the present invention, the image-generated text model configured at the sending end is obtained by fine-tuning the local semantic knowledge base of the sending end, and the text-generated image model configured at the receiving end is obtained by fine-tuning the local semantic knowledge base of the receiving end. In the embodiment of the present invention, the local semantic knowledge base of the sending end and the local semantic knowledge base of the receiving end are synchronized, so that the image-generated text model of the sending end and the text-generated image model of the receiving end are mutually adapted, so that the background knowledge mastered by the large models at both ends of the sending and receiving can be consistent, and the image-generated text model is used at the sending end to extract the semantic information of the IDR frame, and then transmitted to the receiving end, and the text-generated image model is used at the receiving end to restore the IDR frame based on the semantic information, and the restored image is more realistic and reliable.

[0085] Specifically, in the embodiment of the present invention, the semantic communication system further includes: a cloud-based shared knowledge base; the H.26X video stream transmitter constructs a local semantic knowledge base at the transmitter based on the cloud-based shared knowledge base, and keeps synchronization with the cloud-based shared knowledge base; the H.26X video stream receiver constructs a local semantic knowledge base at the receiver based on the cloud-based shared knowledge base, and keeps synchronization with the cloud-based shared knowledge base. In the embodiment of the present invention, the synchronization of the local semantic knowledge base at the transmitter and the local semantic knowledge base at the receiver can be achieved based on the cloud-based shared knowledge base.

[0086] In an embodiment of the present invention, the cloud-based shared knowledge base can be configured by a technical staff, and specifically can be configured according to the field to which the image content of the H.26X video stream to be transmitted belongs. After constructing a local semantic knowledge base at the originating end and a local semantic knowledge base at the receiving end based on the cloud-based shared knowledge base, the text description extracted by the image generation text large model fine-tuned based on the local semantic knowledge base at the originating end is more accurate and reliable, and the image generated by the text generation image large model fine-tuned based on the local semantic knowledge base at the receiving end is more realistic and reliable.

[0087] In an embodiment of the present invention, the background semantic information and dynamic behavior information corresponding to the video stream can also be combined with the local semantic knowledge base of the originating end to obtain the semantic coding result corresponding to the video stream, and transmit it in the form of text. Specifically, the step S106 includes: the H.26X video stream transmitter combines the local semantic knowledge base of the originating end to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object to obtain the semantic coding result, and transmits the semantic coding result in the form of text.

[0088] Correspondingly, at the receiving end, the received semantic encoding result can be semantically decoded based on the local semantic knowledge base of the receiving end. Specifically, the method also includes: S106.5: The H.26X video stream receiving end combines the local semantic knowledge base of the receiving end to semantically decode the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

[0089] In the embodiment of the present invention, the local semantic knowledge base at the sending end and the local semantic knowledge base at the receiving end are synchronized, thereby further ensuring the credibility of semantic transmission.

[0090] In an embodiment of the present invention, in the process of extrapolating and generating the remaining ordinary I frames and interpolating and generating P / B frames, it is necessary to refer to the frame rate and GOP structure of the H.26X video stream. Specifically, in an optional implementation, the frame rate and GOP structure information of the H.26X video stream also need to be transmitted between the sending end and the receiving end, so that the receiving end can restore the original H.26X video in an orderly manner.

[0091] Specifically, in addition to extracting all image frames from the H.26X video stream, the H.26X video stream transmitter also determines the frame rate and GOP structure information of the H.26X video stream. During the transmission process, the H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream.

[0092] In this case, the step S108 includes:

[0093] S1081, the H.26X video stream receiving end determines the total number of image frames restored per second according to the received frame rate information;

[0094] S1082, the H.26X video stream receiving end determines the intervals between I frames, P frames and B frames in a picture group according to the received GOP structure information, and obtains the number and arrangement and combination of I frames, P frames and B frames that need to be restored per second in combination with the total number of image frames restored per second;

[0095] S1083, according to the number and arrangement of I frames, P frames and B frames that need to be restored per second, and based on the restored IDR frames and the received dynamic behavior information, restore the common I frames and restore the predicted frames.

[0096] In the embodiment of the present invention, the process of extrapolating and generating the remaining ordinary I frames and interpolating and generating P / B frames can refer to the frame rate and GOP structure of the H.26X video stream. The total number of image frames generated per second can be determined by the video frame rate, and the interval between I frames, P frames and B frames in a group of frames can be determined by the GOP structure. The combination of the two can obtain the number and specific arrangement and combination of I frames, P frames and B frames that need to be generated per second. Therefore, after completing the time synchronization between the receiving end and the sending end, the specific time nodes for generating ordinary I frames and P / B frames can be obtained based on the above information.

[0097] In the embodiment of the present invention, during H.26X video transmission, the sending end extracts semantic information and transmits it through a semantic channel, and the receiving end generates image frames using the semantic information obtained from the semantic channel and restores the original video.

[0098] For ease of understanding, the semantic information extraction process of the sending end in the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below. Figure 2 As shown, Figure 2 A schematic diagram of a semantic information extraction process at a sending end in a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown.

[0099] First, the H.26X video stream is decoded to obtain image frames, and the image-generated text model is used to extract the image semantic information of the instant decoding refresh frame (i.e., IDR frame) in the H.26X video stream. The IDR frame image semantic information can not only obtain the key background information that is updated instantly in the video, but also identify the main target objects in the video and their location area information in the IDR frame. Then, based on the IDR frame combined with the remaining ordinary I frames, the foreground target objects contained in the ordinary I frame are identified, and they are used as dynamic target objects and the location area of ​​the target object is determined. The prediction error and motion vector contained in the P / B frame are extracted. By combining the dynamic target object and the location area with the prediction error and motion vector, the dynamic behavior recognition of the target object can be realized.

[0100] The image semantic information extracted from the IDR frame is used as the background semantic information of the video stream, and the dynamic behavior of the identified target object is used as the dynamic information of the video stream object. The video stream background semantic information and the target object dynamic behavior information are combined with the local semantic knowledge base at the originating end to obtain the complete semantic information of the video transmitted in text form. This information enters the semantic channel after semantic encoding processing.

[0101] In the embodiment of the present invention, the H.26X video stream is decomposed into a series of image frames and then the video data is compressed using semantic communication technology to obtain complete video semantic information transmitted in text form, which can effectively alleviate the transmission bandwidth pressure of the H.26X video stream.

[0102] In an embodiment of the present invention, when the channel quality is good, it can support more information transmission, and the video background semantic information can be directly transmitted in the form of images (for example, sketches, contours, etc.) or IDR image frames can be directly transmitted; when the channel quality is poor, it can only support a small amount of information transmission, and the background semantic information can be transmitted in the form of text, thereby reducing the channel transmission pressure.

[0103] For ease of understanding, the semantic information recovery process of the receiving end in the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below. Figure 3 As shown, Figure 3 A schematic diagram of a semantic information recovery process at a receiving end in a video transmission method based on key frame semantics provided by an embodiment of the present invention is shown.

[0104] In an embodiment of the present invention, the receiving end first receives the semantic information of the video stream through a semantic channel, and then uses semantic decoding processing to obtain the background semantic information and object dynamic information in the video stream in combination with the local semantic knowledge base of the receiving end. Subsequently, based on the acquired video stream background semantic information and object dynamic information, the text-generated image large model is used to generate an IDR frame; the generated IDR frame is then combined with the dynamic behavior information of the foreground object, and the remaining ordinary I frames are extrapolated according to the number of I frames determined in the GOP structure; similarly, according to the number of P / B frames determined in the GOP structure, all P / B frames are interpolated and generated according to the I frame combined with the object dynamic behavior information. Among them, the process of generating image frames in combination with the object dynamic behavior information can correspond to the regional position of the background image depicted by the IDR frame according to the action trajectory of the dynamic behavior of the P / B frame. Finally, according to the frame sorting method in the GOP structure, all generated I frames and P / B frames are jointly encoded to restore the original H.26X video stream.

[0105] For ease of understanding, the video transmission method based on key frame semantics provided by an embodiment of the present invention is explained below based on an exemplary embodiment:

[0106] For example, a sample video in H.264 format is recorded in advance, which records the complete process of three white A4 papers with red, green and blue sticky notes respectively moving into and out of the initial shot. The following will take the transmission of this video as an example to further illustrate the specific implementation process of the video transmission method based on key frame semantics.

[0107] At the sending end, firstly, all the image frames in the video are extracted using the FFmpeg gadget, and grouped according to the GOP structure. The first I frame of the GOP structure is the IDR frame, and thus 6 IDR frames in the video are obtained, namely, before the A4 paper with a red sticky note is moved in, before the A4 paper with a red sticky note is moved in, before the A4 paper with a green sticky note is moved in, before the A4 paper with a green sticky note is moved in, before the A4 paper with a blue sticky note is moved in, and before the A4 paper with a blue sticky note is moved in. Then, the semantic information of each IDR frame image is extracted using the image-to-text model (e.g., CM3Leon) fine-tuned by the local semantic knowledge base of the receiving end, such as "a keyboard is placed on the desk", "there is an A4 paper with a pink sticky note on the desk", "there is an A4 paper with a blue sticky note on the desk", etc. The semantic information of each IDR frame image can be transmitted as independent background semantic information in the form of text through the semantic channel.

[0108] The IDR frame combined with the other ordinary I frames can obtain the foreground target object as "A4 paper with sticky notes attached". The target object and its position area in the I frame combined with the motion vector information contained in the P / B frame can obtain the dynamic information of the target object, that is, the "track information of the A4 paper with sticky notes of different colors moving in or out" in the video. The foreground target object behavior information can be transmitted as video stream object dynamic information in text form through the semantic channel, thereby transmitting the video stream background semantic information and video stream object dynamic information as video stream semantic information to the receiving end.

[0109] At the receiving end, the semantic information of the video stream transmitted to the receiving end is first obtained from the semantic channel, and the semantic decoding function module is used to parse the background semantic information and object dynamic information of the transmitted video. For background semantic information, such as "a keyboard is placed on the desk", "an A4 paper with a pink sticky note on the desk", "an A4 paper with a blue sticky note on the desk", etc., the diffusion model (Diffusion Models) fine-tuned by the local semantic knowledge base can be used to generate IDR frames, that is, to generate 6 IDR frame images of three A4 papers with red, green and blue sticky notes respectively moving in or out of the background one after another. On this basis, combined with the number of I frames and P / B frames implied by the GOP structure, the remaining ordinary I frames are extrapolated and P / B frames are generated by interpolation. The generation of I frames can refer to the target object and its location area information in the object dynamic information, that is, "the A4 paper with the sticky note is located in the left area of ​​the screen", "the A4 paper with the sticky note is located in the middle area of ​​the screen"; the generation of P / B frames should be combined with the object dynamic behavior in the foreground target object behavior information, that is, "the A4 paper with the sticky note moves from the left side of the screen to the central area", "the A4 paper with the sticky note moves from the center of the screen to the left area". All the generated image frames are encoded in the standard H.264 format, and the H.264 video with the same semantics as the original video can be restored.

[0110] Based on the same inventive concept, an embodiment of the present invention provides a video transmission device based on key frame semantics, which is applied to a semantic communication system for transmitting an H.26X video stream, wherein the semantic communication system includes an H.26X video stream transmitter and an H.26X video stream receiver; the device includes:

[0111] The video decoding module is used for the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream;

[0112] The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, common I frames and predicted frames;

[0113] An extraction module is used for the H.26X video stream sending end to extract semantic information of the IDR frame through a graph-based text model fine-tuned based on a local semantic knowledge base of the sending end, and use the semantic information as the background semantic information of the H.26X video stream;

[0114] A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the common I frame in combination with the semantic information of the IDR frame;

[0115] The second determination module is used for the H.26X video stream transmitter to extract the prediction error and motion vector information from the prediction frame, and obtain the dynamic behavior information of the foreground target object in combination with the position area information of the foreground target object;

[0116] A transmission module, used for the H.26X video stream sending end to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text;

[0117] The first recovery module is used for the H.26X video stream receiving end to recover the IDR frame by using the received background semantic information through the Wensheng graph large model fine-tuned based on the local semantic knowledge base of the receiving end;

[0118] The second recovery module is used for the H.26X video stream receiving end to recover the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information;

[0119] The video encoding module is used for the H.26X video stream receiving end to encode the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain the restored H.26X video stream.

[0120] Based on the same inventive concept, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the video transmission method based on key frame semantics described in any of the above embodiments.

[0121] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the video transmission method based on key frame semantics described in any of the above embodiments are implemented.

[0122] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the video transmission method based on key frame semantics described in any of the above embodiments.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0125] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0126] The above is a detailed introduction to a video transmission method based on key frame semantics provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video transmission method based on key frame semantics, characterized in that: A semantic communication system for transmitting an H.26X video stream, the semantic communication system comprising an H.26X video stream transmitter and an H.26X video stream receiver; the method comprising: The H.26X video stream sender decodes the H.26X video stream through the H.26X video decoder and extracts all image frames in the H.26X video stream; The H.26X video stream transmitter classifies all extracted image frames to determine IDR frames, common I frames, and predicted frames; The H.26X video stream transmitter extracts semantic information of the IDR frame through a graph-based text model fine-tuned based on a local semantic knowledge base of the transmitter, and uses the semantic information as background semantic information of the H.26X video stream; The H.26X video stream transmitter determines the foreground target object and the location area information of the foreground target object from the ordinary I frame in combination with the semantic information of the IDR frame; The H.26X video stream transmitter extracts the prediction error and motion vector information from the prediction frame, and combines the location area information of the foreground target object to obtain the dynamic behavior information of the foreground target object; The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text; The H.26X video stream receiver recovers the IDR frame by using the received background semantic information through a fine-tuned large model of the text graph based on the local semantic knowledge base of the receiver; The H.26X video stream receiving end recovers the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information; The H.26X video stream receiving end encodes the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain a restored H.26X video stream.

2. The video transmission method based on key frame semantics according to claim 1, characterized in that: The H.26X video stream receiver recovers the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information, including: Recovering a normal I frame based on the location area information of the foreground target object in the received dynamic behavior information and the recovered IDR frame; Based on the prediction error and motion vector information in the dynamic behavior information and the restored ordinary I frame, the predicted frame is restored.

3. The video transmission method based on key frame semantics according to claim 1 or 2, characterized in that: In addition to extracting all image frames from the H.26X video stream, the H.26X video stream transmitter also determines the frame rate and GOP structure information of the H.26X video stream; The H.26X video stream transmitter transmits the frame rate information and GOP structure information of the H.26X video stream; The H.26X video stream receiver recovers the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information, including: The H.26X video stream receiving end determines the total number of image frames restored per second based on the received frame rate information; The H.26X video stream receiver determines the intervals between I frames, P frames and B frames in a picture group according to the received GOP structure information, and obtains the number and arrangement of I frames, P frames and B frames that need to be restored per second in combination with the total number of image frames restored per second; According to the number and arrangement of I frames, P frames and B frames that need to be restored per second, ordinary I frames and predicted frames are restored based on the restored IDR frames and the received dynamic behavior information.

4. The video transmission method based on key frame semantics according to claim 1, characterized in that: The H.26X video stream transmitter transmits the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text, including: The H.26X video stream transmitter combines the local semantic knowledge base of the transmitter to semantically encode the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object to obtain a semantic encoding result, and transmits the semantic encoding result in the form of text.

5. The video transmission method based on key frame semantics according to claim 4, characterized in that: The method further comprises: The H.26X video stream receiving end combines the receiving end local semantic knowledge base to semantically decode the received semantic encoding result to obtain the received background semantic information and the received dynamic behavior information.

6. The video transmission method based on key frame semantics according to claim 1, characterized in that: The semantic communication system further includes: a cloud-based shared knowledge base; The H.26X video stream sender builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base; The H.26X video stream receiver builds a local semantic knowledge base based on the cloud shared knowledge base and keeps it synchronized with the cloud shared knowledge base.

7. A video transmission device based on key frame semantics, characterized in that: A semantic communication system for transmitting an H.26X video stream, the semantic communication system comprising an H.26X video stream transmitter and an H.26X video stream receiver; the device comprising: The video decoding module is used for the H.26X video stream transmitter to decode the H.26X video stream through the H.26X video decoder and extract all image frames in the H.26X video stream; The classification module is used by the H.26X video stream transmitter to classify all extracted image frames and determine IDR frames, common I frames and predicted frames; An extraction module is used for the H.26X video stream sending end to extract semantic information of the IDR frame through a graph-based text model fine-tuned based on a local semantic knowledge base of the sending end, and use the semantic information as the background semantic information of the H.26X video stream; A first determination module is used for the H.26X video stream transmitter to determine the foreground target object and the location area information of the foreground target object from the common I frame in combination with the semantic information of the IDR frame; The second determination module is used for the H.26X video stream transmitter to extract the prediction error and motion vector information from the prediction frame, and obtain the dynamic behavior information of the foreground target object in combination with the position area information of the foreground target object; A transmission module, used for the H.26X video stream sending end to transmit the background semantic information of the H.26X video stream and the dynamic behavior information of the foreground target object in the form of text; The first recovery module is used for the H.26X video stream receiving end to recover the IDR frame by using the received background semantic information through the Wensheng graph large model fine-tuned based on the local semantic knowledge base of the receiving end; The second recovery module is used for the H.26X video stream receiving end to recover the common I frame and the predicted frame according to the recovered IDR frame and the received dynamic behavior information; The video encoding module is used for the H.26X video stream receiving end to encode the restored IDR frame, the restored common I frame and the restored prediction frame through the H.26X video encoder to obtain the restored H.26X video stream.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executed, the processor implements the steps in the video transmission method based on key frame semantics as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the video transmission method based on key frame semantics as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the video transmission method based on key frame semantics described in any one of claims 1-6 are implemented.

Images