Multi-path embedded video coding method and system based on generative artificial intelligence

Abstract

The invention provides a multi-path embedded video coding method and system based on generative artificial intelligence. The multi-path embedded video coding method comprises the following steps: firstly, coding input cue words and demand parameters, and outputting cue word vectors and demand parameter code streams; then decoding the cue word vector and the demand parameter code stream, and outputting a feature vector and a feature map; extracting the feature vector, the feature map and the representation guide vector to obtain multi-embedded feature information; reconstructing a picture by using the obtained multi-embedded feature information through a generative AI and obtaining a video to be iterated; performing video quality evaluation on the video to be iterated; and generating a binary code stream for transmission through the current cue word vector, the demand parameter code stream and the multi-embedded feature information input. According to the method, important information characteristics are transmitted in a highly abstracted manner, and a high-definition video is reconstructed at a receiving end, so that the storage space required by video coding and the size of a transmission information stream are greatly reduced, and higher-definition transmission under a narrower bandwidth is realized or video coding under an extremely low bandwidth becomes possible.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and video coding technology, and in particular to a technical solution for multi-path embedded video coding and video decoding based on generative artificial intelligence. Background Art

[0002] In the field of video coding and decoding research, how to reduce the information flow in the video coding and storage process is an important topic. Although many related technologies have been widely used, it is still a difficult problem to further significantly reduce the storage occupancy and the size of the transmission information flow. At present, the difficulty of video coding compression is gradually increasing, and the compression efficiency is constantly decreasing. In the field of video coding and decoding compression, traditional methods mainly use entropy coding, the correlation between the previous and next frames and the correlation within the frame based on image pixels to obtain and characterize the motion characteristics of related pixels by relying on pixel-level information and traditional statistical calculations.

[0003] Therefore, traditional codec design still needs to store and transmit a large amount of motion information, related feature pixels, and key complete frame image data. Due to the limits of information transmission theory, the performance of traditional codec methods is gradually approaching the limit. In addition, traditional video coding methods cannot perform customized encoding based on the semantic information and content of the video, nor can they perform customized decoding based on the needs of the decoding user.

[0004] When applied to abstract or personalized expressions, traditional video encoding and decoding technologies are unable to cope with the situation. Especially in extremely low-bandwidth network communications, how to transmit important and critical video information from the sender to the receiver while occupying as few resources as possible is a difficult problem. Traditional methods are usually time-consuming or even impossible to complete. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to transmit important and critical video information from a sending end to a receiving end while occupying as few resources as possible.

[0006] The present invention solves the above technical problems through the following technical means: a multi-path embedded video encoding method based on generative artificial intelligence, including encoding steps and decoding steps, wherein prompt words and required parameters from both the decoding end and the encoding end exist, and the encoding includes the following steps:

[0007] Step S101, inputting the prompt words and requirement parameters of the decoding end and the prompt words and requirement parameters of the encoding end into the prompt word and requirement parameter encoding module (10), encoding the input prompt words and requirement parameters and outputting them into prompt word vectors and requirement parameter code streams, and transmitting them to the prompt word vector decoding module (20), while the output prompt word vectors and requirement parameter code streams are also input into the semantic information encoding and code stream compression module (60), and are packaged as the input of this part into a binary code stream that can be transmitted;

[0008] Step S102, after obtaining the prompt word vector and the required parameter code stream, the prompt word vector decoding module (20) decodes them and outputs a feature vector and a feature map;

[0009] Step S103, after the feature vector and feature map are output, the feature vector, feature map and representation guidance vector are input into the video extraction module (30), and multiple embedded feature information is obtained through the video extraction module (30);

[0010] Step S104, inputting the obtained multi-embedded feature information into the first generative AI decoding module (40), reconstructing the picture through generative AI and obtaining the video to be iterated;

[0011] Step S105, receiving the video to be iterated and evaluating the video quality thereof. If the evaluation obtains the corresponding index parameter and determines that the video does not need to be iterated, the iteration is completed. If the evaluation obtains the corresponding index parameter and determines that the video still needs to be iterated;

[0012] Step S106, at this time, the semantic feature extraction and internal optimization iteration are completed, the current prompt word vector and the required parameter code stream and the multi-embedded feature information are input into the semantic information encoding and code stream compression module (60) to generate a binary code stream for transmission, and the encoder stops encoding.

[0013] As an optimized technical solution, step S101 specifically includes:

[0014] Inputting the decoding end prompt words and requirement parameters and the service (encoding end) prompt words and requirement parameters into the embedding layer of the prompt words and requirement parameter encoding module (10) for mode conversion;

[0015] Then input into the Encoder of the prompt word and requirement parameter encoding module (10), and output as prompt word vector and requirement parameter code stream, wherein the Encoder uses a basic learner based on the Transformer learner idea to perform feature extraction;

[0016] In the step S102, the prompt word vector decoding module (20) has a basic characterizer and can output a feature vector and a feature map.

[0017] As an optimized technical solution, step S103 specifically includes:

[0018] The original video information is input into a video segmentation discriminator of a video extraction module (30), and the video segmentation discriminator reads the entire video or a very short segment of a real-time video and determines whether there is editing or drastic scene switching, thereby dividing the entire video into smaller video segments;

[0019] The smaller video segments after segmentation are input into the feature extraction module. In the feature extraction module, the expression and expression strength of different feature extraction modules in the feature extraction module are partially selected according to the results of phase weight encoding. After feature extraction, each small video segment generates a corresponding multi-embedded feature information representation block. By stacking and splicing several multi-embedded feature information representation blocks, the corresponding multi-embedded feature information of the entire video is obtained.

[0020] As an optimized technical solution, in step S103, reading the entire video or a very short segment of the real-time video and determining whether there is editing or drastic scene switching, and dividing the entire video into smaller video segments includes:

[0021] In the default state, the video segmentation discriminator uses T seconds as a marking point, calculates the structural consistency parameters of the current frame and the previous frame and judges that if the structural similarity between the two exceeds a certain fixed initial threshold, the similarity is considered acceptable. The current frame and the previous frame are continuous continuous pictures. If the current predetermined number of frames are all continuous pictures, they are all packaged into 1 group of video clips. If the current predetermined number of frames are not all continuous pictures, they are segmented and packaged into several groups of video clips according to the SSIM threshold, where the total number of each continuous video frame is less than or equal to the set slicing time x video frame rate. The entire video is divided into smaller video clips by using a segmentation discriminator, and the small video clips generated after segmentation are then input into the feature extraction module.

[0022] As an optimized technical solution, in step S103, in the feature extraction module, based on the result of phase weight encoding, partially selecting the expression and expression strength of different feature extraction modules in the feature extraction module specifically includes:

[0023] First, the video clips are input into the video content understanding module, which extracts features of semantic entities and background and foreground information in the video clips and generates representation prompt words for the generative AI to complete video decoding and generate the video to be iterated.

[0024] At the same time, after the phase weight coding completes the weighted expression, feature extraction modules with different degrees of expression are selected according to the needs of the decoding end. If it is necessary to complete the pixel-level complete information expression, each feature extraction module will be fully called. If only the most abstract semantic feature information video needs to be expressed, only the representation prompt words generated by the video content understanding module are used. For the remaining basic parameters, the default parameters are directly called in the first generative AI decoding module (40) without customization and pixel-level precise expression. If it is necessary to modify the video feature representation strength, video restoration degree and personalized expression degree according to the needs of the decoding end, the input prompt word information is expressed through phase weight coding and different feature extraction and feature descriptors are called. At the same time, different feature extraction and feature descriptors are shown with different expression strengths according to the different phase weight expression strengths.

[0025] As an optimized technical solution, step S104 specifically includes:

[0026] By decomposing the multi-embedded feature information into blocks, an embedded feature semantic information representation block is obtained. The representation prompt word is input into the representation prompt word embedding layer to complete the embedding operation of the prompt word to generate high-dimensional array information that can be read by the model. If the edge information feature map, motion information feature map, semantic feature map, semantic control feature map, semantic internal consistency feature map, semantic super-resolution feature map and personalized style feature vector exist, they are correspondingly input into the auxiliary representation and guidance embedding layer. In the auxiliary representation and guidance embedding layer, the above-representable feature maps and feature vectors are uniformly encoded and embedded, and output as a high-dimensional array in a unified format to ensure that the input specifications and data structure are unified in the auxiliary representation hierarchical expression module. The basic control parameters are input into the parameter loading model to load customized parameter values. If the auxiliary representation and guidance embedding layer does not receive any information, the information of the same data structure is directly constructed and filled with the number 0. If some values ​​are missing in the parameter loading model, they are loaded with default parameters. If the representation prompt word embedding layer does not receive any information, the first generative AI decoding module (40) stops working.

[0027] After completing the data loading of the above three parts, the data generated by the representation prompt word embedding layer is input into the representation prompt word expression core module in the generative AI video reasoning model; the data generated by the auxiliary representation and guidance embedding layer is input into the auxiliary representation hierarchical expression module in the generative AI video reasoning model to express its corresponding features according to different personalized features and representation strengths. Finally, the generative AI model is used to generate the video to be iterated / reconstructed.

[0028] As an optimized technical solution, step S106 specifically includes:

[0029] Insert the head and tail of the iterated multi-embedded feature information into the head identification information and the tail identification information. The inserted identification information is mainly used to mark the head and tail of the information fragment. At the same time, the head contains basic description information of the multi-embedded semantic feature semantic information. After the information is recorded, the multi-embedded feature information representation block 1 to the semantic information representation block n are losslessly compressed. After the lossless compression is completed, the compressed multi-embedded feature information compressed representation block is input into the multi-embedded feature information compressed representation block similarity comparison mapping model. The model compares the similarity between the feature representation blocks and replaces the representation blocks with different representations and high similarity through transfer learning.

[0030] When the module is executed normally, several mapping blocks with smaller volumes are obtained. After the migration mapping is completed, the header identification information and the tail identification information are spliced ​​to the output content and entropy coding is performed to encode a pure binary bit stream. When the corresponding binary code stream is obtained, the encoder part is executed.

[0031] As an optimized technical solution, the decoding step specifically includes:

[0032] Step S107, the decoder receives the transmitted binary code stream, inputs the binary code stream into the code stream decoding module (70), and decodes the multi-embedded feature information and the prompt word vector and the required parameter code stream, wherein the prompt word vector and the required parameter code stream can be defaulted;

[0033] Step S108, input the obtained multi-embedded feature information, prompt word vector and required parameter bitstream into the second generative AI decoding module (80), mainly using the multi-embedded feature information, the prompt word vector and the required parameter bitstream as guidance and configuration parameters, to generate videos with different degrees of personalized or specialized semantics required by the corresponding receiver. If the prompt word vector and the required parameter bitstream are missing, the original video is directly generated and expressed.

[0034] The present invention also provides a system corresponding to any of the above-mentioned multi-path embedded video encoding methods based on generative artificial intelligence, including an encoder and a decoder, wherein the encoder includes:

[0035] The requirement parameter encoding module (10) is used to receive the prompt word and the requirement parameter at the decoding end and the prompt word and the requirement parameter input at the encoding end, encode the input prompt word and the requirement parameter and output them into a prompt word vector and a requirement parameter code stream, and transmit them to the prompt word vector decoding module (20). At the same time, the output prompt word vector and the requirement parameter code stream are also input into the semantic information encoding and code stream compression module, and are packaged as the input of this part into a binary code stream that can be transmitted;

[0036] A prompt word vector decoding module (20) is used to decode the prompt word vector and the required parameter code stream after obtaining them, and output a feature vector and a feature map;

[0037] A video extraction module (30) is used to receive the feature vector and feature map output by the prompt word vector decoding module (20), and obtain multi-embedded feature information after processing;

[0038] A first generative AI decoding module (40) is used to receive the multi-embedded feature information, reconstruct the picture through generative AI and obtain the video to be iterated;

[0039] A video quality comprehensive evaluation module (50) is used to receive a video to be iterated and evaluate its video quality. If the evaluation obtains corresponding index parameters and determines that the video does not need to be iterated, the iteration is completed. If the evaluation obtains corresponding index parameters and determines that the video still needs to be iterated in this module,

[0040] A semantic information encoding and code stream compression module (60) is used to receive the current prompt word vector and the required parameter code stream and multiple embedded feature information after the semantic feature extraction and internal optimization iteration are completed, and generate a binary code stream for transmission;

[0041] The decoder comprises:

[0042] A code stream decoding module (70) is used to receive the binary code stream transmitted by the encoder, and decode the multi-embedded feature information and the prompt word vector and the required parameter code stream, wherein the prompt word vector and the required parameter code stream can be defaulted;

[0043] The second generative AI decoding module (80) is used to generate videos with different degrees of personalized or specialized semantics required by the corresponding receiver based on the multi-embedded feature information, the prompt word vector and the required parameter bitstream as guidance and configuration parameters. If the prompt word vector and the required parameter bitstream are missing, the original video is directly generated and expressed.

[0044] The advantages of the present invention are: the present invention highly abstracts the important information features of transmission, and reconstructs high-definition video at the receiving end, greatly reducing the storage space required for video encoding and the size of the transmission information flow, achieving higher-definition transmission under narrower bandwidth or making video encoding under extremely low bandwidth possible, and can perform customized encoding and decoding according to the needs of the decoding side. By using different feature representation vectors adopted by this method, variable semantic expressions can be performed for different groups of people, different audiences, and even for machines or industrial equipment, effectively solving the problem that traditional encoding methods cannot customize feature expressions according to needs, specifically including:

[0045] In terms of time domain information processing, traditional encoders use key frames and reference frames to eliminate redundancy by referring to repeated information between frames in the time domain information. The present invention uses the method of understanding-extracting information-reconstructing to read and understand key information such as background, foreground, semantics, etc. in each continuous video segment, extract relevant features and edge information, reconstruct and achieve pixel-level image restoration, select partial video feature expression, and personalized video reconstruction. At the same time, the method of mutual migration of feature blocks is used to further reduce information redundancy between video segments, and further reduce storage and bandwidth usage;

[0046] In terms of personalized expression, traditional encoders perform pixel-level video restoration in a lossy or lossless manner, and cannot change the video content, such as characters, semantics, style, duration, and rhythm, according to the needs of the sending end and the receiving end. Using the present invention, personalized encoding of the video can be achieved by inputting prompt words at the sending end and the receiving end. For example, if you need to delete the role of a certain character, you only need to input the corresponding prompt words at the sending end, and the encoding system will extract and discard the characteristics and semantic information of the character, and the decoded video will no longer contain the character. If you need to change the style of the video or adapt to young users, you only need to input the corresponding prompt words, and the encoding system will adjust the video style characteristics and process the inappropriate content (such as deletion, mosaic, occlusion, etc.), so as to achieve personalized expression. Similarly, for the personalized needs of video duration and clarity, the length and clarity of the video after encoding and decoding can also be controlled by the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a structural diagram of an encoder of a multi-path embedded video coding system based on generative AI according to an embodiment of the present application;

[0048] Figure 2 is a decoder structure diagram of a multi-path embedded video coding system based on generative AI in an embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of the overall framework flow of the multi-path embedded video coding system based on generative AI in this application;

[0050] Figure 4 It is the internal structure diagram of the video extraction module of this application;

[0051] Figure 5 It is a diagram of the multi-embedded feature information structure of the present application;

[0052] Figure 6 This is the internal structure diagram of the generative AI decoding module of this application;

[0053] Figure 7 It is the internal structure diagram of the video quality comprehensive evaluation module of this application;

[0054] Figure 8 This is the internal structure diagram of the video semantic optimization module of this application;

[0055] Fig. 9 This is the internal structure diagram of the semantic information encoding and code stream compression module of the present application;

[0056] Fig.10 It is the internal structure diagram of the multi-embedded feature optimization module of the present application;

[0057] Fig.11 This is the internal structure diagram of the prompt word and requirement parameter encoding module of this application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1 and Figure 2 As shown, a multi-path embedded video coding system based on generative AI according to an embodiment of the present application is divided into two parts: an encoder and a decoder.

[0060] The video encoder includes a prompt word and demand parameter encoding module 10, a prompt word vector decoding module 20, a video extraction module 30, a generative AI decoding module 40, a video quality comprehensive evaluation module 50, and a semantic information encoding and code stream compression module 60, wherein the output end of the prompt word and demand parameter encoding module 10 is simultaneously connected to the input end of the prompt word vector decoding module 20 and the semantic information encoding and code stream compression module 60, the output end of the prompt word vector decoding module 20 is simultaneously connected to the generative AI decoding module 40 and the semantic information encoding and code stream compression module 60 after passing through the video extraction module 30, and the output end of the generative AI decoding module 40 is connected to the video quality comprehensive evaluation module 50. The input end of the prompt word and demand parameter encoding module 10 is used to receive the prompt word and demand parameter input by the user (decoding end) / service (encoding end), and the output end of the semantic information encoding and code stream compression module 60 is used to output a binary code stream to the video decoder.

[0061] The video decoder includes a code stream decoding module 70 and a generative AI decoding module 80 connected to each other. The code stream decoding module 70 is used to receive the binary code stream transmitted by the video encoder, and the generative AI decoding module 80 outputs a video reconstructed based on semantics.

[0062] See also Figure 3 The method for encoding using the multi-path embedded video encoding system based on generative AI of the present invention comprises the following steps:

[0063] 1. Coding:

[0064] The initial state and workflow of the encoder are defined by the original semantic information video and the prompt words and required parameters from the user (decoding end) / from the service (encoding end). There are four situations: the original semantic information video exists, the prompt words and required parameters from the user (decoding end) exist, and the prompt words and required parameters from the service (encoding end) exist; the original semantic information video exists, the prompt words and required parameters from the user (decoding end) exist, and the prompt words and required parameters from the service (encoding end) do not exist; the original semantic information video exists, the prompt words and required parameters from the user (decoding end) do not exist, and the prompt words and required parameters from the service (encoding end) exist; the original semantic information video exists, the prompt words and required parameters from the user (decoding end) do not exist, and the prompt words and required parameters from the service (encoding end) do not exist. The original semantic information video is a required input for this system, and the prompt words and required parameters from the user (decoding end) / from the service (encoding end) are non-essential parameters. In the following introduction, it is expressed as if the prompt words and required parameters from the user (decoding end) / from the service (encoding end) both exist.

[0065] In the case where both the prompt word and the required parameters from the user (decoding end) / from the service (encoding end) are present, encoding includes the following steps:

[0066] Step S101: Input the user (decoding end) prompt words and requirement parameters and the service (sending encoding end) prompt words and requirement parameters. Fig.11 , specifically including:

[0067] The user (decoding end) prompt words and demand parameters and the service (encoding end) prompt words and demand parameters are input into the embedding layer of the prompt word and demand parameter encoding module 10 for modal conversion, that is, text information such as words and numbers are converted into feature maps through the embedding layer;

[0068] Then input it into the encoder of the prompt word and requirement parameter encoding module 10, and output it as the prompt word vector and the requirement parameter code stream. Among them, the encoder must use a basic learner based on the idea of ​​including but not limited to similar Transformer learners to perform feature extraction. The Transformer learner has the characteristics of position encoding and multi-head self-attention mechanism, which can unify and effectively extract global feature information and local feature information.

[0069] Step S102, after obtaining the prompt word vector and the required parameter code stream, they are input into the prompt word vector decoding module 20 for decoding. The module has a basic characterizer similar to the Decoder architecture and can output feature vectors and feature maps.

[0070] Step S103, after the output feature vector and feature map are completed, they are input into the video extraction module 30 together with the characterization guidance vector, and at the same time, refer to Figure 4 The video extraction module 30 further includes a video segment discriminator, a phase weight coding, and a feature extraction module. The feature vector, the feature map, and the characterization guidance vector are input into the phase weight coding part of the video extraction module 30, and the phase weight coding guides the partial or complete expression of the feature extraction of the video clip.

[0071] In this step, the original video information is input into the video segmentation discriminator of the video extraction module 30, which reads the entire video or a very short segment of the real-time video (sliced ​​according to the real-time requirement) and determines whether there is editing or drastic scene switching. The segmentation discriminator has adjustable parameters and default parameter functions. By default, the video segmentation discriminator uses T seconds as a mark point (for example, 2 seconds. For videos with high real-time requirements, the mark point can also be set directly according to the number of frames, with a minimum of 1 frame), calculates the structural consistency parameters of the current frame and the previous frame (the structural consistency parameter calculation method includes but is not limited to the structural similarity index SSIM (Structural Similarity Index Measurement, SSIM), multi-scale structural similarity (Mult-Scale Structural Similarity Index Measure, MS-SSIM) and other methods) and judges that if the structural similarity between the two exceeds a certain fixed initial threshold (the threshold is not automatically updated, and here the structural consistency calculation method SSIM is taken as an example, the threshold of SSIM includes but is not limited to 0.6, 0.7 and other values ​​that can be taken within the definition domain; at the same time, the exceeding here refers to the convergence of the two images in structural consistency. If the parameter obtained by a certain calculation method is that the smaller the value, the higher the convergence, then it becomes less than here, and the not exceeding in the following text becomes greater than or equal to), then it is judged that the similarity is acceptable, and the current frame and the previous frame are continuous continuous pictures. If the video is 60fps, 2s corresponds to 120 frames. If the current 120 frames are all continuous pictures, they are all packaged into one group of video clips. If the video is 60fps, 2s corresponds to 120 frames. If the current 120 frames are not all continuous pictures, they are segmented and packaged into several groups of video clips according to the SSIM threshold. Among them, the total number of each continuous video frame is less than or equal to the set slice time x video frame rate. The entire video is divided into smaller video clips by using a segmentation discriminator, and the small video clips generated after segmentation are then input into the feature extraction module.

[0072] In the feature extraction module, the expression and expression intensity of different feature extraction modules in the feature extraction module will be partially selected according to the results of phase weight encoding. After the video clip is input into the feature extraction module, it will be input into the video content understanding module respectively. The video content understanding module will complete the feature extraction of the semantic entities, background foreground and other information in the video clip and generate representation prompt words for the subsequent generative AI to complete video decoding and generate the video to be iterated. At the same time, under the weighted expression completed by phase weight encoding, feature extraction modules with different degrees of expression will be selected according to the needs of the user / decoding end. If the pixel-level complete information expression is required, the edge information descriptor, motion information descriptor, semantic extraction module, semantic control feature extraction module, semantic internal consistency feature extraction module, semantic super-resolution feature extraction module, personalized style feature extraction module, video basic information reading and basic control parameter fine-tuning module will be fully called. If only the most abstract semantic feature information video needs to be expressed, there is no need to call the above modules, and only the video content understanding module needs to be used to generate representation prompt words. For the remaining basic parameters, the default parameters are directly called in the generative AI decoding module 40 without customization and pixel-level precise expression. If it is necessary to modify the video feature representation strength, video restoration degree and personalized expression degree according to the user / decoding end requirements, the input prompt word information is expressed through phase weight encoding and different feature extraction and feature descriptors are called. At the same time, different feature extraction and feature descriptors are shown with different expression strengths according to the different phase weight expression strengths.

[0073] It should be noted that the optional expression modules and feature descriptors above represent different purposes and functions.

[0074] The edge information feature descriptor is mainly used to extract all edge contour information and features in the picture, and select the contour feature expression strength of different semantic entities according to the phase weight coding result and generate the edge information feature map. The motion information descriptor contains three sub-parts: motion vector direction feature descriptor, optical flow motion estimation descriptor and semantic trajectory and relative position descriptor. The motion vector direction feature descriptor is mainly used to extract the feature information of motion vectors and their directions of different semantic entities in the video clip and generate the motion vector direction feature map; with the support of phase weight coding, the motion vector direction feature expression strength of different semantic entities can also be selected according to the representation guidance vector, so as to purposefully adjust or reduce the total amount of transmitted information. The optical flow motion estimation descriptor is used to describe the optical flow motion information features of motion semantics in different foregrounds and backgrounds in the video clips. This feature is used to assist in expressing the global motion information and motion blur control in the picture. The semantic extraction module includes foreground semantic entity extraction, background semantic entity extraction and language and text extraction, which is mainly used to extract the semantics of foreground semantic entities, background semantic entities and language and text entities in the video clips. The semantic entities of the foreground and background can be extracted and used for the subsequent video reconstruction and semantic-based super-resolution of the generative AI decoder, and can also be used for the compression of semantic duplication and redundancy of multiple video clips (mainly manifested in the compression of multiple embedded feature information representation blocks in step S130 to generate mapping blocks). The language and text entity semantic extraction module can extract the text content and its meaning feature information, which is used to restore the corresponding text with higher definition and can be personalized and modified into other languages ​​and texts in different language environments (such as Chinese to English, Japanese to Hebrew, etc.) according to the requirements of the representation guidance vector. The semantic control feature extraction module includes semantic trajectory and relative position descriptors, which are mainly used for the accurate description and guidance of the motion trajectory of local and specific semantics; through the user / decoding end prompt words or the representation guidance vector under the different intensities and semantic entities of phase weight coding, the accurate motion trajectory description of the semantic entity and semantic entity set of the specific target is achieved, and it is also used to ensure the consistency and personalized correction of the motion trajectory of the semantic entity. The semantic internal consistency feature extraction module is mainly used to ensure the internal consistency of the semantic entity between different frames generated by the generative AI decoder, and to keep the pixel level the same and the structure consistent within the semantics. The semantic super-resolution feature extraction module is mainly used for image denoising and super-resolution based on semantic information. The module can simultaneously extract the pixel noise features and semantic entity noise features of the specified semantic entity area and generate corresponding feature maps. At the same time, it will also extract low-resolution feature maps for video key frames for subsequent super-resolution denoising and pixel-level video image restoration of the specified semantic entity in the generative AI decoder.The personalized style feature extraction module is mainly used to extract the style and style feature information of the overall video clip, which can be used to accurately restore the style, contrast, dynamic range, highlight and dark details of the video; it can also be used to characterize different style conversion features under the guidance of prompt word vectors for generating personalized style expression features and their corresponding picture details in the generative AI decoder. The video basic information reading and basic control parameter fine-tuning module is used to read basic information such as video frame number, frame rate, screen ratio, color space, and control parameters for fine-tuning some semantic entities. The above control parameters and basic video information will be input into the generative AI decoder to control the consistency of the generated video basic parameter information.

[0075] Specifically, when it comes to how to use the above-mentioned feature descriptors and feature maps to solve the problem of inter-frame content redundancy, the first thing that needs to be made clear is that in a certain video clip, if you want to accurately restore all the semantics and motion information in the foreground and background of the original video, you need to partially or completely call the following feature descriptors and modules: motion information descriptor, edge information feature descriptor, optical flow motion estimation descriptor, semantic control feature extraction module, semantic extraction module, language and text extraction module, and semantic internal consistency feature extraction module. By calling the above-mentioned feature descriptors and modules, the video clip is abstractly extracted as a record of the entity interaction of multiple foreground semantics + background semantics. It is no longer concretely stored in the form of multiple continuous frame images. By extracting and separating the entity semantics, it is no longer necessary to use pixel records, transmission and restoration, but to directly restore the semantic entity and reconstruct the video clip. Therefore, the semantic entity motion information recorded between frames is extracted and stored by the motion vector direction feature descriptor, optical flow motion estimation descriptor, semantic trajectory and relative position descriptor, foreground semantic entity extraction, and background semantic entity extraction. In addition, the edge information feature descriptor, semantic understanding module, language and text extraction module and semantic internal consistency feature extraction module are used to read and obtain the specific details of the semantic entity and the necessary description information and overlapping relationship between the front and back, which can effectively reduce the information duplication of semantic entities between frames. So far, through the bidirectional feature description and semantic entity understanding, extraction and motion information extraction, the redundant information between frames in the video clip has been removed.

[0076] It is necessary to add that the video basic information reading and basic control parameter fine-tuning module can also play an additional regulatory role. If the user / decoder needs to reduce the total length of a video of 1 hour to 30 minutes or expand it to 2 hours, this part will read the original video length and guide the expression of different feature extraction modules and descriptors according to the feature importance level and constrain the video generation length in the subsequent video reconstruction module, gradually and finally achieve the purpose of personalized control of duration expression through iteration. If the user / decoder needs to delete a certain part, a certain semantics, a certain fragment, a certain character, etc. and reduce the total length, keep the total length unchanged or increase the total length, this part will guide the expression of some modules and control the overall length of the reconstructed / to-be-iterated video. If the encoder requires real-time transmission, the control parameters of this part can also control the specified number of encoding frames, where the minimum transmission size is 1 frame, so as to improve the encoding efficiency and ensure the coherence of encoding and transmission.

[0077] After completing the video extraction of the video, multiple embedded feature information will be obtained. Among them, multiple embedding is reflected in different representations and feature extraction modules that can be selected according to the needs of the user / decoding end. The multiple embedded feature information includes Figure 5 In the structure shown, each short video segment will generate a corresponding multi-embedded feature information representation block after feature extraction, and stacking and splicing several multi-embedded feature information representation blocks will correspond to the multi-embedded feature information of the entire video segment.

[0078] Step S104, when the multi-embedded feature information is obtained, part of the data is input into the generative AI decoding module 40 for generating a video reference to be iterated. Figure 6As shown. The embedded feature semantic information representation block is obtained by disassembling the multi-embedded feature information in blocks. Specifically, in each embedded feature semantic information representation block, the representation prompt word is input into the representation prompt word embedding layer to complete the embedding operation of the prompt word to generate high-dimensional array information that can be read by the model. As long as the edge information feature map, motion information feature map, semantic feature map, semantic control feature map, semantic internal consistency feature map, semantic super-resolution feature map and personalized style feature vector exist, they are input into the auxiliary representation and guidance embedding layer accordingly. In the auxiliary representation and guidance embedding layer, the above-representable feature maps and feature vectors will be uniformly encoded and embedded, and output as a high-dimensional array in a unified format to ensure that the input specifications and data structures are unified in the auxiliary representation hierarchical expression module. The basic control parameters are input into the parameter loading model to load customized parameter values. Among them, if the auxiliary representation and guidance embedding layer does not receive any information, the information of the same data structure is directly constructed and filled with the number 0. If there is a partial missing value in the parameter loading model, it is loaded with the default parameters. If the representation prompt word embedding layer does not receive any information, the generative AI decoding module 40 stops working. After completing the data loading of the above three parts, the data generated by the representation prompt word embedding layer is input into the representation prompt word expression core module in the generative AI video reasoning model; the data generated by the auxiliary representation and guidance embedding layer is input into the auxiliary representation hierarchical expression module in the generative AI video reasoning model to express its corresponding features according to different personalized features and representation strengths. Finally, the generative AI model is used to generate the video to be iterated / reconstructed. Among them, the video to be iterated refers to the video used for iterative optimization by the encoder part, and the reconstructed video refers to the final video presented to the user by the decoder part. The generative AI video reasoning model is mainly responsible for generating the reasoning framework structure of key frames or videos, including but not limited to generative base reasoning models based on Diffusion-like, VAE-like (VQ-VAE, AE), GAN-like, and multiple method fusion (Transformer-Diffusion, VAE-Diffusion, Transformer-GAN and GAN-Diffusion, etc.).

[0079] Step S105: After the AI ​​decoding module 40 completes the generation of the video to be iterated, it is input into the video quality comprehensive evaluation module 50. Figure 7 In the video quality comprehensive evaluation module 50, there are 2 required input items and 2 optional input items, which are: original semantic information video and video to be iterated; user (receiving end) prompt words and required parameters and service (encoding end) prompt words and required parameters.

[0080] The original semantic information video will be input into the video image stacking embedding layer; the embedding layer will perform frame stacking operation on the original semantic information video and convert the stacked images into high-dimensional arrays using the Encoder-like base learner similar to the previous module description. The processing of the video to be iterated is the same as the processing of the original semantic information video, which also stacks the video frame images and inputs them into the Encoder-like base learner to convert the stacked images into high-dimensional arrays.

[0081] When processing user (receiving end) prompt words and demand parameters and service (encoding end) prompt words and demand parameters, they are input into the corresponding embedding layer to convert them from text and numbers into high-dimensional vectors or high-dimensional arrays.

[0082] The core idea of ​​the above embedding layer processing is to use a unified type of base learner to convert modal information such as video, image, text and number into a unified modal data that can be processed, feature extracted and analyzed. If the user (receiving end) prompt words and demand parameters and the service (encoding end) prompt words and demand parameters exist, the four high-dimensional arrays obtained by the user (receiving end) prompt words and demand parameters and the service (encoding end) prompt words and demand parameters as well as the original semantic information video and the video to be iterated in the embedding layer are used as input to the reference quality evaluation model. In the reference quality evaluation model, there is a gradient evaluation of the video image at the pixel level, feature level, semantic level and fragment level. Among them, the guidance expressed by the prompt words and demand parameters of the user / decoding end after the embedding layer is compared before and after, and the original semantic information video and the video to be iterated are evaluated in both directions. The evaluation contents include but are not limited to: brightness consistency, chromaticity consistency, color space consistency, contrast consistency, dynamic range consistency, pixel consistency, semantic consistency, painting style consistency, semantic recognizability consistency, emotional style consistency, language and text consistency, character action consistency, semantic relative position relationship consistency, semantic overlap consistency, foreground consistency, background consistency, semantic target trajectory consistency, semantic internal consistency, semantic internal coherence consistency, inter-frame contrast consistency, inter-frame optical flow feature consistency, inter-frame brightness consistency, inter-frame smoothness consistency, inter-frame semantic loss loss consistency, etc. The evaluation results obtained after the evaluation are expressed abstractly in the form of feature vectors and input into the comprehensive evaluation model. Similarly, a no-reference quality evaluation will be performed separately for the video to be iterated, and the high-dimensional array obtained by embedding the prompt words and demand parameters of the user (decoding end) / service (encoding end) will be input into the no-reference quality evaluation model. The evaluation indicators for the no-reference quality evaluation of the iterative video include but are not limited to: brightness, chroma, color space range, contrast, dynamic range, pixel resolution, picture contrast, semantic resolution, style expression, semantic recognizability, emotional style expression, language clarity, character action coherence, semantic relative position relationship expression, semantic overlap, foreground clarity, background clarity, semantic target trajectory description, semantic internal coherence, inter-frame contrast, feature extraction difficulty, inter-frame optical flow feature clarity, inter-frame brightness and darkness change, inter-frame smoothness, inter-frame semantic loss loss, etc. After completing the above no-reference evaluation, the no-reference quality evaluation result feature vector is obtained and input into the comprehensive evaluation large language model. The large model comprehensively gives the semantic guidance vector, representation guidance vector, and multi-embedded feature optimization guidance vector for the performance of the iterative video and the relevant feature performance of the original semantic information, for the next round of video iteration.Among them, the semantic guidance vector mainly acts on the original semantic information video, and is used to optimize and improve the original semantic information, and performs operations including but not limited to changing brightness, contrast, semantic clarity, etc. to improve the original semantic information video. The representation guidance vector mainly acts on the video understanding module, and is mainly used for tuning and adaptive strength expression of various feature extraction modules, feature descriptors, semantic understanding modules, etc. By optimizing feature extraction and semantic understanding, the expression effect and subjective and objective video quality of the video to be iterated are improved, and semantic-level and fragment-level optimization is completed for the video to be iterated. The multi-embedded feature optimization guidance vector mainly acts on the multi-embedded semantic features, and by fine-tuning the parameters, feature vectors, and feature maps of the multi-embedded semantics and its internal representation blocks, pixel-level optimization adjustment of the video to be iterated is achieved.

[0083] After the operation of the video quality comprehensive evaluation module 50 is completed, a judgment parameter will be generated, and it will be judged whether the iteration of the current video to be iterated has been completed. If the module obtains the corresponding index parameter through evaluation and judges that the video still needs to be iterated, it will judge whether the iteration is completed or not. If the video to be iterated judges whether the iteration is completed or not, the relevant difference parameters are calculated and sent to the semantic optimization module 91, the representation guidance module 92, and the multi-embedded feature optimization module 93, and the iteration continues. When the semantic optimization module 91 receives the relevant parameters, it optimizes the original semantic information video. If the parameter is default, no optimization is performed; when the representation guidance module 92 receives the relevant parameters, it optimizes the video extraction module 30. If the parameter is default, no optimization is performed; when the multi-embedded feature optimization module 93 receives the relevant parameters, it optimizes the multi-embedded feature information content expressed. If the parameter is default, no optimization is performed.

[0084] Specifically, the following are performed: semantic optimization module 91, representation guidance vector and multi-embedded feature optimization module 93. The function of the representation guidance vector has been described in S103 and will not be repeated here. Specifically, in the semantic optimization module 91, the corresponding Figure 8, input the original semantic information video into the video image stacking embedding layer, similar to step S105, stack the video images and perform the embedding operation to complete the modal conversion, and then input the Encoder after the embedding is completed. The Encoder here and the subsequent Decoder also include but are not limited to the Encoder-like structure and the Decoder-like structure in the Transformer. By encoding the original semantic video frame by frame, a set of feature maps of all frames is obtained. The feature map set contains complete information for each video frame image. If the reverse operation is performed at this time, the original frame image will be directly generated. The feature map set and the semantic optimization guidance vector are input into the semantic information optimization model, and the model will complete the feature-level and semantic-level optimization of the original video according to the guidance vector and obtain the optimized feature map set. At this time, the optimized feature map set is input into the Decoder and the optimized original semantic information video is decoded. Specifically to the multi-embedded feature optimization module 93, as Fig.10 By inputting the multi-embedded feature optimization guidance vector into the multi-embedded feature optimization model, the multi-embedded feature semantic representation block inside the multi-embedded semantic feature is optimized at the pixel level and feature level. The optimized multi-embedded feature information can achieve a better expression of the feature by modifying the internal feature map and feature vector.

[0085] It should be noted that the Encoder-like learners and Decoder-like representations mentioned include but are not limited to the base learners split from the Encoder-Decoder architecture in the Transformer architecture, the Encoder / Decoder base learners in the Bidirectional Encoder Representations from Transformers (BERT) architecture based on the Transformer architecture, and the Image Encoder / Decoder base learners in the Swin-Transformer architecture. As long as the Encoder / Decoder-like base learners and embedding layer network structures and Convolutional Neural Network (CNN)-like structures that can uniformly convert different modal information to each other are applicable to this patent proposal.

[0086] Step S106, when the video quality evaluation comprehensive module is completed, it is determined that the parameters meet the conditions, and the iteration is completed. At this time, the system enters the semantic information encoding and code stream compression module 60 to perform operations. When the system executes this step, the bidirectional iterative optimization of the original video and the video to be iterated is completed. Fig. 9As shown. The header and tail of the iterated multi-embedded feature information are inserted into the header identification information and the tail identification information. The inserted identification information is mainly used to mark the head and tail of the information fragment. At the same time, the header also contains basic description information of the multi-embedded semantic feature semantic information, such as: the number of multi-embedded feature representation blocks, the total size of the multi-embedded feature representation blocks and verification information. After the information is recorded, the multi-embedded feature information representation block 1 to the semantic information representation block n (n is the total number of representation blocks contained in a certain multi-embedded feature information) are losslessly compressed. Among them, the lossless compression method includes but is not limited to Huffman coding, compressing sparse matrices using triple representation method and cross linked list method and other lossless compression methods. After the lossless compression is completed, the compressed multi-embedded feature information compression representation block 1 to the multi-embedded feature information compression representation block n are input into the multi-embedded feature information compression representation block similarity comparison mapping model. The model can replace the representation blocks with different representations and high similarity by comparing the similarity between the feature representation blocks through transfer learning.

[0087] Specifically, it is implemented in terms of how to use representation blocks and mapping blocks to solve the problem of redundant memory of multiple segments (solving the internal redundancy problem of multiple embedded feature information). If there is a situation where the semantic entities and other information between different segments are highly similar, the representation block and the mapping block are used to combine and replace the representation block where the semantic entities and other information are highly similar. Through this method, the semantic entity and its motion information inside the representation block are supplemented, deleted, fine-tuned, rotated, inverted and mirrored by the mapping block to express the represented block. Its mapping and mutual reference range is all feature needle blocks in a single transmission. If the complete video is processed at a single time, the full mutual reference of the full video representation block is performed; if the video corresponding to a certain time segment is processed or sent at a single time, the mutual reference within the representation block of the corresponding video segment in the time segment is performed. In the extreme case, if only one frame of the image is transmitted at a time, the reference representation operation of the representation block and the mapping block is not performed and is directly entropy encoded into a binary bit stream. It can effectively eliminate the redundant information of multiple segments in the video, and do not repeat the memory, storage and transmission of redundant semantic entity information and its motion information.

[0088] When the module performs reverse input, if the header identification information is read to obtain that there are currently 2 feature representation blocks, but there are only multiple embedded feature information compression representation block 1 and mapping block 1, then the multiple embedded feature information compression representation block 1 and mapping block 1 are reversely input into the multiple embedded feature information compression representation block similarity comparison mapping model, and multiple embedded feature information compression representation block 1 and multiple embedded feature information compression representation block 2 will be obtained. When the module is executed normally, several mapping blocks with smaller volumes will be obtained. After the migration mapping is completed, the header identification information and the tail identification information are spliced ​​to the output content and entropy coding is performed to encode a pure binary bit stream (a digital stream consisting only of 0 and 1). The binary bit stream obtained by entropy coding can be adapted to cross-platform applications of most communication platforms. It should be additionally pointed out that step 106 can be executed in reverse. When step S106 is executed and the corresponding binary code stream is obtained, the encoder part is executed at this point.

[0089] 2. Decoding, including:

[0090] Step S107: After the decoder receives all binary code streams of a video and verifies that they are correct, it inputs the binary code stream into the code stream decoding module. Code stream decoding corresponds to Fig. 9 The reverse operation. The bitstream is first input into the entropy decoding module corresponding to the entropy coding and decoded to generate multiple embedded feature information with head and tail identifiers. At this time, some of the multiple embedded feature information compression representation blocks are missing in the multiple embedded feature information. At this time, the mapping block needs to be used to complete the mapping of the missing representation blocks. The specific examples are as follows:

[0091] Generally, when the multi-embedded feature information compression representation block n is missing, the multi-embedded feature information compression representation block n-1 and the mapping block n-1 exist. At this time, the multi-embedded feature information compression representation block n is replaced by the multi-embedded feature information compression representation block n-1 and the mapping block n-1. Generally, the multi-embedded feature information compression representation block n and the mapping block n appear in pairs. There are also additional situations, such as the current multi-embedded feature information is: multi-embedded feature information compression representation block 1, mapping block 1, mapping block 1, mapping block 1, mapping block 6, multi-embedded feature information compression representation block 6, mapping block 6, multi-embedded feature information compression representation block 8. At this time, the actual restoration results are: multi-embedded feature information compression representation block 1, multi-embedded feature information compression representation block 2 (consisting of multi-embedded feature information compression representation block 1 and the first mapping block 1), multi-embedded feature information compression representation block 3 (consisting of multi-embedded feature information compression representation block 1 and the second mapping block 1), multi-embedded feature information compression representation block 4 (consisting of multi-embedded feature information compression representation block 1 and the third mapping block 1), multi-embedded feature information compression representation block 5 (consisting of multi-embedded feature information compression representation block 6 and the first mapping block 6), multi-embedded feature information compression representation block 6, multi-embedded feature information compression representation block 7 (consisting of multi-embedded feature information compression representation block 6 and the second mapping block 6), and multi-embedded feature information compression representation block 8.

[0092] When the compressed representation block is completely mapped and restored, the decompression operation is performed on all the internal representation blocks to restore the information in the representation block to the state before compression. At the same time, the basic description information in the header identification information is used to verify the multi-embedded feature information. When the verification is correct, the header identification information and the tail identification information are discarded. Finally, this step obtains the complete multi-embedded feature information.

[0093] Step S108: After the code stream decoding module completes the operation, the complete multi-embedded feature information is input into the generative AI decoding module 80 for video reconstruction, which is the same as the operation of step S104 in the encoder and will not be repeated here. After this sub-step is completed, the video based on semantic reconstruction is restored. At this point, the decoder is finished.

[0094] At this point, the multi-path embedded video coding system based on generative AI has been fully implemented.

[0095] 1. This method uses a breakthrough video encoding and decoding system based on a generative AI model, which can significantly reduce the storage space and information flow size required for storage and transmission.

[0096] 2. After adopting this method, different degrees of media source information expression can be controlled, including but not limited to lossy compression, variable semantic information, variable semantic target trajectory information, variable content body, variable target posture information, variable background information, semantic information encryption, high-density abstract feature expression, etc.

[0097] 3. After adopting this method, it is possible to customize the encoding and decoding expression of media sources according to the personalized needs of users (decoding end) / services (encoding end). When the receiving end does not need too much redundant semantic and feature information, the necessary semantic and feature information can be discarded and transmitted before transmission, further reducing the size of the required transmission information stream.

[0098] 4. After using this method, different from traditional video encryption methods, this method can realize semantic-level feature selection encryption, and the encrypted information is more refined; this method helps to meet more detailed encryption requirements and provide an effective solution for media sources that need to be split and transmitted or low-bandwidth and ultra-low-bandwidth media encryption transmission.

[0099] 5. After using this method, the transmission integrity of the semantic feature expression information can be fed back and corrected according to the needs of the user (decoding end) / service (encoding end).

[0100] 6. After using this method, the information flow size of the transmitted content and the specifications and size of the media information generated after the information flow is decoded can be flexibly controlled according to the needs of the decoding end, so that media information with higher resolution, stronger readability, clearer semantics, and better satisfaction of the receiver's needs can be decoded compared to the original media information.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-path embedded video coding method based on generative artificial intelligence, characterized by: The method includes an encoding step and a decoding step, wherein the prompt words and the required parameters from the decoding end and the encoding end are present, and the encoding includes the following steps: Step S101, inputting the prompt words and requirement parameters of the decoding end and the prompt words and requirement parameters of the encoding end into the prompt word and requirement parameter encoding module (10), encoding the input prompt words and requirement parameters and outputting them into prompt word vectors and requirement parameter code streams, and transmitting them to the prompt word vector decoding module (20), while the output prompt word vectors and requirement parameter code streams are also input into the semantic information encoding and code stream compression module (60), and are packaged as the input of this part into a binary code stream that can be transmitted; Step S102, after obtaining the prompt word vector and the required parameter code stream, the prompt word vector decoding module (20) decodes them and outputs a feature vector and a feature map; Step S103, after the feature vector and feature map are output, the feature vector, feature map and representation guidance vector are input into the video extraction module (30), and multiple embedded feature information is obtained through the video extraction module (30); Step S104, inputting the obtained multi-embedded feature information into the first generative AI decoding module (40), reconstructing the picture through generative AI and obtaining the video to be iterated; Step S105, receiving the video to be iterated and evaluating the video quality thereof. If the evaluation obtains the corresponding index parameter and determines that the video does not need to be iterated, the iteration is completed. If the evaluation obtains the corresponding index parameter and determines that the video still needs to be iterated; Step S106, at this time, the semantic feature extraction and internal optimization iteration are completed, the current prompt word vector and the required parameter code stream and the multi-embedded feature information are input into the semantic information encoding and code stream compression module (60) to generate a binary code stream for transmission, and the encoder stops encoding.

2. The multi-path embedded video coding method based on generative artificial intelligence as claimed in claim 1, characterized in that: The step S101 specifically includes: Inputting the decoding end prompt words and requirement parameters and the service (encoding end) prompt words and requirement parameters into the embedding layer of the prompt words and requirement parameter encoding module (10) for mode conversion; Then input into the Encoder of the prompt word and requirement parameter encoding module (10), and output as prompt word vector and requirement parameter code stream, wherein the Encoder uses a basic learner based on the Transformer learner idea to perform feature extraction; In the step S102, the prompt word vector decoding module (20) has a basic characterizer and can output a feature vector and a feature map.

3. The multi-path embedded video coding method based on generative artificial intelligence as claimed in claim 1, characterized in that: The step S103 specifically includes The original video information is input into a video segmentation discriminator of a video extraction module (30), and the video segmentation discriminator reads the entire video or a very short segment of a real-time video and determines whether there is editing or drastic scene switching, thereby dividing the entire video into smaller video segments; The smaller video segments after segmentation are input into the feature extraction module. In the feature extraction module, the expression and expression strength of different feature extraction modules in the feature extraction module are partially selected according to the results of phase weight encoding. After feature extraction, each small video segment generates a corresponding multi-embedded feature information representation block. By stacking and splicing several multi-embedded feature information representation blocks, the corresponding multi-embedded feature information of the entire video is obtained.

4. The method for multi-path embedded video coding based on generative artificial intelligence as claimed in claim 3, characterized in that: In step S103, the entire video or a very short segment of the real-time video is read and it is determined whether there is editing or drastic scene switching, and the entire video is divided into smaller video segments, including: In the default state, the video segmentation discriminator uses T seconds as a marking point, calculates the structural consistency parameters of the current frame and the previous frame and judges that if the structural similarity between the two exceeds a certain fixed initial threshold, the similarity is considered acceptable. The current frame and the previous frame are continuous continuous pictures. If the current predetermined number of frames are all continuous pictures, they are all packaged into 1 group of video clips. If the current predetermined number of frames are not all continuous pictures, they are segmented and packaged into several groups of video clips according to the SSIM threshold, where the total number of each continuous video frame is less than or equal to the set slicing time x video frame rate. The entire video is divided into smaller video clips by using a segmentation discriminator, and the small video clips generated after segmentation are then input into the feature extraction module.

5. The multi-path embedded video coding method based on generative artificial intelligence as claimed in claim 1, characterized in that: In step S103, in the feature extraction module, based on the result of phase weight coding, partially selecting the expression and expression strength of different feature extraction modules in the feature extraction module specifically includes: First, the video clips are input into the video content understanding module, which extracts features of semantic entities and background and foreground information in the video clips and generates representation prompt words for the generative AI to complete video decoding and generate the video to be iterated; At the same time, after the phase weight coding completes the weighted expression, feature extraction modules with different degrees of expression are selected according to the needs of the decoding end. If it is necessary to complete the pixel-level complete information expression, each feature extraction module will be fully called. If only the most abstract semantic feature information video needs to be expressed, only the representation prompt words generated by the video content understanding module are used. For the remaining basic parameters, the default parameters are directly called in the first generative AI decoding module (40) without customization and pixel-level precise expression. If it is necessary to modify the video feature representation strength, video restoration degree and personalized expression degree according to the needs of the decoding end, the input prompt word information is expressed through phase weight coding and different feature extraction and feature descriptors are called. At the same time, different feature extraction and feature descriptors are shown with different expression strengths according to the different phase weight expression strengths.

6. The method for multi-path embedded video coding based on generative artificial intelligence as claimed in claim 5, characterized in that: In step S103, each feature extraction module and feature descriptor includes: Edge information descriptor, motion information descriptor, semantic extraction module, semantic control feature extraction module, semantic internal consistency feature extraction module, semantic super-resolution feature extraction module, personalized style feature extraction module, video basic information reading and basic control parameter fine-tuning module; The edge information feature descriptor is mainly used to extract all edge contour information and features in the picture, and select the contour feature expression strength of different semantic entities according to the phase weight coding result and generate an edge information feature map; The motion information descriptor includes three sub-parts: a motion vector direction feature descriptor, an optical flow motion estimation descriptor, and a semantic trajectory and relative position descriptor. The motion vector direction feature descriptor is mainly used to extract feature information of motion vectors and their directions of different semantic entities in a video clip and generate a motion vector direction feature map. With the support of phase weight coding, the expression strength of motion vector direction features of different semantic entities is selected according to the representation guidance vector, and the total amount of transmitted information is purposefully adjusted or reduced. The optical flow motion estimation descriptor is used to describe the optical flow motion information features of motion semantics in different foregrounds and backgrounds in a video clip, and the features are used to assist in expressing global motion information and motion blur control in the picture. The semantic extraction module includes foreground semantic entity extraction, background semantic entity extraction and language and text extraction, which is mainly used to extract the semantic extraction of foreground semantic entities, background semantic entities and language and text entities in video clips. The foreground and background semantic entities are extracted for subsequent video reconstruction and semantic-based super-resolution of the generative AI decoder, and are also used for front- and back-end semantic duplication and redundancy compression of multiple video clips. The language and text entity semantic extraction module extracts the text content and its meaning feature information, which is used to restore the corresponding text with higher clarity and modify it into other languages ​​in different language environments according to the requirements of the representation guidance vector; The semantic control feature extraction module includes semantic trajectory and relative position descriptors, which are mainly used to accurately describe and guide the motion trajectory of local and specific semantics. Through the expression of different intensities and semantic entities of phase weight coding by the decoding end prompt words or representation guidance vectors, the accurate motion trajectory description of the semantic entity and semantic entity set of the specific target can be achieved. At the same time, it is used to ensure the consistency and personalized correction of the motion trajectory of the semantic entity. The semantic internal consistency feature extraction module is mainly used to ensure the internal consistency of semantic entities between different frames generated by the generative AI decoder, and to keep the pixel level and structure consistent within the semantics; The semantic super-resolution feature extraction module is mainly used for image denoising and super-resolution based on semantic information. The module simultaneously extracts the pixel noise features and semantic entity noise features of the specified semantic entity area and generates the corresponding feature map. It also extracts low-resolution feature maps for the video key frames for subsequent super-resolution denoising and pixel-level video image restoration of the specified semantic entity in the generative AI decoder. The personalized style feature extraction module is mainly used to extract the style and style feature information of the overall video clip, to accurately restore the style, contrast, dynamic range, highlights and dark details of the video, and to characterize different style conversion features under the guidance of the prompt word vector for generating personalized style expression features and their corresponding picture details in the generative AI decoder; The video basic information reading and basic control parameter fine-tuning module is used to read the video frame number, frame rate, aspect ratio, color space basic information and some semantic entity fine-tuning control parameters. The control parameters and basic video information will be input into the generative AI decoder part to control the consistency of the generated video basic parameter information.

7. The method for multi-path embedded video coding based on generative artificial intelligence according to claim 1, characterized in that: The step S104 specifically includes: By decomposing the multi-embedded feature information into blocks, an embedded feature semantic information representation block is obtained. The representation prompt word is input into the representation prompt word embedding layer to complete the embedding operation of the prompt word to generate high-dimensional array information that can be read by the model. If the edge information feature map, motion information feature map, semantic feature map, semantic control feature map, semantic internal consistency feature map, semantic super-resolution feature map and personalized style feature vector exist, they are correspondingly input into the auxiliary representation and guidance embedding layer. In the auxiliary representation and guidance embedding layer, the above-representable feature maps and feature vectors are uniformly encoded and embedded, and output as a high-dimensional array in a unified format to ensure that the input specifications and data structure are unified in the auxiliary representation hierarchical expression module. The basic control parameters are input into the parameter loading model to load customized parameter values. If the auxiliary representation and guidance embedding layer does not receive any information, the information of the same data structure is directly constructed and filled with the number 0. If some values ​​are missing in the parameter loading model, they are loaded with default parameters. If the representation prompt word embedding layer does not receive any information, the first generative AI decoding module (40) stops working. After completing the data loading of the above three parts, the data generated by the representation prompt word embedding layer is input into the representation prompt word expression core module in the generative AI video reasoning model; the data generated by the auxiliary representation and guidance embedding layer is input into the auxiliary representation hierarchical expression module in the generative AI video reasoning model to express its corresponding features according to different personalized features and representation strengths. Finally, the generative AI model is used to generate the video to be iterated / reconstructed.

8. The method for multi-path embedded video coding based on generative artificial intelligence as claimed in claim 1, characterized in that: Step S106 specifically includes: Insert the head and tail of the iterated multi-embedded feature information into the head identification information and the tail identification information. The inserted identification information is mainly used to mark the head and tail of the information fragment. At the same time, the head contains basic description information of the multi-embedded semantic feature semantic information. After the information is recorded, the multi-embedded feature information representation block 1 to the semantic information representation block n are losslessly compressed. After the lossless compression is completed, the compressed multi-embedded feature information compressed representation block is input into the multi-embedded feature information compressed representation block similarity comparison mapping model. The model compares the similarity between the feature representation blocks and replaces the representation blocks with different representations and high similarity through transfer learning. When the module is executed normally, several mapping blocks with smaller volumes are obtained. After the migration mapping is completed, the header identification information and the tail identification information are spliced ​​to the output content and entropy coding is performed to encode a pure binary bit stream. When the corresponding binary code stream is obtained, the encoder part is executed.

9. The method for multi-path embedded video coding based on generative artificial intelligence according to claim 1, characterized in that: The decoding step specifically includes: Step S107, the decoder receives the transmitted binary code stream, inputs the binary code stream into the code stream decoding module (70), and decodes the multi-embedded feature information and the prompt word vector and the required parameter code stream, wherein the prompt word vector and the required parameter code stream can be defaulted; Step S108, input the obtained multi-embedded feature information, prompt word vector and required parameter bitstream into the second generative AI decoding module (80), mainly using the multi-embedded feature information, the prompt word vector and the required parameter bitstream as guidance and configuration parameters, to generate videos with different degrees of personalized or specialized semantics required by the corresponding receiver. If the prompt word vector and the required parameter bitstream are missing, the original video is directly generated and expressed.

10. A multi-path embedded video coding system based on generative artificial intelligence, characterized in that: Includes an encoder and a decoder, where the encoder includes: The requirement parameter encoding module (10) is used to receive the prompt word and the requirement parameter at the decoding end and the prompt word and the requirement parameter input at the encoding end, encode the input prompt word and the requirement parameter and output them into a prompt word vector and a requirement parameter code stream, and transmit them to the prompt word vector decoding module (20). At the same time, the output prompt word vector and the requirement parameter code stream are also input into the semantic information encoding and code stream compression module, and are packaged as the input of this part into a binary code stream that can be transmitted; A prompt word vector decoding module (20) is used to decode the prompt word vector and the required parameter code stream after obtaining them, and output a feature vector and a feature map; A video extraction module (30) is used to receive the feature vector and feature map output by the prompt word vector decoding module (20), and obtain multi-embedded feature information after processing; A first generative AI decoding module (40) is used to receive the multi-embedded feature information, reconstruct the picture through generative AI and obtain the video to be iterated; A video quality comprehensive evaluation module (50) is used to receive a video to be iterated and evaluate its video quality. If the evaluation obtains corresponding index parameters and determines that the video does not need to be iterated, the iteration is completed. If the evaluation obtains corresponding index parameters and determines that the video still needs to be iterated in this module, A semantic information encoding and code stream compression module (60) is used to receive the current prompt word vector and the required parameter code stream and multiple embedded feature information after the semantic feature extraction and internal optimization iteration are completed, and generate a binary code stream for transmission; The decoder comprises: A code stream decoding module (70) is used to receive the binary code stream transmitted by the encoder, and decode the multi-embedded feature information and the prompt word vector and the required parameter code stream, wherein the prompt word vector and the required parameter code stream can be defaulted; The second generative AI decoding module (80) is used to generate videos with different degrees of personalized or specialized semantics required by the corresponding receiver based on the multi-embedded feature information, the prompt word vector and the required parameter bitstream as guidance and configuration parameters. If the prompt word vector and the required parameter bitstream are missing, the original video is directly generated and expressed.

Images