Coding and decoding method for content generated by artificial intelligence

Abstract

The invention provides a coding and decoding method for artificial intelligence generated content, and the coding method comprises the steps: S1, inputting semantics and priori; s2, generating content input; s3, semantic extraction and encoding: constructing a semantic encoding network to extract and encode semantic information of the input generated content; s4, fusing the semantic information and the coded content: fusing the extracted semantic information and the coded content by using a fusion coding model to form richer and more accurate coded representation; and S5, outputting a coding code stream. According to the method, the coding and decoding efficiency and quality of the AIGC generated content can be improved, meanwhile, the requirement for computing resources is reduced, and the AIGC technology is promoted to be applied to more fields.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a coding and decoding method for artificial intelligence generated content (AIGC). Background Art

[0002] AIGC (Artificial Intelligence Generated Content) technology has developed very rapidly in the areas of text-generated images and text-generated videos. At present, in the area of ​​text-generated images, image generation technology has developed from the early GAN (Generative Adversarial Network) to the more advanced diffusion model. The Stable Diffusion model generates images by gradually denoising, which can produce higher quality results. In the area of ​​video generation, video generation technology is developing from simple image splicing to more complex spatiotemporal modeling. The latest research includes Make-A-Video, Imagen Video, and Gen-2, which are gradually becoming dominant and leading the development of video generation and editing technology. AIGC technology is moving towards a multimodal direction, such as combining the generation of text, images, 3D, and video to achieve richer content creation.

[0003] In terms of coding and decoding technology, traditional source coding and channel coding technologies mainly focus on how to reduce redundancy and interference in the data transmission process to improve transmission efficiency and reliability. However, with the development of AIGC technology, more and more content is generated by AI. These contents themselves carry a lot of prior knowledge. Integrating semantic information into the coding and decoding technology processing can better and more efficiently encode and decode the AI-generated content. For example, by using semantics to pre-process the original data, extracting useful semantic information, and then transmitting it through source coding and channel coding, such a system design can make the coding and decoding process more intelligent, improve the efficiency of coding and decoding and the semantic accuracy of data transmission. References [1][2][3] The latest technology of Wensheng video introduced in the literature contains a wealth of prior knowledge. [4][5] The latest progress in coding and decoding technology is introduced. At present, the main problems existing in this type of background technology are: the current coding and decoding technology does not fully consider the prior knowledge and semantic information contained in the Vincent video, resulting in the encoding effect of the Vincent video not being improved as it should be. The difficulty of solving the above problems and defects is: this problem is due to new problems caused by new technologies, which have been rarely involved in previous work and require high innovation capabilities. It is difficult to solve. The significance of solving the above problems and defects is: there will be new innovations in the industry and technology, which is of great significance. Summary of the invention

[0004] The present invention provides a coding and decoding method for artificial intelligence generated content, which can improve the coding and decoding efficiency and quality of artificial intelligence generated content (AIGC), while reducing the demand for computing resources and promoting the application of AIGC technology in more fields.

[0005] The technical solution of the present invention is as follows: According to one aspect of the present invention, there is provided a method for encoding AI-generated content, comprising the following steps: S1. Semantic and prior input; S2. Input of AI-generated content; S3. Semantic extraction and encoding: constructing a semantic encoding network to extract and encode the semantic information of the input AI-generated content; S4. Fusion of semantic information with the AI-generated content to be encoded: using a fusion encoding model to fuse the extracted semantic information with the encoded content to form a richer and more accurate encoding representation; S5. Encoded bitstream output.

[0006] Optionally, in the above-mentioned encoding method for AI-generated content, in step S1, at the initial stage of the encoding process, a natural language description of a text image or text video is input, and combined with relevant semantic information and prior knowledge, and the prior knowledge includes information related to the type, scene and main composition of the content.

[0007] Optionally, in the above-mentioned encoding method for AI-generated content, in step S2, the AI-generated content is input into the encoding system, and the AI-generated content includes images, video frames or other forms of media data as direct input to the encoding algorithm.

[0008] Optionally, in the above-mentioned encoding method for AI-generated content, in step S3, a semantic encoding network is constructed using a deep learning model of a self-attention mechanism to extract and encode the semantic information of the input AI-generated content.

[0009] Optionally, in the above-mentioned encoding method for AI-generated content, semantic information is encoded through a semantic encoding network, high-level semantic information is compressed and encoded using a deep learning network with an attention mechanism, and a semantic feature stream is output through feature selection and dimensionality reduction.

[0010] Optionally, in the above-mentioned encoding method for artificial intelligence generated content, in step S4, a fusion coding model is used to achieve semantic information alignment, inter-frame prediction combined with semantics, and optimization of the transformation basis.

[0011] Optionally, in the above-mentioned encoding method for artificial intelligence-generated content, in step S4, semantic information and encoded content are aligned by using semantic segmentation technology based on deep learning; a spatiotemporal prediction model based on deep learning is adopted in combination with semantic inter-frame prediction; in the selection of transformation basis, a quantization matrix optimization method based on visual perception is adopted, and an entropy coding method based on machine learning is designed to dynamically adjust the coding strategy according to the statistical characteristics of the content; and a deep learning model is introduced to optimize the entropy coding process.

[0012] According to another aspect of the present invention, there is provided an encoding method for artificial intelligence generated content, comprising the following steps: T1. bitstream input: at the decoding end, the encoded data stream is input, including compressed video frames, semantic information and other auxiliary data transmitted from the encoding end; T2. a fusion decoding model is used to process the input bitstream, so that the model can understand the semantic encoding strategy of the encoding end and decode the video content accordingly; T3. the encoded semantic information is converted back to the original semantic representation through a semantic decoding network; T4. semantic information output: after the decoding process is completed, semantic information is output, which is consistent with the original semantic information before encoding; and T5. content output: the decoding system outputs the final Vincent image or Vincent video content.

[0013] According to the technical solution of the present invention, the beneficial effects produced are: The method of the present invention utilizes the high-level semantic information of the generated content to efficiently assist in coding and correlate, realizes an efficient coding and decoding algorithm for artificial intelligence generated content (AIGC), and designs a new coding and decoding algorithm in a targeted manner for the semantics and related prior knowledge in the generated content, thereby realizing a more efficient and accurate coding and decoding algorithm for the generated content. Compared with the traditional coding and decoding method, the method of the present invention enhances the understanding of the coding and decoding content. By combining the semantic information, the coding and decoding technology can better understand the scene content in the Vincent map and Vincent video, thereby performing more accurate intra-frame and inter-frame prediction. The method of the present invention better optimizes the transformation basis: using prior knowledge to select a transformation basis that is more suitable for a specific type of content can improve the efficiency of the transformation step, thereby improving the compression performance. The method of the present invention can improve the efficiency of entropy coding, design a more effective entropy coding model based on the statistical characteristics of the content, reduce the amount of data after coding, improve the compression ratio, and improve the compression efficiency.

[0014] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0016] Figure 1 is a flow chart of an encoding method for artificial intelligence generated content of the present invention; Figure 2 It is a flowchart of a decoding method for artificial intelligence generated content of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0018] The coding and decoding method for artificial intelligence generated content of the present invention, by constructing a coding method combining semantic information and traditional coding and decoding technology, makes full use of prior knowledge in Vincent graphs and Vincent videos, and optimizes and improves the coding efficiency of coding and decoding modules such as intra-frame and inter-frame prediction, transform basis and entropy coding in scene content and video. Specifically, the present invention uses the intrinsic semantic information and prior knowledge of artificial intelligence generated content (AIGC) to achieve efficient coding and decoding of artificial intelligence generated content (AIGC), solves the fusion of semantic information and coding in semantic perception coding, and enables the encoder to understand and emphasize the semantic optimization of frame prediction of the key part of the content: optimize the intra-frame and inter-frame prediction algorithms using prior knowledge, and improve the accuracy of prediction by understanding the temporal coherence and logical relationship of the scene content, thereby improving the compression efficiency. Design an entropy coding model based on the statistical characteristics of the content, and use a machine learning algorithm to dynamically adjust the coding strategy to reduce the amount of data after coding.

[0019] The principle of the present invention is to optimize the encoding and decoding process by using the semantic information and prior knowledge in artificial intelligence generated content (AIGC), encode the semantic information in the text image and text video through the semantic coding network, and perform intra-frame and inter-frame prediction in combination with the encoded semantic features, thereby improving the accuracy of the prediction. At the same time, the prior knowledge is used to select the transformation basis that best suits the characteristics of the video content to improve the efficiency of the transformation step. In addition, the present invention also designs an entropy coding model based on the statistical characteristics of the content to reduce the amount of data after encoding and further improve the compression ratio and compression efficiency.

[0020] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0021] like Figure 1 and Figure 2As shown in the figure, they are respectively a flow chart of the encoding and decoding method for artificial intelligence generated content of the present invention, and a method for encoding and decoding artificial intelligence generated content (AIGC) in an embodiment of the present invention, and the specific encoding steps are as follows: S1. Semantic and prior input: At the initial stage of the encoding process, the natural language description (prompt) of the text image or text video is input, and combined with relevant semantic information and prior knowledge. This prior knowledge includes information related to the type, scene, and main composition of the content, which is used to guide the subsequent encoding strategy.

[0022] S2. AIGC input: AIGC is input into the encoding system. These contents include images, video frames or other forms of media data, which will serve as direct input to the encoding algorithm.

[0023] S3. Semantic extraction and encoding: Construct a semantic encoding network to extract and encode the semantic information of the input AI-generated content. Use a deep learning model with a self-attention mechanism to construct a semantic encoding network to extract and encode the semantic information of the input AI-generated content. This step enhances the model's ability to focus on key content through the attention mechanism, thereby improving the quality and efficiency of encoding. Encode the semantic information through the semantic encoding network, compress and encode the high-level semantic information using the deep learning network with an attention mechanism, and output the semantic feature stream through feature selection and dimensionality reduction.

[0024] S4. Fusion of semantic information and coded content: The extracted semantic information is fused with the AIGC to be coded using a fusion coding model to achieve semantic information alignment, inter-frame prediction combined with semantics, optimization of transform bases and other technical processing to form a richer and more accurate coding representation. The fusion coding model mainly achieves semantic information alignment, inter-frame prediction combined with semantics, optimization of transform bases and other technical processing. Among them:

[0025] Semantic information alignment: By using deep learning-based semantic segmentation techniques, semantic information and encoded content can be aligned more accurately. These techniques enable pixel-level semantic understanding, resulting in finer semantic alignment during encoding and decoding.

[0026] Combined semantic inter-frame prediction: Using deep learning-based spatiotemporal prediction models, such as STRPM (Spatiotemporal Residual Predictive Model), can effectively predict inter-frame changes in high-resolution videos. This model can retain more spatiotemporal information and improve prediction accuracy.

[0027] Optimization of transform base selection: In the selection of transform base, a quantization matrix optimization method based on visual perception is adopted, such as the MTF (Modulation Transfer Function) model based on DCT, to adapt to the HEVC (High Efficiency Video Coding) standard. This method can further improve the compression performance without increasing the complexity of coding. Design and optimization of entropy coding model: Design an entropy coding method based on machine learning, such as adaptive entropy coding, to dynamically adjust the coding strategy according to the statistical characteristics of the content. In addition, a deep learning model can be introduced to optimize the entropy coding process and achieve more efficient compression by learning the complex distribution of data.

[0028] S5. Coded bitstream output: Output the encoded data stream, including compressed video frames, encoded semantic streams, and other auxiliary information. The output bitstream should be designed to be effectively transmitted and accurately restored at the decoding end.

[0029] The decoding step at the decoding end of the present invention includes: T1. Code stream information input: At the decoding end, the encoded data stream is first input, which includes compressed video frames, semantic information and other auxiliary data transmitted from the encoding end.

[0030] T2. Fusion decoding model: Use the fusion decoding model to process the input bitstream. This allows the model to understand the semantic encoding strategy of the encoder and decode the video content accordingly. The decoding process involves inverse transform operations and entropy decoding strategies in the optimal selection of transform bases.

[0031] T3. Semantic Decoding Network: The decoding network is responsible for converting the encoded semantic information back to the original semantic representation. It uses a similar deep learning model as the encoding end, but for decoding operations.

[0032] T4. Semantic information output: After the decoding process is completed, the semantic information is output. This information should be consistent with the original semantic information before encoding to ensure the integrity and accuracy of the information.

[0033] T5. Content output: Finally, the decoding system outputs the final Vincent image or Vincent video content. These contents should be able to faithfully reflect the original input and be consistent with the original content in terms of vision and semantics.

[0034] The method of the present invention constructs a codec algorithm model that combines semantics and prior knowledge, allowing the codec algorithm to fully utilize the correlation between semantic information and prior knowledge and images, image blocks and pixels, so that the codec algorithm can more accurately understand and process scene content, accurately estimate and analyze dynamic content transformation and scene conversion, and achieve better intra-frame and inter-frame prediction. According to prior knowledge, the transformation base that best suits the characteristics of the video content is selected, thereby improving the efficiency of the transformation step. According to prior knowledge, the transformation base that best suits the characteristics of the relevant content is selected, thereby improving the efficiency of the transformation step. A more efficient compression rate is achieved.

[0035] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

[0036] References: [1] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S.,... & Taigman, Y. (2022). Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792. [2] Wang, [3] Sun, M., Wang, W., Qin, Z., Sun, J., Chen, S., & Liu, J. (2024).GLOBER: coherent non-autoregressive video generation via global guided videodecoder. Advances in Neural Information Processing Systems, 36. [4] Huang, X., Tang, R., Pan, R., Yin, H., Wang, Z., Wang, S., & Ma,S. (2024). Parallelized RDOQ Algorithm and Fully Pipelined HardwareArchitecture for AVS3 Video Coding. IEEE Transactions on Circuits and Systemsfor Video Technology. [5] Jassem, R., Damak, T., Ben Ayed, M. A., & Masmoudi, N. (2024).Inter prediction multiple reference frames impact on H266-VVC encoder.Multimedia Tools and Applications, 83(17), 50329-50348.

Claims

1. A method for encoding content generated by artificial intelligence, characterized in that: The following steps are involved: S1. Semantic and prior input; S2. Input of AI-generated content; S3. Semantic Extraction and Encoding: Construct a semantic encoding network to extract and encode the semantic information of the input AI-generated content; S4. Fusion of semantic information and coded content: The extracted semantic information is fused with the AI-generated content to be coded using a fusion coding model to form a richer and more accurate coded representation; S5. Encoded code stream output.

2. The encoding method for artificial intelligence generated content according to claim 1, characterized in that: In step S1, at the initial stage of the encoding process, a natural language description of a text image or text video is input and combined with relevant semantic information and prior knowledge, and the prior knowledge includes information related to the type, scene and main composition of the content.

3. The encoding method for artificial intelligence generated content according to claim 1, characterized in that: In step S2, the AI-generated content is input into a coding system, wherein the AI-generated content includes images, video frames or other forms of media data as direct input to a coding algorithm.

4. The method for encoding artificial intelligence generated content according to claim 1, characterized in that: In step S3, the semantic encoding network is constructed using a deep learning model with a self-attention mechanism to extract and encode the semantic information of the input artificial intelligence-generated content.

5. The encoding method for artificial intelligence generated content according to claim 4, characterized in that: The semantic information is encoded by the semantic encoding network, and the high-level semantic information is compressed and encoded by the deep learning network of the attention mechanism. The semantic feature stream is output by selecting and reducing the dimensions of the features.

6. The method for encoding artificial intelligence generated content according to claim 1, characterized in that: In step S4, the fusion coding model is used to achieve semantic information alignment, semantically combined inter-frame prediction, and optimization of the transform basis.

7. The method for encoding artificial intelligence generated content according to claim 6, characterized in that: In step S4, semantic information and coded content are aligned by using a semantic segmentation technique based on deep learning; the inter-frame prediction combined with semantics adopts a spatiotemporal prediction model based on deep learning; in the selection of the transform basis, a quantization matrix optimization method based on visual perception is adopted, and an entropy coding method based on machine learning is designed to dynamically adjust the coding strategy according to the statistical characteristics of the content; And introduce deep learning models to optimize the entropy coding process.

8. A method for encoding artificial intelligence generated content, characterized in that: The following steps are involved: T1. Stream input: At the decoding end, the encoded data stream is input, including compressed video frames, semantic information and other auxiliary data transmitted from the encoding end; T2. Use the fusion decoding model to process the input bitstream so that the model can understand the semantic encoding strategy of the encoder and decode the video content accordingly; T3. Convert the encoded semantic information back to the original semantic representation through the semantic decoding network; T4. Semantic information output: After the decoding process is completed, the semantic information is output, which is consistent with the original semantic information before encoding; and T5. Content output: The decoding system outputs the final Vincent image or Vincent video content.

Images