Coding intelligent optimization method and system based on category theory modeling

Abstract

The invention discloses a coding intelligent optimization method and system based on category theory modeling, and belongs to the technical field of intelligent coding. The method comprises the steps of determining a modeling norm of a relation set between extracted data based on a coding task, and constructing a coding model according to the modeling norm; generating a content data set, and labeling the content data set based on the relationship set required by the coding task; determining a compression target according to the data scaling structure under the coding task; based on the compression target, training the coding model on the labeled content data set; and using the trained coding model to complete the coding task. According to the method, by introducing the combined model training and compression optimization process, the reasoning performance under compact representation is improved, so that the coding performance is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent coding technology, and specifically relates to a coding intelligent optimization method and system based on category theory modeling. Background Art

[0002] Data encoding and decoding aims to encode a piece of raw data into a compact matrix and discretized binary bit stream through technical means, and then use technical means to decode the obtained binary bit stream into the original data or some transformation of the original data. Data encoding and decoding is an important technology in data storage and transmission. Because it has the ability to extract compact essential semantic information from data samples or data sets, it is also an important part of artificial intelligence models and has received increasing attention from the academic community in recent years.

[0003] The correlation between coding and intelligent technology is reflected in two aspects. On the one hand, coding can realize data exchange and improve the computational efficiency of the model. In addition, entropy can not only remove redundancy, but also play the role of information bottleneck and retain intrinsic semantics, which also inspires the design of machine learning models. For example, a large number of studies on deep learning theories and methods have adopted explicit or implicit information bottlenecks, which fully demonstrates the important role of insights from coding theory in improving the learning ability of models. On the other hand, intelligent technologies that promote models to provide stronger capabilities have also promoted the progress of coding. The emergence of predictive coding and hybrid coding have verified the use of fine-grained models and more diverse modes to improve coding efficiency.

[0004] However, the deep joint optimization of coding technology and intelligent technology is still lacking in theory and practice. Previous work has considered coding technology and intelligent framework relatively independently. Currently, there is a lack of a macro perspective and a unified theoretical framework to further reveal the intrinsic relationship between the two and guide the development of technology. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a coding intelligent optimization method and system based on category theory modeling. Based on the category theory-guided coding framework for different downstream tasks, the inference performance under compact representation is improved by introducing a combined model training and compression optimization process to improve the coding performance.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0007] A coding intelligent optimization method based on category theory modeling, the method comprising:

[0008] Based on the encoding task F, determine the relationship set between the extracted data The modeling paradigm of , and according to the modeling paradigm, construct the encoding model; wherein, is the data domain, and X is the object in the data domain;

[0009] Generate a content dataset and annotate the content dataset based on the relationship set K required by the encoding task E;

[0010] Determine the compression target according to the data compression structure under the encoding task F;

[0011] Based on the compression target, training the encoding model on the annotated content dataset;

[0012] Use the trained encoding model to complete the encoding task F.

[0013] Furthermore, the step of determining a modeling paradigm for extracting a set of relationships between data based on the encoding task, and constructing an encoding model according to the modeling paradigm, includes:

[0014] Get the relationship set between the extracted data under the encoding task F

[0015] Based on the task of encoding the relationship between the task F and the task of extracting the data, the set of inter-task domain relationships is obtained based on the analogy axiom. in, is the task domain, and the analogy axiom is the set of relationships between data Encodes all the information of data X, so the relationship set between the data Can be used by any downstream tasks;

[0016] According to the relationship set between task domains Determine a relational modeling paradigm between data; wherein the relational modeling paradigm between data includes: relational modeling based on serialization or relational modeling based on hierarchy;

[0017] Determine the encoding model based on the relationship modeling paradigm between data.

[0018] Furthermore, the content data set is labeled based on the relationship set K required by the encoding task F, including:

[0019] Based on abstract axioms, the set of relationships between data Select the subset required by the encoding task F, and obtain the relationship C(X) after data compression according to the subset required by the encoding task F; wherein the abstract axiom is that the relationship C(X) after data compression is equivalent to the relationship set between the data Select the subset required for downstream tasks;

[0020] According to the relationship C(X) after data compression, the relationship set K(F) required by the encoding task F is determined. i (C(X)), Among them, Y is the encoding result of data X, D is the distance function, ∈ is the set threshold, F i (C(X)) represents the i-th task F i , perform reasoning and recognition based on the compressed relation set C(X);

[0021] The content dataset is labeled according to the relationship set K required by the encoding task F.

[0022] Furthermore, in the case where the downstream task is intelligent coding, the compression targets include: bitstream length, model length and performance.

[0023] Furthermore, the encoding model includes an encoder, an entropy model and an end-to-end decoder; and training the encoding model on the annotated content data set based on the compression target includes:

[0024] Randomly select an image from the content dataset The image Use the encoder to extract features and discretize them into code streams

[0025] Estimating bitrate using entropy model The mean and variance of the code stream length are obtained, and the code stream length loss L is obtained by minimizing the code stream length En , by minimizing the model length, the length constraint on the encoder is obtained in, represents the parameters of the jth encoder;

[0026] The code stream Input is an end-to-end decoder, output is a reconstructed image And based on the reconstructed image And the relationship set K obtains the performance loss L perf And the constraints on the decoder in, Denotes the jth decoder D j Parameters;

[0027] Based on the performance loss L perf , bitstream length loss L En , length constraints on the encoder Length constraints on decoders Get the total loss U;

[0028] According to the total loss U and the gradient descent algorithm, the parameters of the encoder, entropy model and decoder are adjusted.

[0029] Furthermore, the total loss J is the number of downstream tasks, N j For downstream tasks F j The amount of data contained.

[0030] Furthermore, when the downstream task is video coding for machine vision, the compression targets include: bitstream length and performance.

[0031] Furthermore, the encoding model includes an encoder, an entropy model and an end-to-end decoder; and training the encoding model on the annotated content data set based on the compression target includes:

[0032] Randomly select an image from the content dataset The image Use the encoder to extract features and discretize them into code streams

[0033] Estimating bitrate using entropy model The mean and variance of the code stream length are obtained, and the code stream length loss L is obtained by minimizing the code stream length En , by minimizing the model length, the length constraint on the encoder is obtained in, represents the parameters of the jth encoder;

[0034] The code stream Input is an end-to-end decoder, output is a reconstructed image And based on the reconstructed image And the relationship set K obtains the performance loss L perf ;

[0035] Based on the performance loss L perf and the bitstream length loss L En Get the total loss U;

[0036] According to the total loss U and the gradient descent algorithm, the parameters of the encoder, entropy model and decoder are adjusted.

[0037] Furthermore, the total loss J is the number of downstream tasks, N j For downstream tasks F j The amount of data included, J is the number of downstream tasks, N j For downstream tasks F j The amount of data contained.

[0038] A coding intelligent optimization system based on category theory modeling, the system comprising:

[0039] The encoding model generation module is used to determine the relationship set between the extracted data based on the encoding task F. The modeling paradigm of , and according to the modeling paradigm, construct the encoding model; wherein, is the data domain, and X is the object in the data domain;

[0040] A content data annotation module is used to generate a content data set and annotate the content data set based on the relationship set K required by the encoding task F;

[0041] A compression target determination module, used to determine a compression target according to the data scaling structure under the encoding task F;

[0042] A coding model training module, used for training the coding model on the annotated content data set based on the compression target;

[0043] The coding model inference module is used to complete the coding task F using the trained coding model.

[0044] Compared with the prior art, the present invention has the following positive effects: by using the axiom system based on category theory, an adaptive compression framework is generated for different downstream tasks, and then the model training and compression method optimization are combined based on this compression framework. Experiments have shown that compared with the prior art, the present invention provides a more general intelligent compression framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Exchange diagram for the ideal coding framework.

[0046] Figure 2 Exchange graphs for smart coding frameworks.

[0047] Figure 3 Exchange graphs for an intelligent coding framework for coding tasks.

[0048] Figure 4 Flowchart of the encoding intelligent optimization method. DETAILED DESCRIPTION

[0049] In order to make the above features and advantages of the present invention more obvious and easy to understand, the following embodiments are specifically cited and described in detail with reference to the accompanying drawings. It should be noted that the specific number of layers, number of functions and settings of layers given in the following embodiments are only a preferred implementation method, and those skilled in the art can select the number and set certain layers according to actual needs, which should be understood.

[0050] The present invention uses the axiom system of category theory to generate a coding system for downstream tasks. The axiom system includes analogy axioms, abstract axioms and generalization axioms. First, the present invention agrees on the notation: for any data X, the set of relationships between it and other data is expressed as in Defined as a data domain. is the task domain, and task F is the task domain Any downstream task in is the output space corresponding to task F. For example, if task F is a recognition task, the output space is the recognized category; if F is a segmentation task, the output space For the segmentation diagram.

[0051] 1) Analogy Axiom: A set of relationships between data It encodes all the information of data X and can be used by any task F;

[0052]

[0053] is the set of relations between task domains, and the analogy axiom determines the task F and the extracted relation set The relationship between tasks.

[0054] 2) Abstract axiom: The relation C(X) after data compression is equivalent to the set of all relations from the data Select the subset required by the downstream task F;

[0055]

[0056] 3) Generalization axiom: Data compression leads to the decoupling of the data manifold structure and intrinsic randomness, and intelligent coding is achieved by optimizing manifold structure learning;

[0057]

[0058] Among them, MDL is the minimum description length, which means that under task F, the compression of N data can be decomposed into the compression of their common structure and the compression of the intrinsic randomness of each data.

[0059] Figure 1 The ideal coding framework exchange diagram is the theoretical framework of the present invention based on the category theory axiom system. Figure 2 The intelligent coding framework exchange graph is used to approximate the ideal coding framework in practice. In particular, when the loss function drops to 0 and the various parameter selections reach the optimal, it can be considered that the intelligent coding framework has achieved the ideal coding framework.

[0060] Based on this axiom system, the present invention is oriented to a specific coding task, such as Figure 3 and Figure 4 As shown, the steps include steps 1 to 5.

[0061] Step 1: Based on the encoding task, determine the modeling paradigm for extracting the set of relationships between the data, and construct the encoding model based on the modeling paradigm.

[0062] The present invention obtains the encoding task F and extracts the relation set After the task, based on the above analogy axiom, we get the set of relationships between tasks: Then, according to the relationship between tasks Determine the relationship modeling paradigm between the data. The present invention proposes two relationship modeling paradigms: serialization-based relationship modeling and hierarchical-based relationship modeling. Based on the determined relationship modeling paradigm, the present invention can build a coding model, which may include an encoder C, an entropy model, and an end-to-end decoder D, wherein the encoder and the decoder may adopt a multi-layer fully connected network, a deep convolutional network (U-Net structure), a Transformer structure, or a multi-layer gated recurrent network. For example, for serialized data such as text and audio, serialization modeling is generally used, and features can be extracted based on Transformer or RNN models; for video images, hierarchical modeling is generally used, and CNN, GNN and other models can be used to extract features.

[0063] Step 2: Generate a content dataset and annotate the content dataset based on the relationship set K required by the encoding task F.

[0064] Taking image coding as an example, the present invention first collects a large number of images to form a content data set.

[0065] According to the abstract axiom, the relationship set K required for the coding task F is determined. For image coding, the relationship set K required for the coding task F determined by the present invention (i.e., the relationship set K between the input image X and the reconstructed image Y) is:

[0066]

[0067] That is, the distance between the reconstructed image Y and the input image X needs to be less than the distance threshold σ. Where D(·,·) is the distance function, and the mean square error distance or feature perception distance can be selected as needed. i (C(X)) represents the i-th task F i , reasoning and recognition are performed based on the compressed relationship set C(X).

[0068] Finally, the content dataset is labeled according to the relationship set K required by the encoding task F.

[0069] Step 3: Determine the compression target according to the data compression structure under the encoding task F.

[0070] Based on the data compression structure involved in the generalization axiom, this method determines the compression target. The compression target mainly includes three items: bitstream length, model length and performance, which can be selected according to the specific downstream task.

[0071] Step 4: Based on the compression target, train the encoding model on the labeled content dataset.

[0072] When training the encoding model, the present invention randomly selects images from the annotated content dataset The image Use encoder C to extract features and discretize them into code stream as follows:

[0073]

[0074] Q is a discretization operation. Input is an end-to-end decoder, output is a reconstructed image Among them, there are J downstream tasks, each of which contains N j data.

[0075] Taking intelligent coding as an example, the loss function of training is:

[0076]

[0077] Among them, the jth encoder C j The parameters are The jth downstream task decoder D j The parameters are The first term represents the performance loss, where is the label of the downstream task, L Perf is the selected task loss function, and the second term is the code stream length loss, L En is the entropy constraint loss function. The third and fourth terms are model length constraints, where the first term is the length constraint for the encoder and the second term is the length constraint for the decoder. Com is the regularization constraint loss.

[0078] Taking video encoding for machine vision as an example, the training loss function is:

[0079]

[0080] The first term represents the performance loss, where is the label of the downstream task, L Perf is the selected task loss function, and the second term is the code stream length loss, L En is the entropy constrained loss function.

[0081] In summary, from the perspective of coding, the present invention encodes the training image using an encoder and discretizes it into a bitstream. Further, the entropy model is used to estimate the mean and variance of the bitstream, thereby obtaining an estimated bitstream length. The MDL(C(X)) estimate is obtained by minimizing the bitstream length, and the MDL(C) estimate is obtained by minimizing the model length.

[0082] From the decoding perspective, the bitstream is input into the decoder, and the performance loss target is obtained through a predetermined relationship set K. Joint model training and compression optimization are performed through the gradient descent algorithm.

[0083] Step 5: Use the trained encoding model to complete the encoding task.

[0084] During inference, the encoder is used to extract the original image into a bitstream and then transmit it to the decoding end for decoding.

[0085] Table 1 shows the comparison between the present invention and the VTM coding framework under human vision and machine vision. It can be seen that the present invention has a good performance in LPIPS. Alex LPIPS SQ , average intersection-to-union ratio and accuracy have achieved better technical results.

[0086] Table 1

[0087]

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A coding intelligent optimization method based on category theory modeling, characterized in that: The method comprises: Based on the encoding task F, determine the relationship set between the extracted data The modeling paradigm of , and according to the modeling paradigm, construct the encoding model; wherein, is the data domain, and X is the object in the data domain; Generate a content dataset and annotate the content dataset based on the relationship set K required by the encoding task F; Determine the compression target according to the data compression structure under the encoding task F; Based on the compression target, training the encoding model on the annotated content dataset; Use the trained encoding model to complete the encoding task F.

2. The method according to claim 1, characterized in that The step of determining a modeling paradigm for extracting a set of relationships between data based on the encoding task, and constructing an encoding model according to the modeling paradigm, includes: Get the relationship set between the extracted data under the encoding task F Based on the task of encoding the relationship between the task F and the task of extracting the data, the set of inter-task domain relationships is obtained based on the analogy axiom. in, is the task domain, and the analogy axiom is the set of relationships between data Encodes all the information of data X, so the relationship set between the data Can be used by any downstream tasks; According to the relationship set between task domains Determine a relational modeling paradigm between data; wherein the relational modeling paradigm between data includes: relational modeling based on serialization or relational modeling based on hierarchy; Determine the encoding model based on the relationship modeling paradigm between data.

3. The method according to claim 1, characterized in that The labeling of the content data set based on the relationship set K required by the encoding task F includes: Based on abstract axioms, the set of relationships between data Select the subset required by the encoding task F, and obtain the relationship C(X) after data compression according to the subset required by the encoding task F; wherein the abstract axiom is that the relationship C(X) after data compression is equivalent to the relationship set between the data Select the subset required for downstream tasks; According to the relationship C(X) after data compression, the relationship set required for the encoding task F is determined Among them, Y is the encoding result of data X, D is the distance function, ∈ is the set threshold, F i (C(X)) represents the i-th task F i , perform reasoning and recognition based on the compressed relation set C(X); The content dataset is labeled according to the relationship set K required by the encoding task F.

4. The method according to claim 1, characterized in that: When the downstream task is intelligent coding, the compression targets include: bitstream length, model length and performance.

5. The method according to claim 4, characterized in that The encoding model includes an encoder, an entropy model and an end-to-end decoder; and training the encoding model on the labeled content data set based on the compression target includes: Randomly select an image from the content dataset The image Use the encoder to extract features and discretize them into code streams Estimating bitrate using entropy model The mean and variance of the code stream length are obtained, and the code stream length loss L is obtained by minimizing the code stream length En , by minimizing the model length, the length constraint on the encoder is obtained in, represents the parameters of the jth encoder; The code stream Input is an end-to-end decoder, output is a reconstructed image Based on the reconstructed image And the relationship set K obtains the performance loss L pref And the constraints on the decoder in, Denotes the jth decoder D j Parameters; Based on the performance loss L pref , bitstream length loss L En , length constraints on the encoder Length constraints on decoders Get the total loss U; According to the total loss U and the gradient descent algorithm, the parameters of the encoder, entropy model and decoder are adjusted.

6. The method according to claim 5, characterized in that The total loss J is the number of downstream tasks, N j For downstream tasks F j The amount of data contained.

7. The method according to claim 1, characterized in that When the downstream task is video coding for machine vision, the compression objectives include: bitstream length and performance.

8. The method according to claim 7, characterized in that The encoding model includes an encoder, an entropy model and an end-to-end decoder; and training the encoding model on the labeled content data set based on the compression target includes: Randomly select an image from the content dataset The image Use the encoder to extract features and discretize them into code streams Estimating bitrate using entropy model The mean and variance of the code stream length are obtained, and the code stream length loss L is obtained by minimizing the code stream length En , by minimizing the model length, the length constraint on the encoder is obtained in, represents the parameters of the jth encoder; The code stream Input is an end-to-end decoder, output is a reconstructed image Based on the reconstructed image And the relationship set K obtains the performance loss L perf ; Based on the performance loss L perf and the bitstream length loss L En Get the total loss U; According to the total loss U and the gradient descent algorithm, the parameters of the encoder, entropy model and decoder are adjusted.

9. The method according to claim 8, characterized in that The total loss J is the number of downstream tasks, N j For downstream tasks F j The amount of data included, J is the number of downstream tasks, N j For downstream tasks F j The amount of data contained.

10. A coding intelligent optimization system based on category theory modeling, characterized in that: The system comprises: The encoding model generation module is used to determine the relationship set between the extracted data based on the encoding task F. The modeling paradigm of , and according to the modeling paradigm, construct the encoding model; wherein, is the data domain, and X is the object in the data domain; A content data annotation module is used to generate a content data set and annotate the content data set based on the relationship set K required by the encoding task F; A compression target determination module, used to determine a compression target according to the data scaling structure under the encoding task F; A coding model training module, used for training the coding model on the annotated content data set based on the compression target; The coding model inference module is used to complete the coding task F using the trained coding model.

Images