Machine-oriented image encoding method and system based on prompt learning
By employing a prompt-based learning-based image coding method, generalized features are extracted and combined with task characteristics for compression encoding and decoding feature transformation. This solves the problem that existing technologies cannot support a variety of machine intelligence tasks, achieving efficient and flexible encoding compression, applicable to a wide range of machine vision tasks, and optimizing the allocation of computing resources.
Patent Information
- Application Number
- CN202410926536.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2044-07-11
AI Technical Summary
Existing image coding schemes such as CDVS and CDVA cannot support a wide range of machine intelligence tasks and cannot be adaptively adjusted according to task characteristics, resulting in limited practical application scenarios.
A cue-based learning approach is adopted. At the encoding end, generalization features are extracted and compressed with the current task's cue for information selection characteristics. At the decoding end, feature transformation is performed to output modulation task features suitable for the current task. Compressed cue is designed to adapt to the needs of different tasks.
It enables a wide range of applications supporting various machine vision tasks, improves coding efficiency and flexibility, optimizes computing resources in collaboration between edge devices and the cloud, and enhances intelligent analysis efficiency.
Smart Images
Figure CN119011855B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image compression coding, in particular to a machine-oriented image coding method and system based on prompt learning. BACKGROUND
[0002] In recent years, with the rapid development of deep learning, human beings further move towards the intelligent era, and more and more image and video data will be collected, compressed and transmitted to directly serve machine vision tasks such as public safety monitoring, automatic driving and intelligent production and broadcasting. Recently, coding schemes for machine intelligence tasks such as the international standard of compact descriptors for visual search (CDVS) and the international standard of compact descriptors for video analysis (CDVA) can support intelligent analysis tasks such as retrieval with smaller transmission bandwidth. Specifically, CDVS standardizes the feature extraction method and feature compression method, and CDVA strengthens the feature extraction scheme and video processing process based on CDVS with deep learning. However, the representation involved in CDVS and CDVA is too compact and limited, so it cannot support common visual tasks such as target detection, semantic segmentation, instance segmentation and pedestrian re-identification in other practical applications, and therefore the practical application scenarios are limited.
[0003] It is of important practical value to explore an image coding framework that can support a wide range of machine intelligence tasks and adaptively adjust according to the characteristics of each task, thereby realizing efficient coding compression. SUMMARY
[0004] The purpose of the present application is to provide a machine-oriented image coding method and system based on prompt learning, which can support a wide range of machine intelligence tasks and adaptively adjust according to the characteristics of each task, thereby realizing efficient coding compression, and has important practical value.
[0005] The purpose of the present application is realized by the following technical solutions:
[0006] A machine-oriented image coding method based on prompt learning, comprising:
[0007] At the encoding end, the generalization features extracted from the input image are compressed and coded with information selection characteristics in combination with the compression prompts corresponding to the current task, and the encoded code stream is outputted;
[0008] At the decoding end, the reconstructed features decoded from the encoded code stream are combined with the current task to perform feature transformation, and the modulation task features suitable for the current task are outputted.
[0009] A machine-oriented image coding system based on prompt learning comprises:
[0010] An encoding module is arranged at an encoding end and is used for performing information selection characteristic compression coding on generalized features extracted from an input image in combination with a compression prompt corresponding to a current task, and outputting an encoding code stream;
[0011] A decoding module is arranged at a decoding end and is used for performing feature transformation on reconstructed features decoded from the encoding code stream in combination with the current task, and outputting modulation task features suitable for the current task.
[0012] A processing device comprises one or more processors, and a memory for storing one or more programs;
[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.
[0014] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the foregoing method is implemented.
[0015] The technical solution provided by the above application can be seen as follows: (1) generalized features for machine intelligent tasks can be extracted, the features can support various machine vision tasks, even those that may appear in the future, have a wide range of applications, and have strong practical application value. (2) The designed compression prompt can be applied to different tasks, so it has strong flexibility and pertinence, and can obtain higher coding efficiency than most traditional image coding methods. (3) In practical application scenarios that require edge device-cloud cooperation, computing resources can be flexibly allocated, and the feature extraction module in the intelligent task analysis process can be transferred to the edge device to complete, thereby greatly improving the execution efficiency of cloud intelligent analysis. BRIEF DESCRIPTION OF DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0017] Figure 1 A flowchart of a machine-oriented image coding method based on prompt learning is provided for the embodiments of the present application.
[0018] Figure 2 A machine-oriented image coding framework based on prompt learning is provided for the embodiments of the present application.
[0019] Figure 3A schematic diagram of obtaining a feature extractor and a transformation layer through large-scale pre-training is provided for an embodiment of the present application.
[0020] Figure 4 A structure schematic diagram of an information selector module is provided for an embodiment of the present application.
[0021] Figure 5 A training schematic diagram of a feature encoder and a feature decoder is provided for an embodiment of the present application.
[0022] Figure 6 An example schematic diagram of a task prompt layer is provided for an embodiment of the present application.
[0023] Figure 7 A schematic diagram of a machine-oriented image coding system based on prompt learning is provided for an embodiment of the present application.
[0024] Figure 8 A schematic diagram of a processing device is provided for an embodiment of the present application. DETAILED DESCRIPTION
[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.
[0026] First, the terms possibly used in the present text are explained as follows:
[0027] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.
[0028] The term “consisting of” means excluding any technical feature element not explicitly listed. If this term is used in a claim, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements recorded in other clauses are not excluded from the overall claim.
[0029] A machine-oriented image encoding method and system based on prompt learning are described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conditions are performed according to the conventional conditions in the art or the conditions recommended by the manufacturer.
[0030] Embodiment one
[0031] The embodiments of the present application provide a machine-oriented image encoding method based on prompt learning, as shown in the following formula (1), which mainly includes: Figure 1
[0032] (1) At the encoding end, the generalization features extracted from the input image are combined with the compression prompt corresponding to the current task to perform compression encoding with information selection characteristics, and an encoded code stream is output.
[0033] The preferred embodiments of this part are as follows: the generalization features extracted from the input image are adaptively generated according to the current intelligent task (referred to as the current task); wherein the compression prompt and the generalization feature space have the same size, and the value of each position in the compression prompt represents the importance of the corresponding position feature to the current task; the generalization features are combined with the compression prompt to perform compression encoding with information selection characteristics.
[0034] In the embodiments of the present application, the encoding end includes a feature extractor, an information selector and a feature encoder; wherein: the feature extractor is used to extract generalization features from input images; the information selector is used to adaptively generate a compression prompt according to the current task; and the feature encoder is used to combine the compression prompt to perform compression encoding on the generalization features with information selection characteristics.
[0035] (2) At the decoding end, the reconstructed features decoded from the encoded code stream are combined with the current task to perform feature transformation, and a modulation task feature suitable for the current task is output.
[0036] In the embodiments of the present application, the reconstructed features decoded from the encoded code stream are then obtained by transformation to obtain task features, and the tasks are gradually migrated to modulation task features suitable for the current task.
[0037] In this embodiment of the invention, the decoding end includes: a feature decoder and a task analysis module; the feature decoder is used to decode reconstructed features from the encoded bitstream; the task analysis module includes multiple sets of transform layers and task prompt layers, the transform layer in the first set inputs the reconstructed features and outputs task features to the task prompt layer in the first set, the task prompt layer transfers the task features and inputs them to the transform layer in the second set, the transform layer in the second set continues to transform to obtain task features; finally, the task prompt layer in the last set outputs modulation task features suitable for the current task.
[0038] Furthermore, a task header can be set, inputting modulation task characteristics and outputting task results.
[0039] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.
[0040] I. Overall Overview of the Plan
[0041] like Figure 2 As shown, this is the overall framework of the method provided by the present invention. (1) Encoding end: For the input image x (size is H×W×3, where H and W are the height and width of the input image, respectively), a feature extractor with high generalization ability f (size is H×W×3) is extracted by a large-scale pre-trained feature extractor. D is the downsampling factor, and C is the number of feature channels; the feature f is input into the information selector, which adaptively generates compression prompts m based on the current intelligent task (the size of the compression prompt m is...). (The feature has the same spatial size as the feature, and its value represents the importance of the feature at the current position to the corresponding task). Then, the feature f and the compression cue m are input into the feature encoder. The encoder performs encoding compression on the feature f according to the compression cue m to adapt to the current task. (2) Decoding end: The bit stream is decoded at the decoding end to obtain the reconstructed features. Reconstruction features The input task analysis module consists of a transform layer pre-trained with the feature extractor and a newly introduced task cueing layer with fewer parameters. The output h of the task analysis module is then input into the task header to obtain the analysis result for the current task. It should be noted that, as... Figure 2 As shown, during downstream intelligent task migration, the learnable parameters only include those from the information selector, the task prompt layer, and the task header.
[0042] II. Detailed introduction of the plan.
[0043] 1. Feature extractor and transform layer.
[0044] In the embodiment of the present application, the feature extractor and the transformation layer are pre-trained together. As shown in Figure 3 The neural network model in conventional computer vision (convolutional neural network CNN, Transformer, etc.) can be used for large-scale pre-training, including but not limited to supervised learning, self-supervised learning, unsupervised learning based on large data sets (ImageNet-1k, ImageNet21K, etc.). The model pre-trained on a large scale has the ability to extract general features and strong migration ability. The model pre-trained on a large scale is divided into two parts, namely the feature extractor on the encoding side and the transformation layer for intelligent task migration on the decoding side.
[0045] 2. Information selector and compression prompt.
[0046] As shown in Figure 4 The input of the information selector is the general feature f, and the output is the compression prompt m. The information selector is composed of light depth separable convolution, batch normalization layer, activation layer, convolution layer and sigmoid layer connected in turn. Different downstream tasks use information selectors with the same structure but different parameters to achieve adaptive encoding compression of different tasks at the encoding end at a very small cost.
[0047] 3. Feature encoder and feature decoder
[0048] As shown in Figure 5 In the feature encoding and decoding training process, the input image x is input into the feature extractor of the large-scale pre-trained model, and the feature extractor outputs the feature f. The feature encoder and the feature decoder with spatial code rate control capability are trained for compressing and decompressing the feature, and the feature encoder can allocate the spatial dimension code rate of the feature f according to the compression prompt m. Figure 5 As shown in the process, the parameters of the feature extractor are fixed.
[0049] In the embodiment of the present application, a plurality of different machine intelligence tasks are completed based on one feature extraction model and one feature encoder and decoder, and the main difference lies in the compression prompt on the encoder side and the task prompt on the decoder side, thereby realizing low-cost intelligent task migration.
[0050] 4. Task prompt layer and task head.
[0051] The basic principle of the task prompt layer is lightweight and high efficiency, that is, under the condition of low complexity and low parameter amount, the task feature output by the transformation layer is as high performance as possible migrated to the modulation task feature capable of completing the current task
[0052] The implementation of the task prompt layer is diversified, and the present application does not limit the form of the task prompt layer, Figure 6An implementation example is provided, including a deep separable convolution layer and a residual connection layer; input task features After transformation by the deep separable convolution layer, the residual connection layer is connected with the input task features (i.e., the task features are corrected in the form of residual connection), and the output of the task prompt layer is obtained, if the task prompt layer is the last group (i.e., the Nth group), the output modulates the task features
[0053] The task head is determined by the downstream task, and the corresponding task result is output, for example, for an image classification task, the corresponding task head is a fully connected layer, and a classification result is output; for a semantic segmentation task, the corresponding task head is a deconvolution layer, and a segmentation result is output.
[0054] In the above scheme provided by the embodiment of the application, first, a pre-training process is performed, the model pre-trained on a large scale has the extraction ability of generalization features and strong migration ability, and can support various intelligent analysis tasks, and this part of training obtains a feature extractor and a transformation layer. Then, the feature encoder and the feature decoder are trained, so that they can adaptively allocate code rates for the feature compression process according to the compression prompt. At this point, the preparation work is completed. In the subsequent intelligent task migration stage based on prompt learning, the feature extractor, the feature encoder and the feature decoder, and the transformation layer are fixed, only the information selector with a small parameter amount, the task prompt and the task head are introduced, and different intelligent tasks are migrated at a very small cost.
[0055] Through the description of the above implementation, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment of the application.
[0056] Embodiment two
[0057] The application also provides a machine-oriented image encoding system based on prompt learning, which is mainly used to implement the method provided in the above embodiments, as shown in the figure, the system mainly includes: Figure 7
[0058] The encoding module is arranged at the encoding end, and is used to compress and encode the generalization features extracted from the input image in combination with the compression prompt corresponding to the current task, and output an encoding code stream;
[0059] The decoding module is arranged at a decoding end, and is configured to combine the reconstructed feature decoded from the encoded code stream with a current task to perform feature conversion, and output modulation task features suitable for the current task.
[0060] In view of the main technical details involved in the above system having been introduced in the previous embodiments, no further elaboration is needed.
[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.
[0062] Embodiment Three
[0063] The application further provides a processing device, as shown in the drawings, mainly comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the foregoing embodiments. Figure 8
[0064] Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device and the output device are connected through a bus.
[0065] In the embodiments of the application, the specific types of the memory, the input device and the output device are not limited; for example:
[0066] The input device can be a touch screen, an image acquisition device, a physical key or a mouse, etc.
[0067] The output device can be a display terminal.
[0068] The memory can be a random access memory (RAM), and can also be a non-volatile memory, for example, a disk memory.
[0069] Embodiment Four
[0070] The application further provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided by the foregoing embodiments is implemented.
[0071] The readable storage medium in the embodiment of the present application can be arranged in the processing device as a computer readable storage medium, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various storage program code media.
[0072] The above merely provides the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or replacements within the technical scope disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A machine-oriented image coding method based on cue learning, characterized in that, include: At the encoding end, the generalized features extracted from the input image are combined with the compression prompts corresponding to the current task to perform compression encoding with information selection characteristics, and the encoded bitstream is output. At the decoding end, the reconstructed features decoded from the encoded bitstream are combined with the current task to perform feature transformation, and output modulation task features suitable for the current task. The decoding end includes: a feature decoder and a task analysis module; The feature decoder is used to reconstruct features from the encoded bitstream; The task analysis module includes multiple transformation layers and task prompting layers. The transformation layer in the first group takes in the reconstructed features and outputs the task features to the task prompting layer in the first group. The task prompting layer transfers the task features and then inputs them to the transformation layer in the second group. The transformation layer in the second group continues to transform and obtain the task features. Finally, the task prompting layer in the last group outputs the modulation task features suitable for the current task.
2. The machine-oriented image coding method based on cue learning according to claim 1, characterized in that, The step of performing information-selective compression encoding by combining the generalized features extracted from the input image with the compression prompts corresponding to the current task includes: The generalized features extracted from the input image are used to adaptively generate compressed prompts based on the current task; wherein the compressed prompts have the same spatial size as the generalized features, and the values at each position in the compressed prompts represent the importance of the features at the corresponding positions to the current task; Combining compression prompts, the generalized features are compressed and encoded with information selection characteristics.
3. A machine-oriented image coding method based on cue learning according to claim 1 or 2, characterized in that, The encoding end includes: a feature extractor, an information selector, and a feature encoder; wherein: The feature extractor is used to extract generalized features from the input image; The information selector is used to adaptively generate compression prompts based on the current task; The feature encoder is used to perform compressed encoding with information selection characteristics on the generalized features in conjunction with compressed cues.
4. The machine-oriented image coding method based on cue learning according to claim 3, characterized in that, The information selector includes: a depth-separable convolutional layer, a batch normalization layer, an activation layer, a convolutional layer, and a sigmoid layer arranged sequentially.
5. The machine-oriented image coding method based on cue learning according to claim 1, characterized in that, The process of reconstructing the features decoded from the encoded bitstream, combining them with the current task to perform feature transformation, and outputting modulation task features suitable for the current task includes: The reconstructed features are decoded from the encoded bitstream. Then, the task features are obtained through transformation, and the task is gradually transferred to the modulation task features suitable for the current task.
6. The machine-oriented image coding method based on cue learning according to claim 1, characterized in that, The task prompting layer includes a depthwise separable convolutional layer and a residual connection layer. The input task features are transformed by the depthwise separable convolutional layer and then connected to the input task features through the residual connection layer to obtain the output of the task prompting layer.
7. A machine-oriented image coding system based on cue-based learning, characterized in that, To implement the method according to any one of claims 1 to 6, comprising: The encoding module, located at the encoding end, is used to extract generalized features from the input image, combine them with compression prompts corresponding to the current task, perform compression encoding with information selection characteristics, and output the encoded bitstream. The decoding module, located at the decoding end, is used to reconstruct the features decoded from the encoded bitstream, combine them with the current task to perform feature transformation, and output modulation task features suitable for the current task.
8. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 6.
9. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.