Variable Coding Rate Semantic Structured Image Coding and Decoding Method and System

Abstract

The present invention discloses a variable bit rate semantic structured image encoding and decoding method and system. By using a rate control mask with the same spatial scale as the input image, variable bit rate coding capability can be introduced for semantic structured image coding, making it more flexible, thereby having higher coding efficiency and higher flexibility when adapting to intelligent analysis tasks in practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression coding, and in particular to a variable bit rate semantic structured image coding and decoding method and system. Background Art

[0002] The demand for coding technologies for intelligent task analysis is growing, and international and domestic standards organizations have already begun standardization work. To more efficiently support human-machine hybrid intelligent applications, existing methods have proposed the concept of semantically structured image code streams. For example, Solution 1: Chinese invention patent authorization announcement number CN110225341B, "A Task-Driven Code Stream Structured Image Coding Method," introduces a regional decision network and alignment module for target detection. Based on compressed features, bounding boxes are extracted for the object's area, and the features are spatially segmented based on the bounding boxes. The segmented features are then entropy-coded sequentially to form a structured code stream.

[0003] However, Solution 1 utilizes a learning-based codec to compress the texture portion of different image objects. This codec cannot support image compression at different bit rates, let alone image compression with different bit rate allocations for different objects, limiting the coding efficiency for downstream intelligent analysis tasks. Solution 2 (Song M, Choi J, Han B. Variable-rate deep image compression through spatially-adaptive feature transform [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 2380-2389.) can achieve image coding compression with bit allocation in the spatial dimension.

[0004] However, when combining Scheme 1 with variable-rate semantically structured image coding, if only a portion of the image is decoded, the error statistics of the non-reconstructed portion during the upsampling and modulation process can affect the reconstructed portion, ultimately leading to erroneous reconstruction, primarily manifesting as noticeable color shift. Therefore, Scheme 2 cannot be directly applied to the semantically structured image coding method in Scheme 1. Summary of the Invention

[0005] The purpose of the present invention is to provide a variable bit rate semantically structured image encoding and decoding method and system, which can introduce variable bit rate encoding capabilities for semantically structured image coding, making it more flexible, thereby having higher coding efficiency and higher flexibility when adapting to intelligent analysis tasks in practical applications.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A variable bit rate semantic structured image encoding and decoding method, comprising:

[0008] The encoding part: performs target detection on the input image to obtain the target detection result; combines the input image with the generated rate control mask of the same spatial scale as the input image to transform and obtain de-redundant image features; inputs the de-redundant image features into two branches, in the first branch, performs super-prior transformation and quantization on the de-redundant image features to obtain quantized super-prior features, performs super-prior inverse transformation on the quantized super-prior features to obtain the overall probability distribution, and entropy codes the size of the input image and the quantized super-prior features and the target detection results respectively, and then sequentially splices them to obtain the bitstream header information; in the second branch, quantizes the de-redundant image features to obtain quantized image features, groups them in combination with the target detection results to obtain the quantized features corresponding to each target, performs entropy coding on the quantized features corresponding to each target in combination with the overall probability distribution, selects all designated targets according to the task settings, and combines the entropy coded bitstreams corresponding to all designated targets to form the texture part of the bitstream; the bitstream header information and the texture part of the bitstream constitute a semantically structured bitstream;

[0009] Decoding part: decode the bitstream header information in the semantically structured bitstream to obtain the size, quantized super-prior features and target detection results of the input image, perform super-prior inverse transformation on the quantized super-prior features to obtain the overall probability distribution and rate control features; take out the bitstream corresponding to each specified target from the texture part, combine the overall probability distribution, perform entropy decoding to obtain the group features corresponding to each specified group, and reorganize the quantized features corresponding to all specified targets into recombined quantized image features according to the position provided by the target detection results; combine the size information of the input image, the recombined quantized image features, the rate control features, and the foreground mask calculated using the target detection results to obtain the reconstructed image through the inverse transformation operation.

[0010] A variable bit rate semantic structured image encoding and decoding system, comprising:

[0011] The encoding unit is configured to execute an encoding portion, the encoding portion comprising: performing target detection on an input image to obtain a target detection result; transforming the input image with a generated rate control mask of the same spatial scale as the input image to obtain de-redundant image features; inputting the de-redundant image features into two branches, wherein in a first branch, the de-redundant image features are subjected to a super-prior transformation and quantization to obtain quantized super-prior features, the quantized super-prior features are subjected to a super-prior inverse transformation to obtain an overall probability distribution, and the size of the input image, the quantized super-prior features, and the target detection results are respectively entropy coded, and then sequentially concatenated to obtain bitstream header information; in a second branch, the de-redundant image features are quantized to obtain quantized image features, grouped in combination with the target detection results to obtain quantized features corresponding to each target, entropy coded in combination with the overall probability distribution for the quantized features corresponding to each target, selecting all designated targets according to a task setting, and combining the entropy coded bitstreams corresponding to all designated targets to form a texture portion of the bitstream; the bitstream header information and the texture portion of the bitstream constitute a semantically structured bitstream;

[0012] A decoding unit is used to execute the decoding part, which includes: decoding the code stream header information in the semantically structured code stream to obtain the size, quantization super-prior features and target detection results of the input image, performing super-prior inverse transformation on the quantization super-prior features to obtain the overall probability distribution and bit rate control features; extracting the code stream corresponding to each specified target from the texture part, combining the overall probability distribution, respectively performing entropy decoding to obtain the group features corresponding to each specified group, and recombining the quantization features corresponding to all specified targets into recombined quantization image features according to the positions provided by the target detection results; combining the size information of the input image, the recombined quantization image features, the bit rate control features, and the foreground mask calculated using the target detection results to obtain a reconstructed image through an inverse transformation operation.

[0013] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0015] A readable storage medium stores a computer program, characterized in that the aforementioned method is implemented when the computer program is executed by a processor.

[0016] It can be seen from the technical solution provided by the present invention that a rate control mask with the same spatial scale as the input image is used to introduce the capability of spatially controllable rate allocation coding compression into the semantically structured image coding technology, so that the semantically structured image coding has greater flexibility and has higher coding efficiency when adapted to downstream intelligent analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A framework diagram of a variable bit rate semantically structured image encoding and decoding method provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of the transformation process and spatial feature transformation in the transformation process provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of the inverse transformation process and the spatial feature transformation process in the inverse transformation process provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a variable bit rate semantically structured image encoding and decoding system provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] First, the following terms may be used in this article:

[0025] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y includes both “X” or “Y” and “X and Y”.

[0026] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0027] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0028] The following describes in detail a variable bitrate semantically structured image encoding and decoding method and system provided by the present invention. Any information not described in detail in the embodiments of the present invention is well known to those skilled in the art. For any unspecified conditions in the embodiments of the present invention, the process shall be carried out in accordance with conventional conditions in the art or the conditions recommended by the manufacturer.

[0029] Example 1

[0030] An embodiment of the present invention provides a variable bit rate semantically structured image encoding and decoding method, which introduces the variable bit rate encoding capability into semantically structured image encoding, thereby making it more flexible and having higher coding efficiency and higher flexibility when adapting to intelligent analysis tasks in practical applications. Figure 1 The overall framework of this method is shown. Generally speaking, it includes two parts: encoding and decoding. Figure 1 The preferred implementation methods of the encoding and decoding parts are introduced in detail.

[0031] 1. Coding part.

[0032] 1. Perform target detection on the input image to obtain the target detection result.

[0033] In the embodiment of the present invention, for the input image x, the size is H×W×C, where H and W represent the height and width of the image respectively, and C is the number of channels. All channels are encoded and decoded simultaneously, and the target detection results are first obtained by the target detection model. The target detection results include: the number of targets, the position and category of each target; wherein the position and category of the kth target are expressed as: {a k ,b k ,hk ,w k ,cls k}, a k ,b k ,h k ,w k ,cls k They represent the horizontal position of the upper left corner coordinate, the vertical position of the upper left corner coordinate, the height, the width and the category of the k-th target respectively.

[0034] In the embodiment of the present invention, the target detection model can be implemented by selecting an existing model, so it is not described in detail.

[0035] In the embodiment of the present invention, the category information has no impact on the encoding and decoding process and is mainly used to support subsequent downstream tasks.

[0036] 2. Combining the input image with the generated rate control mask of the same spatial scale as the input image, and performing a transformation to obtain de-redundant image features.

[0037] In an embodiment of the present invention, the rate control mask is generated using a rate control factor r. The size of the rate control mask is H×W, where H is the height of the input image and W is the width of the input image. The value of each position in the rate control mask is a real value between 0 and 1, and the size of the value represents the relative size of the rate allocated to the pixel at the corresponding position in the input image during the encoding process.

[0038] In the embodiment of the present invention, the number of transformations is set to L, and the transformation process is the same each time. Figure 2 As shown in the left part, the lth (l=1,…,L) transformation process is as follows: 1) The input data and the rate control mask are cascaded in the channel dimension and then nonlinearly transformed and downsampled to generate the modulation feature t l ; 2) Perform nonlinear transformation on the input data to obtain feature y l , combined with the modulation characteristics t l Perform spatial modulation to obtain the characteristics of the lth transformation; Figure 2 The spatial modulation shown in the right part is expressed as: γ l ⊙y l +β l ; Among them, γ l and β l is the adjustment factor, which is obtained by adjusting the modulation characteristic t l Perform nonlinear transformation to obtain: l = 1, ..., L. When the first transformation is performed (l = 1), the input data is the input image x ( Figure 2In the example provided, l = 1. When the transformation is not the first one (l = 2, ..., L), the input data is the characteristics of the last (l-1) transformation. For example, L = 4 can be set. Of course, in actual applications, the specific value of L can be set according to actual conditions.

[0039] Through L transformations, the features of the Lth transformation are obtained, that is, the image features y after the final redundancy removal (size is C y is the number of feature channels, and B is the downsampling multiple after the transformation is completed).

[0040] 3. The de-redundant image features are input into two branches. In the first branch, the de-redundant image features are subjected to super-prior transformation and quantization to obtain quantized super-prior features, and the quantized super-prior features are subjected to super-prior inverse transformation to obtain the overall probability distribution. The size of the input image and the quantized super-prior features and the target detection results are entropy encoded respectively, and then sequentially spliced to obtain the code stream header information; in the second branch, the de-redundant image features are quantized to obtain quantized image features, and then grouped in combination with the target detection results to obtain the quantized features corresponding to each target, and the probability distribution corresponding to each target is obtained according to the overall probability distribution, and the quantized features corresponding to each target are entropy encoded respectively in combination with the corresponding probability distribution. All designated targets are selected according to the task settings, and the entropy coding streams corresponding to all designated targets are combined to form the texture part of the code stream; the code stream header information and the texture part of the code stream are combined to form a semantically structured code stream. Specifically:

[0041] 1) The first branch processing flow.

[0042] A1) The redundant image feature y is transformed into a super-prior feature z through super-prior transformation, and then quantized to obtain a quantized super-prior feature The quantized super-prior features are subjected to super-prior inverse transformation to obtain the overall probability distribution and provided to the second branch.

[0043] A2) Header information encoding.

[0044] The input image size information, the quantized hyper-prior features, and the object detection results are entropy-encoded separately, and then concatenated to obtain the bitstream header information. Here, entropy encoding the object detection results separately refers to entropy encoding the number of objects, as well as the location and category of each object. The syntactic structure of the bitstream header information is defined in Table 1.

[0045] Table 1: Syntax structure of codestream header information

[0046]

[0047] Among them: image_height_minus1 represents the height H of the image; image_width_minus1 represents the width W of the image; side_information_length represents the quantized super prior feature The corresponding bitstream length; bounding_boxes_numbers indicates the number of objects in the image; bounding_box_x, bounding_box_y, bounding_box_h, bounding_box_w, and bounding_box_category represent the horizontal coordinate of the upper left corner, the horizontal coordinate of the upper left corner, the vertical coordinate, the height, the width, and the category information of the current object, respectively. This example only provides a single object. The above five pieces of information for multiple objects are grouped together and arranged in sequence. u represents an unsigned data type. For example, u(32) indicates that the corresponding bitstream segment is 32 bits long.

[0048] In the embodiment of the present invention, the target detection result is a high-level semantic feature of the image, which is generally directly compressed and placed in the header information, and includes relevant information of all targets.

[0049] In the embodiment of the present invention, the code stream length is used for subsequent decoding. The principle is that during actual entropy coding, the length of the encoded data cannot be known, and during decoding, a code stream of the corresponding length must be read before decoding. Therefore, before storing the entropy coded code stream, the corresponding code stream length must be added so that the decoding end can correctly decode the data.

[0050] 2) The second branch processing flow.

[0051] B1) quantizing the de-redundant image feature y to obtain a quantized image feature

[0052] B2) Feature grouping. According to the above target detection results, the quantized image features are grouped Grouping is performed on the spatial dimension to obtain the quantitative features corresponding to each target.

[0053] B3) Texture information encoding.

[0054] Texture information encoding requires the use of the overall probability distribution Overall probability distribution Refers to quantified image features The overall probability distribution of , based on which the probability distribution of each target can be obtained, therefore, the quantized features corresponding to each target can be entropy coded separately in combination with the overall probability distribution. Specifically: for the kth target, first, according to the grouping situation and the overall probability distribution Combined with the autoregressive model, the corresponding probability distribution is obtained Then, the entropy model is combined with the corresponding quantitative features Perform entropy coding. Perform the above operation on the quantized features of all targets to obtain the entropy coded bitstreams of all targets.

[0055] In an embodiment of the present invention, the required targets can be determined based on the needs of the downstream tasks. The number of downstream tasks can be one or more, and the required targets for different downstream tasks can be the same or different. The required number of targets can be less than or equal to the total number of targets. Here, all specified targets are selected based on the settings of the downstream tasks, and the entropy coded streams corresponding to all specified targets are combined to form the texture portion of the bitstream. During the combination, the targets are arranged in ascending order according to the numerical values of the sequence numbers k corresponding to the targets, forming the texture portion of the bitstream. The semantic structure of the texture portion is defined as shown in Table 2.

[0056] Table 2: Syntax structure of texture part

[0057]

[0058] Wherein, object_texture_length_minus1 represents the length of the bitstream corresponding to the texture portion of the currently specified target. It should be noted that the above syntax structure only provides examples related to a single specified target.

[0059] The bitstream header information and the texture part of the bitstream constitute a semantically structured bitstream, and the grammatical structure includes: the height of the input image, the width of the input image, the bitstream length corresponding to the quantized super-prior feature, the number of targets, the position and category of each target, and the bitstream length corresponding to the texture part formed by the combination of all specified targets.

[0060] 2. Decoding part.

[0061] 1. Decode the code stream header information in the semantic structured code stream to obtain the size information of the input image, the quantized super-prior features and the target detection results. Specifically, this part of the decoding can obtain the height H and width W of the image, the quantized super-prior features As well as the location coordinates and category information of different specified targets.

[0062] 2. Super prior inverse transformation. Perform super prior inverse transformation to obtain the overall probability distribution and rate control features Used for subsequent decoding of quantitative features.

[0063] 3. Quantized feature decoding. Extract the code stream corresponding to each specified target from the texture part and combine it with the overall probability distribution Perform entropy decoding to obtain the group features corresponding to each specified group. Specifically: Combine the target detection results obtained by decoding and the overall probability distribution The probability distribution corresponding to each specified target is obtained with the autoregressive model, and entropy decoding is performed to obtain the quantitative features corresponding to each specified target.

[0064] Considering the overall probability distribution obtained by the decoding part Same as the encoding part above, and in the encoding and decoding part, the overall probability distribution is used The method of obtaining the probability distribution corresponding to each target is also the same, so, Figure 1 The relevant processes are summarized in the figure, that is, the part from quantization → inverse super-prior transform → probability estimation is omitted. Of course, this is also the common representation method of the relevant processes in this field.

[0065] 4. Feature Recombination. All the quantitative features corresponding to the specified targets are recombined into recombined quantitative image features according to their positions (included in the target detection results).

[0066] In the embodiment of the present invention, lossless entropy coding and lossless entropy decoding are used. Therefore, for the jth designated target, the quantized feature obtained by entropy decoding is But considering that the specified target can be part of all targets, i.e., recombining quantitative image features The number of quantized features and quantized image features in Different, therefore, the symbols and names are distinguished. Of course, if the specified target includes all targets, then the recombinant quantized image features Equivalent to quantified image features

[0067] 5. Combining the size information of the input image, the reconstructed quantized image features, the bit rate control features, and the foreground mask calculated using the target detection results, the reconstructed image is obtained through an inverse transform operation.

[0068] In the embodiment of the present invention, the foreground mask is a binary image of the same size as the input image. For each pixel on the mask, if the corresponding pixel point on the image belongs to any target, the value is 1, otherwise it is 0.

[0069] In the embodiment of the present invention, two types of inverse transformation are set. The number of inverse transformations of the first type is L′, and the inverse transformation process is the same each time. Figure 3 As shown in the left part, the l′th (l′=1, 2, …, L′) inverse transformation process is as follows:

[0070] 1) Adaptively upsample the foreground mask m to generate a foreground mask m that is consistent with the input feature space scale l′ When the first inverse transformation is performed (l′=1), the input feature is the reconstructed quantized image feature ( Figure 3 (For example, l′=1 is provided), when it is not the first inverse transformation (l′=2,…,L′), the input features are the features of the last (l′-1th) inverse transformation.

[0071] 2) Concatenate the input features and the rate control features in the channel dimension, and then generate the modulation features after nonlinear transformation and upsampling.

[0072] 3) Perform nonlinear transformation on the input features to obtain features Combined with modulation characteristics and the foreground mask m l′ Perform spatial modulation to obtain the result of the l′th inverse transformation. Figure 3 As shown in the right part, spatial modulation is expressed as: and is the adjustment factor, by adjusting the modulation characteristics In the above spatial modulation process, and foreground mask m l′ The purpose of point multiplication is to avoid the modulation process affecting the current image features. The features of the target part to be decoded are not included in the error modulation, which destroys the statistical information of the features. Multiple error modulations will result in the final reconstructed image. Produces serious color difference.

[0073] For example, L′ may be set to 3. Of course, in practical applications, the specific value of L′ may be set according to actual conditions.

[0074] The second type of inverse transformation has a number of 1. After L' inverse transformations, the output features can be reconstructed by another inverse transformation. Specifically: the features output after the L′th inverse transformation are transformed to upsample the features, and then the number of channels is reduced to the same as the number of channels of the input image to obtain the reconstructed image

[0075] Example 2

[0076] The present invention also provides a variable bit rate semantic structured image encoding and decoding system, which is mainly implemented based on the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes:

[0077] The encoding unit is configured to execute an encoding portion, the encoding portion comprising: performing target detection on an input image to obtain a target detection result; transforming the input image with a generated rate control mask of the same spatial scale as the input image to obtain de-redundant image features; inputting the de-redundant image features into two branches, wherein in a first branch, the de-redundant image features are subjected to a super-prior transformation and quantization to obtain quantized super-prior features, the quantized super-prior features are subjected to a super-prior inverse transformation to obtain an overall probability distribution, and the size of the input image, the quantized super-prior features, and the target detection results are respectively entropy coded, and then sequentially concatenated to obtain bitstream header information; in a second branch, the de-redundant image features are quantized to obtain quantized image features, grouped in combination with the target detection results to obtain quantized features corresponding to each target, entropy coded in combination with the overall probability distribution for the quantized features corresponding to each target, selecting all designated targets according to a task setting, and combining the entropy coded bitstreams corresponding to all designated targets to form a texture portion of the bitstream; the bitstream header information and the texture portion of the bitstream constitute a semantically structured bitstream;

[0078] A decoding unit is used to execute the decoding part, which includes: decoding the code stream header information in the semantically structured code stream to obtain the size, quantization super-prior features and target detection results of the input image, performing super-prior inverse transformation on the quantization super-prior features to obtain the overall probability distribution and bit rate control features; extracting the code stream corresponding to each specified target from the texture part, combining the overall probability distribution, respectively performing entropy decoding to obtain the group features corresponding to each specified group, and recombining the quantization features corresponding to all specified targets into recombined quantization image features according to the positions provided by the target detection results; combining the size information of the input image, the recombined quantization image features, the bit rate control features, and the foreground mask calculated using the target detection results to obtain a reconstructed image through an inverse transformation operation.

[0079] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0080] Example 3

[0081] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0082] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0083] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0084] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0085] The output device may be a display terminal;

[0086] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0087] Example 4

[0088] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0089] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0090] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A variable bit rate semantic structured image encoding and decoding method, characterized in that: include: Encoding part: perform target detection on the input image and obtain target detection results; Combining an input image with a generated rate control mask of the same spatial scale as the input image to perform a transformation to obtain de-redundant image features; The de-redundant image features are input into two branches. In the first branch, the de-redundant image features are subjected to super-prior transformation and quantization to obtain quantized super-prior features, the quantized super-prior features are subjected to super-prior inverse transformation to obtain the overall probability distribution, and the size of the input image and the quantized super-prior features and the target detection results are respectively entropy coded, and then sequentially spliced to obtain the code stream header information; in the second branch, the de-redundant image features are quantized to obtain quantized image features, grouped in combination with the target detection results to obtain the quantized features corresponding to each target, and the quantized features corresponding to each target are respectively entropy coded in combination with the overall probability distribution. All designated targets are selected according to the task settings, and the entropy coded code streams corresponding to all designated targets are combined to form the texture part of the code stream; the code stream header information and the texture part of the code stream constitute a semantically structured code stream; Decoding part: Decodes the stream header information in the semantically structured stream to obtain the input image size, quantized super-prior features and target detection results, performs super-prior inverse transformation on the quantized super-prior features to obtain the overall probability distribution and bit rate control features; Extract the bitstream corresponding to each specified target from the texture portion, combine it with the overall probability distribution, and perform entropy decoding to obtain the group features corresponding to each specified group. Based on the position provided by the target detection result, the quantized features corresponding to all specified targets are recombined into a recombined quantized image feature. Combine the size information of the input image, the recombined quantized image feature, the bitrate control feature, and the foreground mask calculated using the target detection result, and perform an inverse transform operation to obtain the reconstructed image. The steps of obtaining a reconstructed image through an inverse transformation operation by combining quantized image features, rate control features, and a foreground mask calculated using target detection results include: Set up two types of inverse transformations. The number of inverse transformations of the first type is L′. The inverse transformation process is the same each time. The l′th inverse transformation process is as follows: Adaptively upsample the foreground mask to generate a foreground mask m that is consistent with the input feature space scale l′ ; The quantized image features and the rate control features are cascaded in the channel dimension, and then the modulation features are generated after nonlinear transformation and upsampling. Perform nonlinear transformation on the quantized image features to obtain features Combined with modulation characteristics and the foreground mask m l′ Perform spatial modulation to obtain the first ′ The characteristics of the inverse transform; where the spatial modulation is expressed as: and is the adjustment factor, by adjusting the modulation characteristics Perform nonlinear transformation to obtain; Among them, l ′ =1,2,…,L′; when l ′ =1, the input feature is the recombined quantized image feature. ′ =2,…,L′, the input feature is the feature of the l-1th inverse transformation; The second type of inverse transformation has a number of 1. After L' inverse transformations, the output features can be reconstructed by another inverse transformation. The second type of inverse transformation includes: feature transformation and channel reduction operations performed sequentially.

2. The variable bit rate semantic structured image encoding and decoding method according to claim 1, characterized in that: The rate control mask is generated using the rate control factor r. The size of the rate control mask is H×W, where H is the height of the input image and W is the width of the input image. The value of each position in the rate control mask is a real value between 0 and 1, and the size of the value represents the relative size of the rate allocated to the pixel at the corresponding position in the input image during the encoding process.

3. The variable bit rate semantic structured image encoding and decoding method according to claim 1 or 2, characterized in that: The method of transforming the input image and the generated rate control mask having the same spatial scale as the input image includes: Set the number of transformations to L. The transformation process is the same each time. The lth transformation process is as follows: The input data and the rate control mask are concatenated in the channel dimension and then nonlinearly transformed and downsampled to generate the modulation feature t l ; Perform nonlinear transformation on the input data to obtain feature y l , combined with the modulation characteristics t l Perform spatial modulation to obtain the characteristics of the lth transformation; spatial modulation is expressed as: γ l ⊙y l +β l ; Among them, γ l and β l is the adjustment factor, which is obtained by adjusting the modulation characteristic t l Perform nonlinear transformation to obtain: l=1,…,L; when l=1, the input data is the input image x, and when l=2,…,L, the input data is the feature of the l-1th transformation.

4. The variable bit rate semantic structured image encoding and decoding method according to claim 1, characterized in that: The target detection results include: the number of targets, the position and category of each target; wherein the position and category of the kth target are expressed as: {a k ,b k ,h k ,w k ,cls k }, a k ,b k ,h k ,w k ,cls k They represent the horizontal position of the upper left corner coordinate, the vertical position of the upper left corner coordinate, the height, the width and the category of the k-th target respectively.

5. The variable bit rate semantic structured image encoding and decoding method according to claim 4, characterized in that: Performing entropy coding on the target detection results separately means performing entropy coding on the number of targets, the position and the category of each target separately; After entropy coding the quantized features corresponding to each target, all the required designated targets are selected according to different task settings. They are arranged in ascending order according to the numerical values of the serial numbers corresponding to the designated targets and combined to form the texture part of the bitstream.

6. The variable bit rate semantic structured image encoding and decoding method according to claim 4 or 5, characterized in that: The grammatical structure of the semantically structured bitstream includes: the height of the input image, the width of the input image, the bitstream length corresponding to the quantized super-prior feature, the number of targets, the position and category of each target, and the bitstream length corresponding to the texture part corresponding to all specified targets.

7. A variable bit rate semantically structured image encoding and decoding system, characterized in that: The method according to any one of claims 1 to 6 is implemented, and the system comprises: The encoding unit is configured to execute an encoding portion, the encoding portion comprising: performing target detection on an input image to obtain a target detection result; transforming the input image with a generated rate control mask of the same spatial scale as the input image to obtain de-redundant image features; inputting the de-redundant image features into two branches, wherein in a first branch, the de-redundant image features are subjected to a super-prior transformation and quantization to obtain quantized super-prior features, the quantized super-prior features are subjected to a super-prior inverse transformation to obtain an overall probability distribution, and the size of the input image, the quantized super-prior features, and the target detection results are respectively entropy coded, and then sequentially concatenated to obtain bitstream header information; in a second branch, the de-redundant image features are quantized to obtain quantized image features, grouped in combination with the target detection results to obtain quantized features corresponding to each target, entropy coded in combination with the overall probability distribution for the quantized features corresponding to each target, selecting all designated targets according to a task setting, and combining the entropy coded bitstreams corresponding to all designated targets to form a texture portion of the bitstream; the bitstream header information and the texture portion of the bitstream constitute a semantically structured bitstream; A decoding unit is used to execute the decoding part, which includes: decoding the code stream header information in the semantically structured code stream to obtain the size, quantization super-prior features and target detection results of the input image, performing super-prior inverse transformation on the quantization super-prior features to obtain the overall probability distribution and bit rate control features; extracting the code stream corresponding to each specified target from the texture part, combining the overall probability distribution, respectively performing entropy decoding to obtain the group features corresponding to each specified group, and recombining the quantization features corresponding to all specified targets into recombined quantization image features according to the positions provided by the target detection results; combining the size information of the input image, the recombined quantization image features, the bit rate control features, and the foreground mask calculated using the target detection results to obtain a reconstructed image through an inverse transformation operation.

8. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images