Pre-analysis based image compression method

Abstract

The present disclosure provides pre-analysis based methods for adaptive compression of image data processed by machine vision tasks. An exemplary method includes receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, wherein the encoding includes performing instance segmentation on an input picture to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merged mask; extracting a region from the input picture containing the merged mask; and compressing image data representing the extracted region.

Description

[0001] Cross-references to related applications

[0002] This disclosure claims priority to U.S. Provisional Application 63 / 378,888, filed October 10, 2022, and U.S. Patent Application 18 / 478,680, filed September 29, 2023, entitled “Image Compression Method Based on Pre-analysis,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to video processing, and more specifically, to methods and apparatus for performing pre-analysis-based image / video compression for machine vision tasks. Background Technology

[0004] With the rise of machine learning technologies and machine vision applications, the amount of video and images processed by machines (collectively referred to as "image data") is growing rapidly. Typical use cases include autonomous driving, intelligent transportation, smart cities, and intelligent content management, which involve machine vision tasks such as object detection, instance segmentation, and object tracking.

[0005] Machine vision tasks require massive amounts of image data, necessitating compression for efficient transmission and storage. However, traditional image / video compression techniques have focused on ensuring image / video quality as perceived by humans, while machines process and understand visual data differently. Therefore, image / video compression techniques adapted for machine vision may differ from traditional methods. Consequently, new compression techniques are needed to achieve performance optimization for machine use. Summary of the Invention

[0006] This disclosure provides a method for adaptively compressing image data processed by machine vision tasks based on pre-analysis. In some embodiments, an exemplary image data processing method includes: receiving a video sequence; encoding one or more input images associated with the video sequence; and generating a bitstream, wherein the encoding includes: performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region.

[0007] In some embodiments, a non-transitory computer-readable storage medium stores a bitstream generated by: performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region to generate the bitstream.

[0008] In some embodiments, the image data processing apparatus includes: a memory storing an instruction set; and one or more processors configured to execute the instruction set to cause the apparatus to perform operations, including: performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region.

[0009] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, cause the computer to perform a method comprising: performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region to generate a bitstream.

[0010] In some embodiments, an image data processing method includes: receiving a bit stream; and decoding one or more images using encoding information of the bit stream, wherein the decoding includes: receiving a bit stream, the bit stream including compressed image data with region information, wherein the region information indicates a region represented by the compressed image data in an image; and decoding the compressed image data based on the region information to reconstruct the image.

[0011] In some embodiments, a non-transitory computer-readable storage medium stores a bit stream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in an image, and the bit stream is processed by a method comprising: decoding the compressed image data based on the region information to reconstruct the image.

[0012] In some embodiments, an image data processing apparatus includes: a memory storing an instruction set; and one or more processors configured to execute the instruction set to cause the apparatus to perform operations including: receiving a bitstream including compressed image data having region information, wherein the region information indicates the region represented by the compressed image data in an image; and decoding the compressed image data based on the region information to reconstruct the image.

[0013] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, cause the computer to perform a method comprising: receiving a bit stream, the bit stream including compressed image data having region information, wherein the region information indicates the region represented by the compressed image data in an image; and decoding the compressed image data based on the region information to reconstruct the image.

[0014] In some embodiments, a computer program product is provided, the computer program product comprising: computer program instructions, and computer program instructions that enable a computer to execute the image data processing method according to the above-described image data processing method embodiments.

[0015] In some embodiments, a computer program is provided that enables a computer to execute the image data processing method according to the above-described image data processing method embodiments. Attached Figure Description

[0016] Embodiments and aspects of this disclosure are illustrated in the following detailed description and accompanying drawings. Various features shown in the figures are not drawn to scale.

[0017] Figure 1 A schematic diagram of an exemplary system for preprocessing and encoding image data according to some embodiments of the present disclosure is shown.

[0018] Figure 2A A schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system consistent with embodiments of this disclosure is shown.

[0019] Figure 2B A schematic diagram of another exemplary encoding process of a hybrid video encoding system consistent with embodiments of this disclosure is shown.

[0020] Figure 3A A schematic diagram illustrating an exemplary decoding process of a hybrid video coding system consistent with embodiments of this disclosure is shown.

[0021] Figure 3B A schematic diagram of another exemplary decoding process of a hybrid video coding system consistent with embodiments of this disclosure is shown.

[0022] Figure 4 The present disclosure illustrates a block diagram of an exemplary apparatus for preprocessing or encoding image data, according to some embodiments thereof.

[0023] Figure 5 A schematic diagram of an exemplary compression method for visual data compression for machine vision, according to some embodiments of the present disclosure, is shown.

[0024] Figure 6A-6E schematically illustrates various methods for preprocessing input images based on pre-analysis results, according to some embodiments of the present disclosure.

[0025] Figure 7 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0026] Figure 8 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0027] Figure 9 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0028] Figure 10 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0029] Figure 11 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0030] Figure 12 This is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0031] Figure 13 A-13D schematically illustrates, according to some embodiments of the present disclosure, preprocessing an input image based on pre-analysis results.

[0032] Figure 14 A and Figure 14 B schematically illustrates, according to some embodiments of the present disclosure, block-based preprocessing of an input image.

[0033] Figure 15 A-15D schematically illustrates, according to some embodiments of the present disclosure, preprocessing an input image based on dilation.

[0034] Figure 16 This is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure.

[0035] Figure 17 This is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, wherein, unless otherwise stated, the same reference numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of this disclosure recounted in the appended claims. Specific aspects of this disclosure are described below in more detail. In case of conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0037] This disclosure relates to machine video coding (VCM), which aims to compress input video and images or feature maps for machine vision tasks. Specifically, according to an exemplary embodiment, prior to encoding, the input image is pre-analyzed to identify segmentation masks corresponding to objects or foregrounds in the input image. The segmentation masks are further pre-processed to form a merged mask, which is then encoded (i.e., compressed). In some embodiments, the pre-processing may also identify extended regions around the merged mask. These extended regions are also compressed by the encoder.

[0038] Consistent with the disclosed embodiments, the pre-analysis and preprocessing can be performed by an image data preprocessor separate from the encoder. Alternatively, the pre-analysis can be performed by the encoder itself. This disclosure does not limit the hardware or software architecture used to implement the image data pre-analysis.

[0039] The disclosed technology is applicable to compressing image data used in any machine vision task, such as object recognition and tracking, face recognition, image / video search, mobile augmented reality (MAR), autonomous vehicles, Internet of Things (IoT), image matching, 3D structure construction, stereo correspondence, motion tracking, etc.

[0040] Figure 1 A block diagram of a system 100 for preprocessing and encoding image data, according to some disclosed embodiments, is shown. The image data may include images (also referred to as “pictures” or “frames”), multiple images, or video. Images are still pictures. Multiple images may be spatially or temporally related or unrelated. Video is a set of images arranged in chronological order.

[0041] like Figure 1As shown, system 100 includes source device 120, which provides encoded video data subsequently decoded by target device 140. Consistent with the disclosed embodiments, each of source device 120 and target device 140 may include any of a variety of devices, including: desktop computers, laptops (e.g., notebook computers), servers, tablets, set-top boxes, mobile phones, vehicles, cameras, image sensors, robots, televisions, wearable devices (e.g., smartwatches or wearable cameras), display devices, digital media players, video game consoles, video streaming devices, and the like. Source device 120 and target device 140 may be configured for wireless or wired communication.

[0042] refer to Figure 1 The source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. The target device 140 may include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. The image / video preprocessor 122 preprocesses image data, i.e., one or more images or one or more videos, and generates an input bitstream for the image / video encoder 124. The image / video encoder 124 encodes the input bitstream and outputs an encoded bitstream 162 via the output interface 126. The encoded bitstream 162 is transmitted via a communication medium 160 and received by the input interface 142. The image / video decoder 144 then decodes the encoded bitstream 162 to generate decoded data, which can be used by the machine vision application 146.

[0043] More specifically, the source device 120 may further include various devices (not shown) for providing source image data to be preprocessed by the image / video preprocessor 122. The devices for providing the source image data may include image / video capture devices, such as cameras, image / video archiving or storage devices containing previously captured images / videos, or image / video feed interfaces for receiving images / videos from image / video content providers.

[0044] Image / video encoder 124 and image / video decoder 144 can each be implemented in any of a variety of suitable encoder or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When encoding or decoding is implemented partially in software, image / video encoder 124 or image / video decoder 144 may store multiple instructions for software in a suitable, non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform techniques consistent with this disclosure. Each of image / video encoder 124 or image / video decoder 144 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) in the respective device.

[0045] The image / video encoder 124 and image / video decoder 144 can operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), AOMedia Video 1 (AV1), Joint Picture Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc. Alternatively, the image / video encoder 124 and image / video decoder 144 can be custom devices that do not conform to existing standards. Although in Figure 1 As not shown, but in some embodiments, the image / video encoder 124 and image / video decoder 144 may each be integrated with the audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units (MUX-DEMUX), or other hardware and software, to process the encoding of audio and video in a common data stream or separate data streams.

[0046] Output interface 126 may include any type of medium or device capable of transmitting encoded bit stream 162 from source device 120 to target device 140. For example, output interface 126 may include a transmitter or transceiver configured to transmit encoded bit stream 162 directly from source device 120 to target device 140 in real time. Encoded bit stream 162 may be modulated according to a communication standard such as a wireless communication protocol and transmitted to target device 140.

[0047] Communication medium 160 may include transient media, such as wireless broadcasting or wired network transmission. For example, communication medium 160 may include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). Communication medium 160 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. In some embodiments, communication medium 160 may include a router, a switch, a base station, or any other device that can be used to facilitate communication from source device 120 to target device 140. For example, a network server (not shown) may receive encoded bit stream 162 from source device 120 and provide encoded bit stream 162 to target device 140, for example, via network transmission.

[0048] The communication medium 160 may also take the form of a storage medium (e.g., a non-transient storage medium), such as a hard disk, flash drive, optical disk, digital video disk, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, the computer device of a media generation apparatus (e.g., an optical disk pressing apparatus) may receive encoded image data from the source device 120 and generate a disk containing the encoded video data.

[0049] Input interface 142 may include any type of medium or device capable of receiving information from communication medium 160. The received information includes encoded bit stream 162. For example, input interface 142 may include a receiver or transceiver configured to receive encoded bit stream 162 in real time.

[0050] Machine vision application 146 includes various hardware and / or software for utilizing decoded image data generated by image / video decoder 144. For example, machine vision application 146 may include a display device for displaying the decoded image data to a user, and may include any of a variety of display devices, such as cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices. As another example, machine vision application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications, such as object recognition and tracking, face recognition, image matching, image / video search, augmented reality, robot vision and navigation, autonomous driving, 3D structure building, stereo mapping, motion tracking, etc.

[0051] Next, combined Figure 2A-2B and Figures 3A-3B This describes exemplary image data encoding and decoding techniques.

[0052] Figure 2AA schematic diagram of an example encoding process 200A consistent with embodiments of this disclosure is shown. For example, the encoding process 200A may be performed by, for example... Figure 1 Encoders such as image / video encoder 124 are executed. Figure 2A As shown, the encoder can encode video sequence 202 into video bitstream 228 according to process 200A. Video sequence 202 may include a set of images (referred to as "raw images") arranged in chronological order. Each raw image of video sequence 202 may be divided by the encoder into multiple basic processing units, multiple basic processing sub-units, or multiple regions for processing. In some embodiments, the encoder may perform process 200A at the basic processing unit level for each raw image of video sequence 202. For example, the encoder may perform process 200A iteratively, wherein the encoder may encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for multiple regions of each raw image of video sequence 202.

[0053] exist Figure 2A In this process, the encoder feeds a basic processing unit (referred to as the "raw BPU") of an original image of video sequence 202 to prediction stage 204 to produce prediction data 206 and prediction BPU 208. The encoder can subtract prediction BPU 208 from the raw BPU to produce residual BPU 210. The encoder can feed residual BPU 210 to transform stage 212 and quantization stage 214 to produce quantization transform coefficients 216. The encoder can feed prediction data 206 and quantization transform coefficients 216 to binary encoding stage 226 to produce video bitstream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as the "forward path". During process 200A, after quantization stage 214, the encoder can feed quantization transform coefficients 216 to inverse quantization stage 218 and inverse transform stage 220 to generate reconstructed residual BPU 222. The encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224, which is used in the next iteration of process 200A during the prediction phase 204. Components 218, 220, 222, and 224 of process 200A can be referred to as "reconstruction paths". The reconstruction paths can be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0054] The encoder can iteratively execute process 200A to encode each raw BPU of the original image (in the forward path) and generate a prediction reference 224 for encoding the next raw BPU of the original image (in the reconstruction path). After encoding all raw BPUs of the original image, the encoder can continue to encode the next image in the video sequence 202.

[0055] Referring to process 200A, the encoder may receive a video sequence 202 generated by a video capture device (e.g., a camera). As used herein, the term "receive" may refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or otherwise inputting data.

[0056] In prediction phase 204, during the current iteration, the encoder may receive a raw BPU and a prediction reference 224, and perform prediction operations to generate prediction data 206 and a prediction BPU 208. The prediction reference 224 can be generated from a reconstruction path in a previous iteration of process 200A. The purpose of prediction phase 204 is to reduce information redundancy by extracting prediction data 206 from the prediction data 206 and prediction reference 224 that can be used to reconstruct the raw BPU into prediction BPU 208.

[0057] Ideally, the predicted BPU 208 should be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is typically slightly different from the original BPU. To record this difference, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate the residual BPU 210. For example, the encoder can subtract the pixel values ​​(e.g., grayscale or RGB values) of the predicted BPU 208 from the corresponding pixel values ​​of the original BPU. Each pixel value of the residual BPU 210 can have a residual value as the result of this subtraction between the corresponding pixel values ​​of the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without a significant quality degradation. Thus, the original BPU is compressed.

[0058] To further compress the residual BPU 210, in the transform phase 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing it into a set of two-dimensional “fundamental patterns,” each fundamental pattern being associated with a “transform coefficient.” The fundamental patterns can have the same size (e.g., the size of the residual BPU 210). Each fundamental pattern can represent a variable frequency component of the residual BPU 210 (e.g., the frequency of brightness variation). No single fundamental pattern can be reproduced from any combination of any other fundamental patterns (e.g., a linear combination). In other words, the decomposition can decompose multiple variations of the residual BPU 210 into the frequency domain. This decomposition is analogous to the discrete Fourier transform of a function, where the fundamental patterns are analogous to the basis functions of the discrete Fourier transform (e.g., trigonometric functions), and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0059] Different transform algorithms can use different base patterns. Various transform algorithms, such as discrete cosine transform, discrete sine transform, etc., can be used in transform stage 212. The transform at transform stage 212 is reversible. That is, the encoder can recover the residual BPU 210 through the inverse operation of the transform (called the "inverse transform"). For example, to recover a pixel of the residual BPU 210, the inverse transform can be to multiply the corresponding pixel values ​​of multiple base patterns by their respective correlation coefficients and sum the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore have the same base pattern). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 based on these transform coefficients without receiving the base pattern from the encoder. Compared to the residual BPU 210, the transform coefficients can have fewer bits, but they can be used to reconstruct the residual BPU 210 without significant quality degradation. Therefore, the residual BPU 210 is further compressed.

[0060] The encoder can further compress the transform coefficients in quantization phase 214. During the transform process, different fundamental modes can represent different frequencies of change (e.g., brightness change frequencies). Since the human eye is generally better at recognizing low-frequency changes, the encoder can ignore information about high-frequency changes without causing a significant degradation in decoding quality. For example, in quantization phase 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to its nearest integer. Through this operation, some transform coefficients of the high-frequency fundamental modes can be converted to zero, while the transform coefficients of the low-frequency fundamental modes can be converted to smaller integers. The encoder can ignore the zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed into transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0061] Because the encoder ignores the remainder of such division during rounding operations, quantization stage 214 can be lossy. Typically, quantization stage 214 can cause the greatest information loss in process 200A. The greater the information loss, the fewer bits the quantization transform coefficients 216 may require. To obtain different levels of information loss, the encoder can use different values ​​for the quantization parameters or any other parameters in the quantization process.

[0062] In the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantization transform coefficients 216 using binary encoding techniques, such as entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 206 and the quantization transform coefficients 216, the encoder may also encode other information at the binary encoding stage 226, such as the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type in the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packaged for network transmission.

[0063] Referring to the reconstruction path of process 200A, in the inverse quantization stage 218, the encoder can perform inverse quantization on the quantization transform coefficients 216 to generate reconstruction transform coefficients. In the inverse transform stage 220, the encoder can generate a reconstruction residual BPU 222 based on the reconstruction transform coefficients. The encoder can add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0064] It should be noted that other variations of process 200A can be used to encode video sequence 202. In some embodiments, the stages of process 200A can be performed by the encoder in different orders. In some embodiments, one or more stages of process 200A can be combined into a single stage. In some embodiments, a single stage of process 200A can be divided into multiple stages. For example, transform stage 212 and quantization stage 214 can be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may be omitted. Figure 2A One or more stages in the process.

[0065] Figure 2B A schematic diagram of another exemplary encoding process 200B consistent with embodiments of this disclosure is shown. For example, the encoding process 200B may be performed by, for example... Figure 1 The process is executed by an encoder such as the image / video encoder 124. Process 200B can be obtained by modifying process 200A. For example, process 200B can be used by an encoder that conforms to a hybrid video coding standard (e.g., H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filtering stage 232 and a buffer 234.

[0066] Generally, prediction techniques can be divided into two categories: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or "intra-frame prediction") can use pixels from one or more encoded neighboring BPUs in the same image to predict the current BPU. That is, the prediction reference 224 in spatial prediction can include the neighboring BPUs. Spatial prediction can reduce the inherent spatial redundancy of the image. Temporal prediction (e.g., inter-image prediction or "inter-frame prediction") can use multiple regions from one or more encoded images to predict the current BPU. That is, the prediction reference 224 in temporal prediction can include multiple encoded images. Temporal prediction can reduce the inherent temporal redundancy of the multiple images.

[0067] Referring to process 200B, in the forward path, the encoder performs the prediction operation in spatial prediction phase 2042 and temporal prediction phase 2044. For example, in spatial prediction phase 2042, the encoder may perform intra-frame prediction. For a given original BPU of a given image being encoded, prediction reference 224 may include one or more adjacent BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same image. The encoder can generate a predicted BPU 208 by extrapolating multiple adjacent BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform the extrapolation at the pixel level, for example, by extrapolating the corresponding pixel value for each pixel of the predicted BPU 208. The multiple adjacent BPUs used for extrapolation can be located from various directions relative to the original BPU, such as in the vertical direction (e.g., at the top of the original BPU), the horizontal direction (e.g., to the left of the original BPU), the diagonal direction (e.g., the lower left, lower right, upper left, or upper right of the original BPU), or any direction defined in the video coding standard used. For intra-frame prediction, prediction data 206 may include, for example, the positions (e.g., coordinates) of the multiple adjacent BPUs used, the sizes of the multiple adjacent BPUs used, the extrapolation parameters, the orientation of the adjacent BPUs used relative to the original BPU, and so on.

[0068] For example, in the temporal prediction phase 2044, the encoder can perform the inter-frame prediction. For a given original BPU of the current image, the prediction reference 224 may include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference image can be encoded and the BPU reconstructed using the BPU. For example, the encoder can add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. After all reconstructed BPUs for the same image have been generated, the encoder can generate a reconstructed image as the reference image. The encoder can perform a "motion estimation" operation to search for matching regions within the range of the reference image (referred to as a "search window"). The position of the search window in the reference image can be determined based on the position of the original BPU in the current image. For example, the search window can be centered at a position in the reference image that has the same coordinates as the original BPU in the current image and can extend outward by a predetermined distance. When the encoder identifies a region similar to the original BPU in the search window (e.g., by using a pixel recursive algorithm, block matching algorithm, etc.), the encoder can determine such a region as a matching region. The matching region may have a different size than the original BPU (e.g., smaller, equal to, greater than, or with a different shape). Since the reference image and the current image are temporally separated on the timeline, the matching region can be considered to "move" to the position of the original BPU over time. The encoder can record the direction and distance of this movement as a "motion vector." When using multiple reference images, the encoder can search for a matching region and determine the relevant motion vector for each reference image. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching regions of each matching reference image.

[0069] The motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the position (e.g., coordinates) of the matching region, multiple motion vectors associated with the matching region, the number of reference images, multiple weights associated with the multiple reference images, etc.

[0070] To generate the predicted BPU 208, the encoder can perform a "motion compensation" operation. This motion compensation can be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., the motion vector) and the prediction reference 224. For example, the encoder can move the matching region of the reference image according to the motion vector to predict the original BPU of the current image. When using multiple reference images, the encoder can move the matching regions of multiple reference images according to their respective motion vectors and average pixel values. In some embodiments, if the encoder has already assigned weights to the pixel values ​​of the matching regions of each matching reference image, the encoder can add a weighted sum of the pixel values ​​of the multiple moved matching regions.

[0071] In some embodiments, inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference images in the same time direction relative to the current image. Bidirectional inter-frame prediction can use a reference image preceding the current image. Bidirectional inter-frame prediction can use one or more reference images in two different time directions relative to the current image.

[0072] Referring again to the forward path of process 200B, after spatial prediction 2042 and temporal prediction stages 2044, in mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra-frame prediction or inter-frame prediction) for the current iteration of process 200B. For example, the encoder can perform rate-distortion optimization techniques, wherein the encoder can select a prediction mode to minimize the value of a cost function based on the bitrate of a candidate prediction mode and the distortion of reconstructing a reference image under that candidate prediction mode. Based on the selected prediction mode, the encoder can generate the corresponding prediction BPU 208 and prediction data 206.

[0073] In the reconstruction path of process 200B, if intra-frame prediction mode has been selected in the forward path, after generating prediction reference 224 (e.g., the current BPU that has been encoded and reconstructed in the current image), the encoder can directly feed prediction reference 224 to spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU in the current image). If inter-frame prediction mode has been selected in the forward path, after generating prediction reference 224 (e.g., all BPUs have been encoded and reconstructed in the current image), the encoder can feed prediction reference 224 to loop filtering stage 232, where the encoder can apply loop filtering to prediction reference 224 to reduce or eliminate distortion (e.g., blockiness) introduced by the inter-frame prediction. The encoder can apply various loop filtering techniques in loop filtering stage 232, such as deblocking, sample adaptive shifting, adaptive loop filtering, etc. The loop filter reference image can be stored in buffer 234 (or "decoded image buffer") for subsequent use (e.g., as an inter-frame prediction reference image for a future image of video sequence 202). The encoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder can encode the loop filter parameters (e.g., loop filter strength), as well as the quantization transform coefficients 216, prediction data 206, and other information in the binary encoding stage 226.

[0074] Figure 3A A schematic diagram of an exemplary decoding process 300A consistent with embodiments of the present disclosure is shown. For example, the decoding process 300A may be performed by, for example... Figure 1 The process is executed by a decoder such as the image / video decoder 144. Process 300A can be... Figure 2A The decompression process corresponding to compression process 200A. In some embodiments, process 300A may be similar to the reconstruction path described in process 200A. Decoder (e.g., Figure 1 The image / video decoder 144 in the image / video decoder can decode the video bitstream 228 into a video stream 304 according to process 300A. The video stream 304 can be very similar to the video sequence 202. However, due to compression and decompression processes (e.g., Figure 2A-2B Information loss during the quantization stage 214) typically results in video stream 304 not being entirely identical to video sequence 202. Similar to... Figure 2A-2BIn processes 200A and 200B, for each image encoded in video bitstream 228, the decoder can perform process 300A at the basic processing unit (BPU) level. For example, the decoder can perform process 300A iteratively, wherein the decoder can decode one basic processing unit in one iteration of process 300A. In some embodiments, the decoder can perform process 300A in parallel on regions of each image encoded in video bitstream 228.

[0075] exist Figure 3A In this process, the decoder may feed a portion of the video bitstream 228 associated with a basic processing unit (referred to as an "encoded BPU") of an encoded image to the binary decoding stage 302. In the binary decoding stage 302, the decoder may decode this portion into prediction data 206 and quantization transform coefficients 216. The decoder may feed the quantization transform coefficients 216 to the inverse quantization stage 218 and the inverse transform stage 220 to generate a reconstruction residual BPU 222. The decoder may feed the prediction data 206 to the prediction stage 204 to generate a prediction BPU 208. The decoder may add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded image buffer in computer memory). The decoder may feed the prediction reference 224 to the prediction stage 204 for use in the next iteration of process 300A to perform a prediction operation.

[0076] The decoder can iteratively execute process 300A to decode each encoded BPU of the encoded image and generate a prediction reference 224 for encoding the next encoded BPU of the encoded image. After decoding all encoded BPUs of the encoded image, the decoder can output the image to video stream 304 for display and continue decoding the next encoded image in video bit stream 228.

[0077] In binary decoding stage 302, the decoder may perform the inverse operation of the binary encoding technique used by the encoder (e.g., entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to prediction data 206 and quantization transform coefficients 216, the decoder may also decode other information in binary decoding stage 302, such as prediction mode, parameters of prediction operation, transform type, parameters of quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. In some embodiments, if the video bitstream 228 is transmitted over the network in the form of data packets, the decoder may unpack the video bitstream 228 before feeding it to binary decoding stage 302.

[0078] Figure 3B A schematic diagram of another exemplary decoding process 300B consistent with embodiments of this disclosure is shown. For example, the decoding process 300B may be performed by, for example... Figure 1 The process is performed by a decoder such as image / video decoder 144. Process 300B can be obtained by modifying process 300A. For example, process 300B can be used by a decoder that conforms to a hybrid video coding standard (e.g., H.26x series). Compared to process 300A, process 300B further divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and also includes a loop filtering stage 232 and a buffer 234.

[0079] In process 300B, for an encoded basic processing unit (referred to as the "current BPU") of an encoded image being decoded (referred to as the "current image"), the prediction data 206 decoded by the decoder from binary decoding stage 302 can contain various types of data depending on the prediction mode used by the encoder to encode the current BPU. For example, if the encoder uses intra-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating that intra-frame prediction is used, parameters for the intra-frame prediction operation, etc. Parameters for the intra-frame prediction operation may include, for example, the positions (e.g., coordinates) of one or more neighboring BPUs used as references, the sizes of the neighboring BPUs, extrapolation parameters, the orientations of the neighboring BPUs relative to the original BPU, etc. As another example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating that inter-frame prediction is used, parameters for the inter-frame prediction operation, etc. The parameters of the inter-frame prediction operation may include, for example, the number of multiple reference images associated with the current BPU, the weights associated with each of the multiple reference images, the positions (e.g., coordinates) of one or more matching regions in each of the reference images, one or more motion vectors associated with each matching region, and so on.

[0080] Based on the prediction mode indicator, the decoder can determine whether to perform spatial prediction (e.g., intra-frame prediction) in the spatial prediction phase 2042 or temporal prediction (e.g., inter-frame prediction) in the temporal prediction phase 2044. Details regarding the execution of such spatial or temporal prediction are provided in... Figure 2B The description has been provided and will not be repeated below. After performing such spatial or temporal prediction, the decoder can generate a prediction BPU 208. The decoder can then add the prediction BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, as shown below. Figure 3A As shown in the image.

[0081] In process 300B, the decoder can feed the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 for performing a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intra-frame prediction in the spatial prediction stage 2042, then after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder can feed the prediction reference 224 directly to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU in the current image). If the current BPU is decoded using inter-frame prediction in the temporal prediction stage 2044, then after generating the prediction reference 224 (e.g., in a reference image where all BPUs have been decoded), the encoder can feed the prediction reference 224 to the loop filtering stage 232 to reduce or eliminate distortion (e.g., blockiness). The decoder can, as shown in... Figure 2B The method described herein applies loop filtering to prediction reference 224. The loop-filtered reference image can be stored in buffer 234 (e.g., a decoded image buffer in computer memory) for subsequent use (e.g., as an inter-frame prediction reference image used as a subsequent encoded image of video bitstream 228). The decoder can store one or more reference images in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, when the prediction mode indicator of prediction data 206 indicates that inter-frame prediction is used to encode the current BPU, the prediction data may further include parameters of the loop filtering (e.g., loop filtering strength).

[0082] Return to reference Figure 1 The image / video preprocessor 122, the image / video encoder 124, and the image / video decoder 144 can all be implemented using any suitable hardware, software, or a combination thereof. Figure 4 This is a block diagram of an example apparatus 400 for processing image data, consistent with embodiments of this disclosure. For example, apparatus 400 may be a preprocessor, encoder, or decoder. Figure 4As shown, device 400 may include processor 402. When processor 402 executes the instructions described herein, device 400 may become a dedicated machine for preprocessing, encoding, and / or decoding image data. Processor 402 may be any type of circuit capable of manipulating or processing information. For example, processor 402 may include any number of CPU processing units (or “CPU”), graphics processing units (or “GPU”), neural processing units (“NPU”), microcontroller units (“MCU”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property cores (IP cores), programmable logic arrays (PLAs), programmable array logic (PALs), general-purpose array logic (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), application-specific integrated circuits (ASICs), or combinations of the like. In some embodiments, processor 402 may also be a group of processors grouped into individual logic components. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b and processor 402n.

[0083] The device 400 may also include a memory 404 configured to store data (e.g., instruction sets, computer code, intermediate data, etc.). For example, such as Figure 4 As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). The processor 402 can access the processed program instructions and data (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the processed data. The memory 404 may include a high-speed random access memory device or a non-transient storage device. In some embodiments, the memory 404 may include any number of random access memories (RAM), read-only memories (ROM), optical discs, magnetic disks, hard disks, solid-state drives, flash drives, secure digital cards (SD cards), memory sticks, compact flash memory (CF cards), and any combination thereof. The memory 404 may also be a group of memories grouped into individual logical components. Figure 4 (Not shown in the image).

[0084] Bus 410 may be a communication device for transmitting data between components within device 400, such as an internal bus (e.g., CPU-memory bus), an external bus (e.g., a Universal Serial Bus port, a Peripheral Component Interconnect Fast Port), and so on.

[0085] For ease of explanation and to avoid ambiguity, processor 402 and other data processing circuitry are collectively referred to as "data processing circuitry" in this disclosure. The data processing circuitry may be implemented entirely in hardware or as a combination of software, hardware, or firmware. Furthermore, the data processing circuitry may be a single, independent module or may be wholly or partially integrated into any other component of device 400.

[0086] The device 400 may further include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, intranet, local area network, mobile communication network, etc.). In some embodiments, the network interface 406 may include any combination of any number of network interface controllers (NICs), radio frequency (RF) modules, transceivers, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, or other similar devices.

[0087] In some embodiments, optionally, the device 400 may further include a peripheral interface 408 to provide connectivity to one or more peripheral devices. For example... Figure 4 As shown, the peripheral devices may include, but are not limited to, cursor control devices (e.g., mouse, touchpad, or touchscreen), keyboards, displays (e.g., cathode ray tube displays, liquid crystal displays, or light-emitting diode displays), video input devices (e.g., cameras or input interfaces coupled to video archives), and so on.

[0088] It should be noted that the video codec (e.g., the codec for executing processes 200A, 200B, 300A, or 300B) can be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. As another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuitry (e.g., FPGA, ASIC, NPU, etc.).

[0089] With the development of multimedia processing, transmission, and applications, visual data is experiencing explosive growth. Furthermore, the storage and transmission of raw visual data is impractical due to the enormous costs involved. Simultaneously, visual data exhibits significant redundancy, prompting the development of compressed representations that enable storage and transmission in practical applications. Typically, the end-user of visual data is human vision, and compressed representations are designed for human perceptual quality. However, in recent years, with the significant advancements in artificial intelligence, numerous machine analysis applications based on deep learning for visual data have emerged, known as machine vision. In machine vision applications, machine analysis performance has superseded human perceptual quality as the ultimate metric for visual data compression, exceeding the scope of visual data compression algorithms designed for human vision. Therefore, there is a need to develop compressed representations for visual data specific to machine vision. To address this issue, this disclosure provides an image data compression method for machine vision using pre-analysis. Specifically, the pre-analysis for machine vision can extract key information for machine analysis to achieve better analysis performance and eliminate information irrelevant to machine analysis to obtain a more compact representation. Furthermore, the disclosed pre-analysis algorithm can be adapted to compression codecs for human vision, meaning it is a general-purpose technique for visual data compression for machine vision applications.

[0090] The development of image / video compression algorithms is driven by coding standards. For texture compression, a series of standards have been developed to compress visual data, such as JPEG and JPEG 2000 for still image compression, and H.264 / AVC, H.265 / HEVC, and VVC (Video Generalized Coding) for video data compression. To further improve compression performance, many algorithms have been developed for future video compression standards, including matrix-weighted intra-frame prediction, quadtree-plus-binary, extended coding unit partitioning, and mode-dependent non-separable quadratic transform. Simultaneously, various optimization algorithms have been proposed for encoder optimization to achieve rate-distortion optimization of texture and feature quality. Furthermore, with the rapid development of visual data understanding, managing thousands of visual data bitstreams compactly and transmitting them simultaneously for further analysis, such as in smart cities and the Internet of Things in Video (IoVT), presents a significant challenge. Moreover, the analytical performance can be significantly affected by the degradation of feature quality in compression aimed at human visual quality. To address this issue, the Moving Picture Experts Group (MPEG) has also developed a compressed visual feature representation standard to reduce the size of the representation data for analyzing features, thereby enabling various intelligent tasks through front-end intelligence. Specifically, standards for Compressed Descriptors for Visual Search (CDVS) ​​and Compressed Descriptors for Video Analytics (CDVA) have been finalized, aiming to obtain very compact descriptors for visual data. Furthermore, standardization for video coding for machines has been initiated, striving to paint a complete picture of compressed representations of visual data in machine vision.

[0091] Furthermore, deep learning can be used for a variety of applications, particularly in the fields of visual data representation and understanding. Specifically, end-to-end compression frameworks based on deep neural networks can be used. Recurrent Neural Networks (RNNs) can be applied to end-to-end learning of image representations to achieve performance comparable to JPEG. Inspired by block-based transforms in traditional image / video compression, end-to-end image compression models based on Convolutional Neural Networks (CNNs) can be combined with Discrete Cosine Transform (DCT) to achieve performance comparable to JPEG at low bitrates. Nonlinear transforms are one of the fundamental characteristics of neural networks, consistent with the Human Visual System (HVS). Therefore, the end-to-end nonlinear transform codec can be optimized using Generalized Segmentation Normalization (GDN) to improve perceptual quality. Based on this, density estimation models can be combined with cascaded GDNs to surpass the compression performance of JPEG2000. In end-to-end image compression, the redundancy of latent coding can be further eliminated under entropy penalty constraints through weight reparameterization, a process achieved through a variational super-prior model. To further leverage the relevance of latent representations, and to achieve superior rate-distortion performance comparable to the state-of-the-art image codec BPG in terms of PSNR and MS-SSIM distortion metrics, an autoregressive model can be used. To further improve the accuracy of the entropy model in bitrate estimation, the distribution of the latent representations can be parameterized using discrete Gaussian mixture likelihood. This allows for the development of a more accurate and flexible entropy model, achieving bitrate-PSNR performance comparable to the latest compression standard VVC.

[0092] Traditional visual data compression typically achieves its effect by compactly representing local and low-level information through block-by-block processing. However, this can limit the compression representation capability because high-level information is not fully utilized. To address this issue, pre-analysis can be used to improve the representation performance, as it can extract global and high-level information from visual data. Specifically, a visual attention-based pre-analysis model can be used to optimize coding parameter settings for better compression performance. Furthermore, pre-analysis can be applied to bitrate control in real-time video coding. From the perspective of simplifying coding complexity, pre-analysis can also be applied to accelerate integer motion estimation in JEM.

[0093] As mentioned above, visual data compression has seen significant development in recent decades. With advancements in various machine analytics tasks, several analytical feature compression algorithms have been proposed to improve the efficiency of visual data compression in machine vision and adapt to the rapid development of machine analytics applications. However, existing visual data codecs primarily focus on signal fidelity and human visual quality, rather than machine vision. To improve the representation efficiency of visual data and adapt to existing codecs, pre-analysis has been investigated, but existing pre-analysis methods mainly focus on human perceptual quality, which limits the compressed representation performance of visual data for machine vision.

[0094] This disclosure provides a pre-analysis-based compression method suitable for machine vision.

[0095] Figure 5 This is a schematic diagram illustrating an exemplary compression method 500 for visual data compression for machine vision, according to some embodiments consistent with this disclosure. For example, method 500 may be performed by an image / video preprocessor 122 ( Figure 1 ) Execute. For example Figure 5 As shown, method 500 includes two phases: a machine vision-based pre-analysis phase 510 and a preprocessing phase 520. Specifically, the machine vision-based pre-analysis phase 510 performs high-level information extraction, such as instance segmentation, using a machine analysis model and detects one or more segmentation masks 512 corresponding to one or more instances (i.e., objects or foregrounds). The pre-analysis phase 510 may also run a classifier (e.g., a neural network classifier) ​​to determine the class associated with each segmentation mask 512. Figure 5 (Not shown in the image). This class can be an object class that indicates a certain type of object associated with each segmentation mask (e.g., face, tree, animal, building, etc.). Preprocessing stage 520 includes operations formulated based on the output of pre-analysis stage 510 (e.g., segmentation masks and predicted categories) to strike a trade-off between machine vision performance and the compressibility of representations. Pre-analysis stage 510 can employ machine analysis models such as Faster R-CNN and Masked R-CNN (which may be referred to as Mask R-CNN or Mask R-CNN).

[0096] For example, Mask R-CNN is a deep learning model that combines object detection and instance segmentation. According to some embodiments, the overall architecture of an exemplary Mask R-CNN may include: (1) a backbone network, (2) a region candidate network (RPN), (3) a region of interest alignment (ROIAlign), and (4) a mask branch (MaskHead).

[0097] (1) Backbone Network: The backbone network can be a pre-trained convolutional neural network, such as a residual network (ResNet) or ResNeXt (Residual Next Step Network). The backbone network processes the input image and generates corresponding feature maps.

[0098] (2) Region Proposal Network (RPN): The RPN is responsible for generating region proposals or candidate bounding boxes that may contain objects within the image. It operates on the feature maps generated by the backbone network and recommends potential regions of interest (ROIs).

[0099] (3) ROIAlign: The main purpose of ROIAlign is to align the features within the region of interest (ROI) with the spatial grid of the output feature map. This alignment prevents information loss that may occur when quantizing the spatial coordinates of the ROI to the nearest integer (as is done in ROI pooling).

[0100] (4) Mask Head: The Mask Head is an additional branch in Mask R-CNN that is responsible for generating segmentation masks for each region proposal. This branch uses alignment features obtained through ROIAlign to predict a binary mask for each object, depicting the pixel-level boundaries of the instance.

[0101] During training, the Mask R-CNN model uses a combination of classification loss, bounding box regression loss, and mask segmentation loss for joint optimization. This enables the Mask R-CNN model to learn to simultaneously detect objects, refine their bounding boxes, and generate accurate segmentation masks.

[0102] As described above, once the pre-analysis stage 510 determines that one or more instances in the input image belong to a region of interest (ROI), the preprocessing stage 520 can further filter out these instances using a corresponding segmentation mask 512. The background portion of the input image can then be omitted and not involved in subsequent processing. Compared to the original input image, the filtered portion with less image information is more likely to achieve better image coding performance, especially for machine vision. In the above example, the machine vision pre-analysis model is instance segmentation, and the output of the pre-analysis stage includes multiple segmentation masks and predicted categories. However, it is contemplated that the disclosed embodiments are not limited to the above examples, and any suitable machine vision pre-analysis and preprocessing techniques can be used to improve the representation efficiency of visual data for machine vision.

[0103] Next, the details of preprocessing stage 520 will be described. Figure 6A-6E schematically illustrates various methods for preprocessing input images based on pre-analysis results according to some embodiments of this disclosure. In some embodiments, the preprocessing operation can be performed directly on the segmentation mask. Specifically, since the compressed visual data is processed by machine vision, only information relevant to machine analysis is necessary and needs to be retained. Machine analysis-related information includes information generated by the pre-analysis stage (e.g., Figure 5 The segmentation mask output by the pre-analysis stage 510 in the process. For example, Figure 6 A shows an input image that displays multiple objects (e.g., people) corresponding to multiple instances (i.e., segmentation masks) identified in the pre-analysis phase. To directly preprocess the segmentation masks, the segmentation masks of the multiple instances are combined to obtain a merged mask in a union manner. For example, as... Figure 6 As shown in B, segmentation masks corresponding to objects (i.e., people) can be combined to form a merged mask. Only the information within the merged mask is retained for further compression. This will be combined... Figure 7-15 Further description Figure 6 Technical aspects of A-6E. However, as those skilled in the art will understand, while some of these figures may be described in conjunction with each other below, this does not necessarily mean that their corresponding embodiments should be implemented in a combined manner.

[0104] In some embodiments, block-based preprocessing can be used to process the segmentation mask output from the pre-analysis stage. Unlike simply retaining the instance mask region to extract the most essential information, the region surrounding the merged mask is also important for achieving better instance discrimination. Inspired by this, as... Figure 6 As shown in C, block-based preprocessing operations can be used to preserve visual information surrounding the instance. Specifically, a sliding window operation S is performed on the merged mask. n The window size is n×n, where n can be, for example, 128, 192, or 256. As long as the sliding window S... n If there is an intersection between the original visual data and the merge mask, then information about the original visual data can be preserved, such as... Figure 6 As shown in C. Figure 14 A and Figure 14 B schematically illustrates block-based preprocessing, which will be described below.

[0105] In some embodiments, dilation-based preprocessing can be used to process the segmentation mask output from the pre-analysis stage. Dilation is an important image processing method. In this document, it works in conjunction with the preprocessing operation to preserve information about the surroundings of the predicted instance. Specifically, as... Figure 6As shown in D, dilation is performed on the merged mask to expand the predicted instance region, with a convolution kernel size of k×k and t iterations, for example, 7×7 and 3 respectively. Similarly, information from the original visual data in the corresponding regions can be preserved for further compression, such as... Figure 6 As shown in D. Figure 15 A-15D schematically illustrates dilation-based preprocessing, which will be described below.

[0106] In some embodiments, the block-based preprocessing and dilation-based preprocessing can be combined to process multiple segmentation masks output by the preprocessing stage. Block-based preprocessing operations are coarse-grained and may introduce redundant information. However, the irregular boundaries of visual data processed using dilation-based preprocessing operations increase the representational overhead of visual data compression, especially for block-based compression codecs such as HEVC and VVC. To address this issue, such as... Figure 6 As shown in E, dilation-based and block-based preprocessing operations can be used. Specifically, dilation-based preprocessing is performed first, and block-based preprocessing is performed based on the output of the dilation-based preprocessing. Here, the sliding window size n s It can be smaller, such as 64 and 128, as... Figure 6 As shown in E.

[0107] In some embodiments, blurring preprocessing can be used to improve the smoothness and representational compactness of image data. To prevent the merged mask region containing the object from becoming blurred, thereby reducing the performance or accuracy of machine vision tasks (e.g., object recognition or tracking), the output I of the blurring preprocessing... blur This can be expressed as a formula, for example, I blur =G k (M bd -M)I+MI, where I is the original image, M is the merge mask, and M bd It is a preprocessed mask based on blocks and / or dilation, and G k It is a Gaussian filter with a convolution kernel size of k. Thus, in block and / or dilated preprocessed images, the blurring preprocessing is applied only to image regions other than the merge mask (i.e., non-overlapping), thereby improving the compression efficiency without compromising machine analysis performance.

[0108] In some embodiments, to improve compression efficiency, a preprocessing stage can be adaptively performed based on the quantization parameter (QP) and / or the sharpness of the input visual data.

[0109] Specifically, when the compression codec is a conventional codec (e.g., HEVC and VVC), a lower QP implies higher representation overhead and better visual data reconstruction quality. When the QP is high (e.g., greater than 32), the representation overhead is limited, and more coding bits should be allocated to the instance regions to achieve better reconstruction quality. Thus, the sliding window size *n* in block-based preprocessing and the convolutional kernel size in dilation-based preprocessing can be smaller, for example, 64 and 3×3, respectively. Furthermore, in the case of high QP, blurring preprocessing can also be applied to the block output and / or dilation-based preprocessing to further improve compression efficiency. Conversely, when the QP is small (e.g., less than 32), the coding overhead is sufficient, and more original visual data information can be preserved for better machine analysis performance. For example, the window size and convolutional kernel size can be 256 and 9×9, respectively.

[0110] Furthermore, the resolution of the input visual data is also important. For high-resolution visual data (e.g., greater than 1920×1080), the sliding window size for block-based preprocessing and the kernel size for dilation-based preprocessing can be set larger, for example, 256 and 7×7, respectively. Conversely, for low-resolution visual data (e.g., less than 1920×1080), the sliding window size for block-based preprocessing and the convolutional kernel size for dilation-based preprocessing can be set smaller, for example, 128 and 3×3, respectively.

[0111] Consistent with the disclosed embodiments, the adaptive preprocessing may be based on QP, the sharpness of the input image, or a combination thereof.

[0112] Figure 7 This is a flowchart of an exemplary image data compression method 700 according to some embodiments of the present disclosure. For example, method 700 may be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into... Figure 4 In the illustrated apparatus 400, method 700 can be executed by apparatus 400. For example... Figure 7 As shown, method 700 includes the following steps 710-740.

[0113] In step 710, the processor performs instance segmentation on the input image to generate one or more segmentation masks.

[0114] Specifically, the processor can execute an instance segmentation algorithm to divide an input image into multiple parts, for example, sets of pixels representing different parts of the input image. The instance segmentation algorithm can assign labels (i.e., categories) to each group of pixels, such that pixels with the same label share certain common features. The processor can then group the pixels according to their labels and assign each group to a specific object, i.e., a segmentation mask. Based on the labels, the processor can determine the boundaries of each segmentation mask. The labels do not necessarily belong to different categories. For example, Figure 6 The two individuals in B may be labeled differently, but they may belong to the same category of objects—people. In some embodiments of this disclosure, each of the two individuals is an instance and can be detected separately by an instance segmentation algorithm. For example, the processor may perform an instance segmentation algorithm on the input image, such as... Figure 13 As shown in A-13D. As a result, pixels belonging to the instance (two people) can be labeled "Person-A" and "Person-B". Simultaneously, segmentation masks 1310 and 1320 are generated accordingly, as shown... Figure 13 As shown in A and 13B, mask 1310 is generated by grouping the pixels of “person-A”, and mask 1320 is generated by grouping the pixels of “person-B”.

[0115] In some embodiments, after generating the segmentation mask, the processor may also execute a classification algorithm to determine the object category associated with each segmentation mask. For example, the classification algorithm may be implemented using a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN).

[0116] Return to reference Figure 7 In step 720, the processor combines one or more segmentation masks to generate a merge mask. Specifically, the merge mask can form a connected region in the input image. For example, as... Figure 6 As shown in B, the segmentation masks corresponding to multiple people can be merged to form a connected region. Please refer to [reference needed]. Figure 13 C. Here, segmentation mask 1310 and segmentation mask 1320 are combined to form merge mask 1330. Combining different segmentation masks reduces the number of masks, thereby reducing the computational burden of image processing. On the other hand, the combination of different segmentation masks, as well as other morphological techniques described below (e.g., block-based preprocessing or dilation-based preprocessing), reduces the risk of losing partial instances due to missed segmentation at step 710, where the lost instances may also be regions of interest (ROIs) of the input image.

[0117] In some embodiments, the multiple segmentation masks generated in step 710 may be isolated from each other. In this case, the merged mask combined with the multiple isolated masks consists of multiple non-overlapping portions. In other embodiments, the multiple segmentation masks generated in step 710 may be adjacent to each other. In this case, the merged mask consists of multiple connected regions. Figure 13 C and Figure 13 As shown in D, the merge mask 1330 and the corresponding region 1340 can be retained in the image for subsequent processing, while the background region 1350 in the input image can be discarded and does not participate in subsequent processing.

[0118] Return to reference Figure 7 In step 730, the processor extracts a region from the input image that includes the merged mask.

[0119] In some embodiments, the extracted region includes only a merge mask. For example, such as... Figure 6 As shown in B, only the merge mask is extracted from the input image.

[0120] In some embodiments, the extraction region includes not only the merge mask but also the extended region surrounding the merge mask. For example, as... Figure 6 As shown in c-6e, block-based preprocessing and dilation-based preprocessing can be used to create an extended region around the merge mask.

[0121] As mentioned above, the merge mask may consist of several non-adjacent parts, which can increase complexity or reduce the efficiency of image encoding to some extent. Morphological processing combined with the merge mask incorporates parts that may be regions of interest (ROIs), and in some cases, it bridges non-overlapping parts into a unified whole.

[0122] When using block-based preprocessing, the processor applies a sliding window to the image region outside the merge mask to identify one or more blocks that do not overlap with the merge mask. The processor then extracts a region from the input image that does not include one or more of these blocks. Figure 8 Block-based preprocessing is shown, which will be described in detail below.

[0123] When using dilation-based preprocessing, the processor applies convolutional kernels to a merge mask to determine a first region extending from a boundary of the merge mask. The processor then extracts the first region and the merge mask from the input image. Figure 9 The dilation-based preprocessing is shown, which will be described in detail below.

[0124] In some embodiments, the block-based preprocessing and the dilation-based preprocessing can be used in combination. First, the processor dilates the merge mask by applying a convolutional kernel to it to determine a first region extending from a boundary of the merge mask. Next, the processor identifies one or more blocks in the input image that do not overlap with the first region or the merge mask. Finally, the processor extracts a region from the input image that does not include the one or more blocks. Figure 10 Block-based and dilation-based preprocessing are shown, which will be described in detail below.

[0125] In some embodiments, blurring preprocessing can be applied to the region including the merged mask. Specifically, in block and / or dilated preprocessed images, blurring preprocessing is applied to the extended region around the merged mask to remove details or simple content in the extended region. Since the extended region does not contain objects, blurring it does not affect the accuracy of the machine vision task. Figure 12 The blurring preprocessing applied to the extended region is shown, which will be described in detail below.

[0126] In some embodiments, the block-based preprocessing, the dilation-based preprocessing, and / or the blurring preprocessing can be adaptively performed based on the quantization parameter (QP) used to compress the input image or the sharpness of the input image. For example, in the case of high QP (e.g., when QP is greater than a predetermined threshold), the sliding window size in the block-based preprocessing and the convolution kernel size in the dilation-based preprocessing can be smaller, and the blurring preprocessing can also be used to blur image regions other than the merge mask. Conversely, in the case of low QP (e.g., when QP is less than or equal to a predetermined threshold), the sliding window size in the block-based preprocessing and the convolution kernel size in the dilation-based preprocessing can be larger, and the blurring preprocessing can be skipped. Figure 12 The QP-based fuzzing preprocessing is illustrated, which will be described in detail below.

[0127] Return to reference Figure 7 In step 740, the processor compresses the image data representing the extracted region. Specifically, the image data can be fed into a codec for compression using the specific QP. The codec can perform encoding and / or decoding according to image / video standards such as High-Level Video Coding (AVC), High-Efficiency Video Coding (HEVC), Multi-Functional Video Coding (VVC), AOMedia Video 1 (AV1), Joint Picture Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc.

[0128] Figure 8Based on some embodiments of this disclosure, Figure 7 The flowchart illustrates an exemplary image data compression method 800 of method 700. For example, method 800 can be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into... Figure 4 In the illustrated apparatus 400, method 800 is made executable by apparatus 400. Method 800 applies block-based preprocessing. For example... Figure 8 As shown, step 730 is further broken down into two sub-steps, 810 and 820. From Figure 7 The other steps of inheritance will not be described further.

[0129] refer to Figure 8 In step 810, the processor identifies one or more blocks in the input image that do not overlap with the merge mask. These one or more blocks are not necessarily coded blocks used to compute the motion vector. In some examples, the one or more blocks may be specifically designed to separate instances and background in the input image. As a non-limiting example, the processor may apply a sliding window to the input image to identify one or more blocks that do not overlap with the merge mask.

[0130] like Figure 14 As shown in Figure A, the window can slide from the top left to the bottom right of the input image. Depending on the required preprocessing granularity, the window size can be, for example, 128×128, 192×192, or 256×256. The window slides pixel-level through A11, A12, ..., A1n, A21, A22, ..., A2n, ... In other words, the sliding step is equal to the width of the window. In the following text, for simplicity, the sliding window will be referred to in some places as block A11, block A12, ..., block A1n, block A21, block A22, ..., block A2n, ...

[0131] When any block fully or partially overlaps with merge mask 1330, it will be marked as an overlapping block and indicated by being filled with solid black, such as... Figure 14 As shown in b. For example. Figure 14 Blocks A12, A13, and A14 in A partially overlap with merge mask 1330, while block A54 completely overlaps with merge mask 1330. Therefore, these blocks are overlapping blocks and... Figure 14 Fill B with pure black. Note that... Figure 14In B, the merge mask 1330 is shown on top of the overlapping block 1410. When the overlapping block 1410 is determined, one or more blocks 1420 in the input image other than the overlapping block (i.e., blocks that do not overlap with the merge mask) can also be identified.

[0132] In some embodiments, the size of the sliding window can be determined based on the quantization parameter (QP) used to compress the input image and / or the sharpness of the input image. When the QP used to compress the input image is high or the sharpness of the input image is low, the representation overhead is limited, and more coding bits should be allocated to the instance region to obtain better reconstruction quality. Thus, the size of the sliding window can be smaller, for example, 64×64. Conversely, when the QP used to compress the input image is low or the sharpness of the input image is high, the size of the sliding window can be larger, for example, 256×256.

[0133] Return to reference Figure 8 In step 820, the processor extracts a region from the input image that does not include one or more blocks 1420. In other words, the processor can extract a region that includes at least the overlapping blocks 1410. In a specific example, the region may consist of overlapping blocks 1410. As those skilled in the art will understand, the region surrounding the merge mask 1330 is also important for better differentiation of instances. Method 800 extends from the merge mask to obtain a region consisting at least of the overlapping blocks, which covers instances with a relatively high probability, thereby reducing missed segmentation in the merge mask.

[0134] Figure 9 Based on some embodiments of this disclosure Figure 7 The flowchart illustrates an exemplary image data compression method 900 of method 700. For example, method 900 can be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into... Figure 4 In the illustrated apparatus 400, method 900 is made executable by apparatus 400. Method 900 applies an expansion-based preprocessing. Figure 9 As shown, step 730 is further broken down into two sub-steps, 910 and 920. From Figure 7 The other steps of inheritance will not be described further.

[0135] Reference Figure 9In step 910, the processor dilates the merge mask to determine a first region extending from a boundary of the merge mask in the input image. As a non-limiting example, the processor may apply a convolutional kernel to the merge mask to determine the first region extending from a boundary of the merge mask.

[0136] like Figure 15 As shown in A-15D, in some embodiments, convolutional kernels having the shapes of kernels 1530, 1540, or 1550 can be used to dilate the merged mask to determine a first region 1510 (not shown for simplicity) extending from the boundary of the merged mask. In a non-limiting example, a 7×7 kernel 153 can be applied for dilation. For example, the merged mask can be dilated using kernel 153 to expand the predicted instance region over three iterations. In each iteration, kernel 153 is applied to the input image to dilate it at the pixel level. Because the kernel size is smaller than the sliding window, the dilation-based preprocessing may have better morphological performance than the block-based preprocessing, but its computational burden and complexity may be higher.

[0137] In some embodiments, the convolutional kernel size can be determined based on the quantization parameter (QP) used to compress the input image and / or the sharpness of the input image. When the QP used to compress the input image is high or the sharpness of the input image is low, the representation overhead is limited, and more coding bits should be allocated to the instance region to obtain better reconstruction quality. Therefore, the convolutional kernel size can be smaller, for example, 3×3. Conversely, when the QP of the input image to be compressed is low or the sharpness of the input image is high, the convolutional kernel size can be larger, for example, 7×7.

[0138] Return to reference Figure 9 In step 920, the processor extracts a first region and an optional dilation mask from the input image. Figure 15 As can be seen from A, the first region 1510 is extracted from the input image through dilation, while the remaining region 1520 in the input image, excluding the first region 1510, can be ignored in subsequent processing. As those skilled in the art will understand, the region surrounding the merge mask is also important for achieving better instance differentiation. Method 900 extends from the merge mask to obtain a first region covering relatively more likely instances, thereby reducing missed segmentation in the merge mask.

[0139] Figure 10 Based on some embodiments of this disclosure Figure 7The flowchart illustrates an exemplary image data compression method 1000, method 700. For example, method 1000 can be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 can be integrated into... Figure 4 In the illustrated apparatus 400, method 1000 can be executed by apparatus 400. Method 1000 simultaneously applies both dilation-based preprocessing and block-based preprocessing. For example... Figure 10 As shown, step 730 is further broken down into three sub-steps 1010, 1020, and 1030. From Figure 7 The other steps of inheritance will not be described further.

[0140] Reference Figure 10 In step 1010, the processor expands the merge mask to determine a first region extending from a certain boundary of the merge mask. The expansion step in step 1010 can follow... Figure 9 The processes described in the corresponding embodiments are omitted here for the sake of brevity.

[0141] Next, in step 1020, the processor identifies one or more blocks that do not overlap with the first region in the input image or the merge mask. Similarly... Figure 8 The described method allows the processor to apply a sliding window to the input image to identify one or more blocks that do not overlap with the first region or the merge mask.

[0142] For example, a window can be slid from the top left corner to the bottom right corner of the input image for pixel-level traversal. The window size can be, for example, 128×128, 192×192, or 256×256, and the sliding step is the width of the window. Any block that fully or partially overlaps with the first region or the merge mask is marked as an overlapping block. When overlapping blocks are identified, one or more other blocks in the input image besides overlapping blocks (i.e., blocks that do not overlap with the first region or the merge mask) can also be identified.

[0143] Still referencing Figure 10 In step 1030, the processor extracts a region from the input image that does not include one or more blocks. In other words, the processor extracts a region that includes at least a plurality of the overlapping blocks. In a specific example, the region may consist of multiple overlapping blocks. As those skilled in the art will understand, the region surrounding the merge mask is also important for achieving better instance differentiation.

[0144] Method 1000 extends from the merge mask to obtain a region consisting of at least overlapping blocks that cover instances with relatively high probability, thereby reducing missed segments in the merge mask. Compared to methods 800 and 900, method 1000 simultaneously applies dilation-based and block-based preprocessing, wherein the block-based preprocessing is performed at a relatively large scale with respect to processing operators (i.e., the sliding window or the convolution kernel), while the dilation-based preprocessing is performed at a relatively small scale. Therefore, method 1000 has better morphological performance than methods 800 or 900 that only employ block-based preprocessing or only employ dilation-based preprocessing. Here, the sliding window size can be smaller than that of method 800, for example, 64×64 or 128×128.

[0145] Figure 11 Based on some embodiments of this disclosure Figure 7 The flowchart illustrates an exemplary image data compression method 1100 of method 700. For example, method 1100 can be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into... Figure 4 In the illustrated apparatus 400, method 1100 is made executable by apparatus 400. Method 1100 applies fuzzy preprocessing. For example... Figure 11 As shown, step 730 is further broken down into two sub-steps, 1110 and 1120. From Figure 7 The other steps of inheritance will not be described further.

[0146] Reference Figure 11 In step 1110, the processor determines a second region in the extraction region that does not overlap with the merge mask. For example... Figure 14 As shown in Figure A, the window can slide from the top left to the bottom right of the input image. When any block fully or partially overlaps with the merge mask 1330, it will be marked as an overlapping block and characterized by being filled with solid black, as shown in Figure A. Figure 14 As shown in B. It should be noted that the merge mask 1330 is shown above the overlapping block 1410. When the overlapping block 1410 is determined, one or more blocks 1420 in the input image, in addition to the overlapping block, can also be identified. These one or more blocks 1420 can be used to form a second region. For example, the processor can determine a second region composed of some or all of the one or more blocks 1420.

[0147] Return to reference Figure 11In step 1120, the processor blurs the second region. In some embodiments, the background portion of the input image is not simply discarded. The second region can be preprocessed with blurring to improve the smoothness and compactness of the image data.

[0148] In further examples, blurring preprocessing can be applied in conjunction with block and / or dilation preprocessing. To prevent blurring of the merged mask region containing the object from affecting the performance or accuracy of machine vision tasks (such as object recognition or tracking), the output I of the blurring preprocessing... blur This can be expressed, for example, as I blur =G k (M bd -M)I+MI, where I is the original image, M is the merge mask, and M bd It is the mask after preprocessing based on blocks and / or expansion, and G k It is a Gaussian filter with a convolution kernel size of k. Thus, in block and / or dilated preprocessed images, the blurring preprocessing is applied only to image regions other than the merge mask (i.e., non-overlapping), thereby improving compression efficiency without compromising machine analysis performance.

[0149] Figure 12 Based on some embodiments of this disclosure Figure 7 The flowchart illustrates an exemplary image data compression method 1200 of method 700. For example, method 1200 can be executed by one or more processors, such as image / video preprocessor 122. Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into... Figure 4 In the illustrated apparatus 400, method 1200 can be executed by apparatus 400. Method 1200 determines whether to perform fuzzy preprocessing based on quantization parameters (QP). Figure 12 As shown, step 730 is further broken down into two sub-steps, 1210 and 1220. From Figure 7 The other steps of inheritance will not be described further.

[0150] Reference Figure 12In step 1210, the processor determines whether to blur at least a portion of the extracted region based on a quantization parameter (QP) used to compress the input image. To further compress the input image, a portion of the extracted region can be blurred in some way. For example, if the input image is to be compressed at low quality (higher QP), at least a portion of the extracted region may be blurred. However, if the input image is to be quantized at more stages (lower QP), the extracted region may not need to be blurred. Applying blurring preprocessing in the case of more quantization stages can significantly degrade the quality of the encoded image, which may be unsuitable for machine vision.

[0151] Still refer to Figure 12 In step 1210, the processor blurs a portion of the extracted region in response to the QP being greater than a predetermined threshold. Furthermore, the blurring preprocessing can also be applied to the output of the block-based and / or expanded preprocessing to further improve compression efficiency. However, if the QP is less than the predetermined threshold, the processor may not blur any portion of the extracted region. In embodiments of this disclosure, the segmentation mask or the merging mask can be applied to the input image to extract instances from the input image. As those skilled in the art will understand, for example, when the merging mask is applied to the input image, a portion of the image (where instances exist) will be covered by the merging mask. Any region in the image that includes the area covered by the merging mask can be referred to as the region including the merging mask, even without further explanation herein, and is inherently present in this disclosure. Or, in other words, the expression "extract a region including the merging mask" can also be restated as "extract a region in the input image covered by the applied merging mask." Both expressions contribute equally to the invention.

[0152] Figure 16 This is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. For example, method 1600 may be executed by one or more processors, such as image / video decoder 144. Figure 1 In some embodiments, the image / video decoder 144 can be integrated into... Figure 4 In the illustrated apparatus 400, method 1600 can be executed by apparatus 400. Method 1600 can decode bitstreams utilizing any of the above-described encoding schemes, for example, Figure 3A and Figure 3B The decoding process is shown below. Figure 16 As shown, method 1600 includes the following steps.

[0153] In step 1610, the processor receives a bit stream comprising compressed image data with region information. The compressed image data includes encoded information characterizing one or more regions extracted from an image. The process of preprocessing and encoding the extracted regions has been described in the above embodiments. The region information indicates one or more locations of the extracted regions within the image.

[0154] In step 1620, the processor reconstructs the image by decoding the compressed image data based on the region information. The decoding bitstream processing procedure is described above. Figure 3A and Figure 3B The description is included here. Since areas outside the extracted region may be blank and no encoded bits are assigned to them, the processor can skip these regions whenever the compressed image is decoded to obtain an image. In this respect, there is no reference relationship between pixels in the extracted region (indicated by the region information) and pixels in areas outside the extracted region (which can be extrapolated from the region information due to the fixed image size). The decoder can then focus on decoding the compressed image data within the extracted region. Therefore, the decoding load on the decoder is reduced compared to decoding the entire image.

[0155] like Figure 17 As shown, step 1620, which involves decoding compressed image data based on the region information to reconstruct the image, can be further decomposed into two steps 1710 and 1720 and an optional step 1730.

[0156] In step 1710, the processor may generate a first region of the image based on the decoded image data. The first region is the extracted region represented by the decoded image data. As previously described, decoded pixel values ​​from the decoded image data can be used to fill the first region.

[0157] In step 1720, the processor can generate a second region of the image by filling pixels. Here, the second region is at least a portion of the area outside the first region. As an example, the processor can combine the first and second regions to form the entire image. As another example, the processor can ignore the second region, so that the second region can be transparent when forming the image. As yet another example, the processor can fill the second region according to a preset scheme. For example, the processor can fill the second region of the image with "black" or "white" pixels, so that the second region can exist as the background of the image. Here, black pixels are pixels with RGB values ​​of (0, 0, 0), and white pixels are (255, 255, 255). In yet another example, the processor can fill the second region with an existing background (e.g., a landscape). For example, the processor can select a landscape image and then expand or shrink it to the size of the image to be decoded. Then, the processor performs a cutout process on the landscape image based on the region information. The remaining part of the cutout landscape image can serve as the background to form the second region of the image. Finally, the processor can merge the first and second regions to form a complete image.

[0158] As an optional step, in step 1730, the processor can recover the location of an instance in the image based on the region information. In some examples, the decoded image can be used for other processing, such as object recognition. The processor can also recover the location of the instance. As a premise, the first region (i.e., the extracted region) can cover one or more instances. By recovering the location of the first region, the locations of the one or more instances are known. Therefore, in subsequent processing, the focus can be placed on the recovered location, such as object recognition.

[0159] It should be noted that, Figure 16 and Figure 17 The described region information is not necessary for decoding the compressed video information. Consistent with the disclosed embodiments, any decision-making scheme can be used to decode video information compressed based on the method described above. For example, the compressed video information can be decoded by a conventional decoder without any special modifications.

[0160] It should be understood that one embodiment of this disclosure may be combined with another embodiment or a plurality of other embodiments.

[0161] In some embodiments, a non-transitory computer-readable storage medium is also provided for storing one or more bit streams processed according to the methods described above. For example, it can be based on methods 700-1200 ( Figure 7-12This generates one or more bit streams. Specifically, according to some embodiments, the one or more bit streams include data generated by an encoder (e.g., Figure 1 The syntax segment encoded by the image / video encoder 122 in the image / video encoder. For example, the one or more bitstreams may be encoded according to... Figure 2A and 2B The bitstream 228 is encoded using the encoding process described herein. Prior to encoding, the input image of the encoder can be processed by a preprocessor (e.g., Figure 1 The image / video preprocessor 122 in the present disclosure performs pre-analysis and preprocessing according to the solution provided in this disclosure, for example, with... Figure 7-12 Describe the relevant processing procedures. Consistent with the disclosed embodiments, one or more bitstreams can also be decoded by a decoder (e.g., Figure 1 The image / video decoder 144 in the image / video decoder decodes the data. For example, one or more bitstreams can be decoded according to... Figure 3A and 3B The decoding process in [the document] decodes the bitstream 228. Furthermore, the decoded information can be used to generate a reconstructed image according to the solution provided in this disclosure, for example, with [other methods]. Figure 16 and Figure 17 Describe the relevant processing procedures.

[0162] In some embodiments, a non-transitory computer-readable storage medium including an instruction set is also provided, and the instruction set can be executed by a device for performing the methods described above (such as the disclosed encoders and decoders). Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs or any other flash memory, NVRAM, caches, registers, any other memory chips or cassette tapes and their networking versions. The device may include one or more processors (CPUs), an input / output interface, a network interface, and / or a memory.

[0163] In some embodiments, a computer program product is provided, including: a computer program instruction set, and the computer program instruction set enabling a computer to perform the steps of the method described in any embodiment of the present disclosure.

[0164] In some embodiments, a computer program is provided, which enables a computer to perform the steps of the methods described in the embodiments of this disclosure.

[0165] The embodiments may be further described using the following terms:

[0166] 1. An image data processing method, comprising:

[0167] Receive a video sequence;

[0168] Encoding one or more input images associated with the video sequence; and

[0169] Generate a one-bit stream.

[0170] The encoding includes:

[0171] Perform instance segmentation on the input image to generate one or more segmentation masks;

[0172] Combine the one or more segmentation masks to generate a merge mask;

[0173] Extract a region containing the merge mask from the input image; and

[0174] The image data representing the extracted region is compressed.

[0175] 2. The method according to Clause 1, wherein the extracted region includes one or more pixels adjacent to the merge mask.

[0176] 3. The method according to Clause 1, wherein extracting the region including the merged mask from the input image includes:

[0177] In the input image, identify one or more blocks that do not overlap with the merge mask; and

[0178] Extract a region from the input image that does not include the one or more blocks.

[0179] 4. The method according to Clause 3, wherein a sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following:

[0180] A quantization parameter (QP) used to compress the input image, or

[0181] The clarity of the input image.

[0182] 5. The method according to Clause 1, wherein extracting the region including the merged mask from the input image includes:

[0183] Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and

[0184] Extract the first region and the dilation mask from the input image.

[0185] 6. The method according to Clause 5, wherein the merge mask is expanded by applying a convolutional kernel to the merge mask, and the size of the convolutional kernel is determined based on at least one of the following:

[0186] A quantization parameter (QP) used to compress the input image, or

[0187] The clarity of the input image.

[0188] 7. The method according to Clause 1, wherein extracting the region including the merged mask from the input image includes:

[0189] Inflate the merge mask to determine a first region extending from a boundary of the merge mask;

[0190] In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and

[0191] Extract a region from the input image that does not include the one or more blocks.

[0192] 8. The method according to Clause 1, wherein extracting the region including the merged mask from the input image further comprises:

[0193] In the extracted region, a second region that does not overlap with the merge mask is determined; and

[0194] Blur the second region.

[0195] 9. The method according to Clause 1, wherein extracting the region including the merged mask from the input image further comprises:

[0196] Based on a quantization parameter QP used to compress the input image, determine whether to blur at least a portion of the extracted region; and

[0197] In response to the QP being greater than a predetermined threshold, the portion of the extracted region is blurred.

[0198] 10. The method according to Clause 9, wherein the portion of the extracted region does not overlap with the merge mask.

[0199] 11. A non-transitory computer-readable storage medium for storing a bit stream, and a method for generating the bit stream comprising:

[0200] Perform instance segmentation on an input image to generate one or more segmentation masks;

[0201] Combine one or more of the segmentation masks to generate a merge mask;

[0202] Extract a region containing the merge mask from the input image; and

[0203] Image data representing the extracted region is compressed to generate the bitstream.

[0204] 12. The medium according to Clause 11, wherein the extraction region includes one or more pixels adjacent to the merge mask.

[0205] 13. The medium according to Clause 11, wherein extracting the region including the merged mask from the input image includes:

[0206] In the input image, identify one or more blocks that do not overlap with the merge mask; and

[0207] Extract a region from the input image that does not include the one or more blocks.

[0208] 14. The medium according to Clause 13, wherein a sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following:

[0209] A quantization parameter QP used to compress the input image, or

[0210] The clarity of the input image.

[0211] 15. The medium according to Clause 11, wherein extracting the region including the merged mask from the input image includes:

[0212] Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and

[0213] Extract the first region and the merge mask from the input image.

[0214] 16. The medium according to Clause 15, wherein the merge mask is expanded by applying a convolutional kernel to the merge mask, and the size of the convolutional kernel is determined based on at least one of the following:

[0215] A quantization parameter QP used to compress the input image, or

[0216] The clarity of the input image.

[0217] 17. The medium according to Clause 11, wherein extracting the region including the merged mask from the input image includes:

[0218] Inflate the merge mask to determine a first region extending from a boundary of the merge mask;

[0219] In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and

[0220] Extract a region from the input image that does not include the one or more blocks.

[0221] 18. The medium according to Clause 11, wherein extracting the region including the merged mask from the input image further comprises:

[0222] In the extracted region, a second region that does not overlap with the merge mask is determined; and

[0223] Blur the second region.

[0224] 19. The medium according to Clause 11, wherein extracting the region including the merged mask from the input image further comprises:

[0225] Based on a quantization parameter (QP) used to compress the input image, determine whether to blur at least a portion of the extracted region; and

[0226] In response to the QP being greater than a predetermined threshold, the portion of the extracted region is blurred.

[0227] 20. The medium according to Clause 19, wherein a portion of the extracted region does not overlap with the merge mask.

[0228] 21. An image data processing apparatus, comprising:

[0229] A memory that stores an instruction set; and

[0230] One or more processors are configured to execute the instruction set to cause the device to perform operations including:

[0231] Perform instance segmentation on an input image to generate one or more segmentation masks;

[0232] Combine the one or more segmentation masks to generate a merge mask;

[0233] Extract a region containing the merge mask from the input image; and

[0234] Compress the image data representing the extracted region.

[0235] 22. The apparatus according to clause 21, wherein the extraction region includes one or more pixels adjacent to the merge mask.

[0236] 23. The apparatus according to clause 21, wherein extracting the region including the merged mask from the input image comprises:

[0237] In the input image, identify one or more blocks that do not overlap with the merge mask; and

[0238] Extract a region from the input image that does not include the one or more blocks.

[0239] 24. The apparatus according to clause 23, wherein a sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following:

[0240] A quantization parameter (QP) used to compress the input image, or

[0241] The clarity of the input image.

[0242] 25. The apparatus according to clause 21, wherein extracting the region including the merged mask from the input image comprises:

[0243] Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and

[0244] Extract the first region and the dilation mask from the input image.

[0245] 26. The apparatus according to Clause 25, wherein the merge mask is expanded by applying a convolutional kernel to the merge mask, and the size of the convolutional kernel is determined based on at least one of the following:

[0246] A quantization parameter (QP) used to compress the input image, or

[0247] The clarity of the input image.

[0248] 27. The apparatus according to clause 21, wherein extracting the region including the merged mask from the input image comprises:

[0249] Inflate the merge mask to determine a first region extending from a boundary of the merge mask;

[0250] In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and

[0251] Extract a region from the input image that does not include the one or more blocks.

[0252] 28. The apparatus according to clause 21, wherein extracting the region including the merged mask from the input image further comprises:

[0253] In the extracted region, a second region that does not overlap with the merge mask is determined; and

[0254] Blur the second region.

[0255] 29. The apparatus according to clause 21, wherein extracting the region including the merged mask from the input image further comprises:

[0256] Based on a quantization parameter QP used to compress the input image, determine whether to blur at least a portion of the extracted region; and

[0257] In response to the QP being greater than a predetermined threshold, the portion of the extracted region is blurred.

[0258] 30. The apparatus according to Clause 29, wherein a portion of the extracted region does not overlap with the merge mask.

[0259] 31. A non-transitory computer-readable storage medium storing an instruction set, wherein when the instruction set is executed by a computer, a method for causing the computer to perform [operations] includes:

[0260] Perform instance segmentation on an input image to generate one or more segmentation masks;

[0261] Combine the one or more segmentation masks to generate a merge mask;

[0262] Extract a region containing the merge mask from the input image; and

[0263] The image data representing the extracted region is compressed to generate a bitstream.

[0264] 32. The medium according to Clause 31, wherein the extraction region includes one or more pixels adjacent to the merge mask.

[0265] 33. The medium according to clause 31, wherein extracting the region including the merged mask from the input image includes:

[0266] In the input image, identify one or more blocks that do not overlap with the merge mask; and

[0267] Extract a region from the input image that does not include the one or more blocks.

[0268] 34. The medium according to Article 33, wherein a sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following:

[0269] A quantization parameter QP used to compress the input image, or

[0270] The clarity of the input image.

[0271] 35. The medium according to clause 31, wherein extracting the region including the merged mask from the input image includes:

[0272] Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and

[0273] Extract the first region and the merge mask from the input image.

[0274] 36. The medium according to Clause 35, wherein the merge mask is expanded by applying a convolutional kernel to the merge mask, and the size of the convolutional kernel is determined based on at least one of the following:

[0275] A quantization parameter QP used to compress the input image, or

[0276] The clarity of the input image.

[0277] 37. The medium according to Article 31, wherein extracting the region including the merged mask from the input image includes:

[0278] Inflate the merge mask to determine a first region extending from a boundary of the merge mask;

[0279] In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and

[0280] Extract a region from the input image that does not include the one or more blocks.

[0281] 38. The medium according to clause 31, wherein extracting the region including the merged mask from the input image further includes:

[0282] In the extracted region, a second region that does not overlap with the merge mask is determined; and

[0283] Blur the second region.

[0284] 39. The medium according to clause 31, wherein extracting the region including the merged mask from the input image further comprises:

[0285] Based on a quantization parameter QP used to compress the input image, determine whether to blur at least a portion of the extracted region; and

[0286] In response to the QP being greater than a predetermined threshold, the portion of the extracted region is blurred.

[0287] 40. The medium according to Clause 39, wherein a portion of the extracted region does not overlap with the merge mask.

[0288] 41. An image data processing method, comprising:

[0289] Receive a one-bit stream; and

[0290] The encoding information of the bitstream is used to decode one or more images.

[0291] The decoding includes:

[0292] Receive a bit stream, the bit stream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in an image; and

[0293] The image is reconstructed by decoding the compressed image data based on the region information.

[0294] 42. The method according to clause 41, wherein decoding the compressed image data based on the region information to reconstruct the image includes:

[0295] Generate a first region of the image based on the decoded image data; and

[0296] A second region of the image is generated by filling pixels.

[0297] 43. The method according to clause 41, wherein decoding the compressed image data based on the region information to reconstruct the image further includes:

[0298] The location of an instance in the image is recovered based on the region information.

[0299] 44. A non-transitory computer-readable storage medium storing a bit stream, the bit stream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in an image, and the bit stream is processed by a method comprising:

[0300] The image is reconstructed by decoding the compressed image data based on the region information.

[0301] 45. The medium according to clause 44, wherein decoding the compressed image data based on the region information to reconstruct the image includes:

[0302] A first region of the image is generated based on the decoded image data; and

[0303] A second region of the image is generated by filling pixels.

[0304] 46. ​​The medium according to clause 44, wherein decoding the compressed image data based on the region information to reconstruct the image further includes:

[0305] The location of an instance in the image is recovered based on the region information.

[0306] 47. An image data processing apparatus, comprising:

[0307] A memory that stores an instruction set; and

[0308] One or more processors are configured to execute the instruction set to cause the device to perform operations including:

[0309] Receive a bit stream, the bit stream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in an image; and

[0310] The image is reconstructed by decoding the compressed image data based on the region information.

[0311] 48. The apparatus according to clause 47, wherein decoding the compressed image data based on the region information to reconstruct the image comprises:

[0312] A first region of the image is generated based on the decoded image data; and

[0313] A second region of the image is generated by filling pixels.

[0314] 49. The apparatus according to claim 47, wherein decoding the compressed image data based on the region information to reconstruct the image further comprises:

[0315] The location of an instance in the image is recovered based on the region information.

[0316] 50. A non-transitory computer-readable storage medium storing an instruction set, wherein when the instruction set is executed by a computer, a method for causing the computer to perform [the following] includes:

[0317] Receive a bit stream, the bit stream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in an image; and

[0318] The image is reconstructed by decoding the compressed image data based on the region information.

[0319] 51. The medium according to clause 50, wherein decoding the compressed image data based on the region information to reconstruct the image includes:

[0320] A first region of the image is generated based on the decoded image data; and

[0321] A second region of the image is generated by filling pixels.

[0322] 52. The medium according to clause 50, wherein decoding the compressed image data based on the region information to reconstruct the image further includes:

[0323] The location of an instance in the image is recovered based on the region information.

[0324] It should be noted that relational terms such as “first” and “second” in this document are used only to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, the words “contains,” “has,” “includes,” and “includes,” as well as other similar forms, are intended to have the same meaning and are open-ended, as one or more items following any of these words do not imply an exhaustive list of such one or more items, or that the list is limited to one or more listed items.

[0325] As used herein, unless otherwise specifically stated, the term "or" covers all possible combinations unless impractical. For example, if a database is specified to include A or B, then unless otherwise explicitly stated or impractical, the database may include A, or B, or A and B. As a second example, if a database is specified to include A, B, or C, then unless otherwise specifically stated or impractical, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0326] It should be understood that the above embodiments can be implemented by hardware, software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-described computer-readable medium. When executed by a processor, the software can perform the disclosed methods. The computer units and other functional units described in this disclosure can be implemented by hardware, software, or a combination of hardware and software. Those skilled in the art should also understand that the multiple modules / units described above can be combined into one module / unit, and each of the above modules / units can be further divided into multiple sub-modules / sub-units.

[0327] In the foregoing specification, numerous specific details have been described with reference to embodiments that may vary depending on the implementation. Certain adaptations and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art upon consideration of the specification and practice of this disclosure herein. The specification and embodiments are intended to be considered exemplary only, and the true scope and nature of this disclosure are indicated by the appended claims. The sequence of steps shown in the figures is for illustrative purposes only and is not intended to be limited to any particular order of steps. Therefore, those skilled in the art will understand that these steps may be performed in different orders while achieving the same method.

[0328] Exemplary embodiments have been disclosed in the accompanying drawings and description. However, many variations and modifications can be made to these embodiments. Accordingly, although specific terms have been used, they are used only in a general and descriptive sense and not for limiting purposes.

Claims

1. An image data processing method, comprising: Receive a video sequence; Encode one or more input images associated with the video sequence; as well as Generate a one-bit stream. The encoding includes: Perform instance segmentation on an input image to generate one or more segmentation masks; Combine the one or more segmentation masks to generate a merge mask; Extract a region including the merged mask from the input image to generate the extracted region; Within the extracted region, a second region that does not overlap with the merge mask is determined; The second region is blurred. And the image data of the extracted region after compression and blurring.

2. The method according to claim 1, wherein, The extracted region includes one or more pixels adjacent to the merge mask.

3. The method according to claim 1 or 2, wherein, Extracting the region including the merged mask from the input image includes: In the input image, identify one or more blocks that do not overlap with the merge mask; and Extract a region from the input image that does not include the one or more blocks.

4. The method according to claim 3, wherein, A sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following: A quantization parameter QP used to compress the input image, or The clarity of the input image.

5. The method according to claim 1 or 2, wherein, Extracting the region including the merged mask from the input image includes: Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and Extract the first region and the merge mask from the input image.

6. The method according to claim 5, wherein, The merge mask is expanded by applying a convolutional kernel, the size of which is determined based on at least one of the following: A quantization parameter QP used to compress the input image, or The clarity of the input image.

7. The method according to claim 1 or 2, wherein, Extracting the region including the merged mask from the input image includes: Inflate the merge mask to determine a first region extending from a boundary of the merge mask; In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and Extract a region from the input image that does not include the one or more blocks.

8. The method according to claim 1 or 2, wherein, Also includes: Determine whether to blur at least one portion of the extracted region based on a quantization parameter QP used to compress the input image; as well as In response to the QP being greater than a predetermined threshold, the portion of the extracted region is blurred.

9. The method according to claim 8, wherein, The portion of the extracted region does not overlap with the merge mask.

10. The method according to claim 1, wherein, The instance segmentation is performed using a convolutional neural network (CNN).

11. A non-transitory computer-readable storage medium having an instruction set and a bit stream stored thereon, the instruction set being executed by one or more processors in a method to generate the bit stream, the method comprising: Perform instance segmentation on an input image to generate one or more segmentation masks; Combine one or more of the segmentation masks to generate a merge mask; Extract a region containing the merged mask from the input image to generate the extracted region; Within the extracted region, a second region that does not overlap with the merge mask is determined; as well as Blur the second region; as well as The image data of the extracted region after blurring is compressed to generate the bitstream.

12. The medium according to claim 11, wherein, The extracted region includes one or more pixels adjacent to the merge mask.

13. The medium according to claim 11 or 12, wherein, Extracting the region including the merged mask from the input image includes: In the input image, identify one or more blocks that do not overlap with the merge mask; and Extract a region from the input image that does not include the one or more blocks.

14. The medium according to claim 13, wherein, A sliding window is used to identify the one or more blocks, and the size of the sliding window is determined based on at least one of the following: A quantization parameter (QP) used to compress the input image, or The clarity of the input image.

15. The medium according to claim 11 or 12, wherein, Extracting the region including the merged mask from the input image includes: Inflate the merge mask to determine a first region extending from a boundary of the merge mask; and Extract the first region and the merge mask from the input image.

16. The medium according to claim 15, wherein, The merge mask is expanded by applying a convolutional kernel to it, and the size of the convolutional kernel is determined based on at least one of the following: A quantization parameter (QP) used to compress the input image, or The clarity of the input image.

17. The medium according to claim 11 or 12, wherein, Extracting the region including the merged mask from the input image includes: Inflate the merge mask to determine a first region extending from a boundary of the merge mask; In the input image, identify one or more blocks that do not overlap with the first region or the merge mask; and Extract a region from the input image that does not include the one or more blocks.

18. An image data processing apparatus, comprising: A memory that stores instruction sets; as well as One or more processors are configured to execute the instruction set to cause the image data processing apparatus to perform operations including: Perform instance segmentation on an input image to generate one or more segmentation masks; Combine the one or more segmentation masks to generate a merge mask; Extract a region containing the merged mask from the input image to generate the extracted region; In the extracted region, a second region that does not overlap with the merge mask is determined; and Blur the second region; and The image data of the extracted region is compressed and represented after blurring.

19. A computer program product comprising: Computer program instructions that cause a computer to perform the method of any one of claims 1 to 10.

Images