Pre-analysis based image compression method

Abstract

This disclosure provides a pre-analysis-based method for adaptively compressing image data consumed by a machine vision task. An example method includes: receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, where the encoding includes: performing instance segmentation of the input pictures to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 378,888, filed October 10, 2022, and claims the benefit of U.S. Patent Application No. 18 / 478,680, entitled "PRE-ANALYSIS BASED IMAGE COMPRESSION METHODS," filed September 29, 2023. Both of these applications are incorporated herein by reference in their entirety.

[0002] Technical Field FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to video processing, and more particularly to methods and apparatus for pre-analysis-based image / video compression for machine vision tasks. [Background technology]

[0003] background

[0003] The rise of machine learning techniques and machine vision applications has led to rapid growth in the amount of video and images (collectively referred to as "image data") consumed by machines. Typical use cases include autonomous driving, intelligent transportation, smart cities, intelligent content management, etc., which incorporate machine vision tasks such as object detection, instance segmentation, and object tracking.

[0004]

[0004] Due to the large amount of image data required by machine vision tasks, it is essential to compress the image data for efficient transmission and storage. However, while traditional image / video compression techniques have focused on ensuring image / video quality as perceived by humans, machines consume and understand visual data differently from human vision. As a result, image / video compression techniques suitable for machine vision may differ from traditional ones. Therefore, new compression techniques are needed to achieve optimized performance for machine use. Summary of the Invention

[0005] Summary of the Disclosure

[0005] This disclosure provides a pre-analysis-based method for adaptively compressing image data consumed by a machine vision task. In some embodiments, an exemplary image data processing method includes: receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, where the encoding includes: performing instance segmentation of the input pictures to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region.

[0006]

[0006] In some embodiments, a non-transitory computer-readable storage medium stores a bitstream generated by a method, the method including performing instance segmentation of an input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region to generate the bitstream.

[0007]

[0007] In some embodiments, the image data processing device includes a memory storing a set of instructions and one or more processors, the one or more processors being configured to execute the set of instructions to cause the image data processing device to perform operations including: performing instance segmentation of the input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region.

[0008]

[0008] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, cause the computer to perform a method including: performing instance segmentation of an input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region to generate a bitstream.

[0009]

[0009] In some embodiments, the image data processing method includes: receiving a bitstream; and decoding one or more pictures by using encoded information in the bitstream, wherein the decoding includes: receiving a bitstream including compressed image data having region information, the region information indicating a region that the compressed image data represents within the picture; and reconstructing the picture by decoding the compressed image data in accordance with the region information.

[0010]

[0010] In some embodiments, a non-transitory computer-readable storage medium stores a bitstream including compressed image data with region information, the region information indicating a region that the compressed image data represents within a picture, and the bitstream is processed by a method including reconstructing the picture by decoding the compressed image data in accordance with the region information.

[0011]

[0011] In some embodiments, the image data processing device includes a memory storing a set of instructions and one or more processors, the one or more processors being configured to execute the set of instructions to cause the image data processing device to perform operations including: receiving a bitstream including compressed image data having region information, the region information indicating a region that the compressed image data represents within a picture; and reconstructing the picture by decoding the compressed image data in accordance with the region information.

[0012]

[0012] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, cause the computer to perform a method including: receiving a bitstream including compressed image data having region information, the region information indicating a region that the compressed image data represents within a picture; and reconstructing the picture by decoding the compressed image data in accordance with the region information.

[0013]

[0013] In some embodiments, a computer program product is provided that includes computer program instructions that enable a computer to carry out an image data processing method according to the image data processing method embodiments described above.

[0014] In some embodiments, a computer program is provided that enables a computer to carry out an image data processing method in accordance with the image data processing method embodiments above.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments and various aspects of the present disclosure are illustrated in the following detailed description and the accompanying drawings, in which various features are not drawn to scale. [Brief explanation of the drawings]

[0016] [Figure 1]

[0016] FIG. 1 is a schematic diagram illustrating an exemplary system for pre-processing and encoding image data according to some embodiments of the present disclosure. [Figure 2A]

[0017] FIG. 1 is a schematic diagram illustrating an exemplary encoding process of a hybrid video encoding system according to an embodiment of the present disclosure. [Figure 2B]

[0018] FIG. 2 is a schematic diagram illustrating another exemplary encoding process of a hybrid video encoding system according to an embodiment of the present disclosure. [Figure 3A]

[0019] FIG. 2 is a schematic diagram illustrating an exemplary decoding process of a hybrid video coding system according to an embodiment of the present disclosure. [Figure 3B]

[0020] FIG. 2 is a schematic diagram illustrating another exemplary decoding process of a hybrid video coding system according to an embodiment of the present disclosure. [Figure 4]

[0021] 1 is a block diagram of an exemplary apparatus for pre-processing or encoding image data according to some embodiments of the present disclosure. [Figure 5]

[0022] FIG. 1 is a schematic diagram illustrating an example compression method for visual data compression for machine vision according to some embodiments of the present disclosure. [Figure 6A]

[0023] 10A-10C illustrate various ways for pre-processing an input picture based on a pre-analysis result according to some embodiments of the present disclosure; [Figure 6B] 10A-10C illustrate various ways to pre-process an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 6C] 10A-10C illustrate various ways to pre-process an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 6D] 10A-10C illustrate various ways to pre-process an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 6E] 10A-10C illustrate various ways to pre-process an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 7]

[0024] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 8]

[0025] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 9]

[0026] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 10]

[0027] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 11]

[0028] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 12]

[0029] 1 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure. [Figure 13A]

[0030] 1 illustrates a schematic diagram of pre-processing an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 13B] 1 illustrates a schematic diagram of pre-processing an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 13C] 1 illustrates a schematic diagram of pre-processing an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 13D] 1 illustrates a schematic diagram of pre-processing an input picture based on a pre-analysis result according to some embodiments of the present disclosure. [Figure 14A]

[0031] 1 illustrates a schematic diagram of pre-processing an input picture based on block-based pre-processing according to some embodiments of the present disclosure. [Figure 14B] 1 illustrates a schematic diagram of pre-processing an input picture based on block-based pre-processing according to some embodiments of the present disclosure. [Figure 15A]

[0032] 1 illustrates a schematic diagram of pre-processing an input picture based on dilation-based pre-processing according to some embodiments of the present disclosure. [Figure 15B] 1 illustrates a schematic diagram of pre-processing an input picture based on extension-based pre-processing according to some embodiments of the present disclosure. [Figure 15C] 1 illustrates a schematic diagram of pre-processing an input picture based on extension-based pre-processing according to some embodiments of the present disclosure. [Figure 15D]1 illustrates a schematic diagram of pre-processing an input picture based on extension-based pre-processing according to some embodiments of the present disclosure. [Figure 16]

[0033] 1 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. [Figure 17]

[0034] 1 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description

[0035] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which like numbers in the various drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following description of exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, these implementations are merely examples of apparatus and methods consistent with aspects related to the present disclosure, as recited in the appended claims. Particular aspects of the present disclosure are described in more detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall control.

[0018]

[0036] This disclosure is directed to Video Coding for Machines (VCM), which aims to compress input video, images, or feature maps for machine vision tasks. Specifically, according to an example embodiment, prior to encoding, the input picture is pre-analyzed to detect segment masks corresponding to objects or foreground within the input picture. The segment masks are further pre-processed to form merged masks, which are then encoded (i.e., compressed). In some embodiments, the pre-processing may also identify stretched regions surrounding the merged mask. The stretched regions are also compressed by the encoder.

[0019]

[0037] According to disclosed embodiments, the pre-analysis and pre-processing may be performed by an image data pre-processor separate from the encoder. Alternatively, the pre-analysis may be performed by the encoder itself. This disclosure does not limit the hardware or software architecture for performing the image data pre-analysis.

[0020]

[0038] The disclosed technology is suitable for compressing image data used by any machine vision task, such as object recognition and tracking, facial recognition, image / video searching, mobile augmented reality (MAR), autonomous vehicles, Internet of Things (IoT), image matching, 3D structure construction, stereo correspondence, motion tracking, etc.

[0021]

[0039] 1 is a block diagram illustrating a system 100 for preprocessing and encoding image data according to some disclosed embodiments. The image data may include an image (also called a "picture" or "frame"), multiple images, or a video. An image is a static picture. Multiple images may or may not be related to each other spatially or temporally. A video is a set of images arranged in a time sequence.

[0022]

[0040] 1, system 100 includes a source device 120 that provides encoded video data that is subsequently decoded by a destination device 140. Consistent with disclosed embodiments, source device 120 and destination device 140 may each include any of a wide range of devices, including a desktop computer, a notebook (e.g., laptop) computer, a server, a tablet computer, a set-top box, a mobile phone, a vehicle, a camera, an image sensor, a robot, a television, a camera, a wearable device (e.g., a smartwatch or wearable camera), a display device, a digital media player, a video game console, a video streaming device, etc. Source device 120 and destination device 140 may be equipped for wireless or wired communication.

[0023]

[0041] Referring to FIG. 1, source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. Destination device 140 may include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. Image / video preprocessor 122 preprocesses image data (i.e., image(s), video(s)) and generates an input bitstream for image / video encoder 124. Image / video encoder 124 encodes the input bitstream and outputs an encoded bitstream 162 via output interface 126. Encoded bitstream 162 is transmitted via communication medium 160 and received by input interface 142. Image / video decoder 144 then decodes encoded bitstream 162 to generate decoded data that can be used by machine vision application 146.

[0024]

[0042] Specifically, source device 120 may further include various devices (not shown) for providing source image data to be preprocessed by image / video preprocessor 122. Devices for providing source image data may include image / video capture devices (cameras, image / video archives or storage devices containing previously captured images / video, image / video supply interfaces that receive images / video from image / video content providers, etc.).

[0025]

[0043] Each of the image / video encoder 124 and the image / video decoder 144 may be implemented as any of a wide variety of suitable encoder or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If the encoding or decoding is implemented partially in software, the image / video encoder 124 or the image / video decoder 144 may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware by using one or more processors to perform techniques according to this disclosure. Each of the image / video encoder 124 or the image / video decoder 144 may be included within one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the respective device.

[0026]

[0044] Image / video encoder 124 and image / video decoder 144 may operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video1 (AV1), Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc. Alternatively, image / video encoder 124 and image / video decoder 144 may be custom devices that do not conform to an existing standard. Although not shown in FIG. 1 , in some embodiments, image / video encoder 124 and image / video decoder 144 may each be integrated with an audio encoder and decoder and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams.

[0027]

[0045] Output interface 126 may include any type of medium or device capable of transmitting encoded bitstream 162 from source device 120 to destination device 140. For example, output interface 126 may include a transmitter or transceiver configured to transmit encoded bitstream 162 in real time directly from source device 120 to destination device 140. Encoded bitstream 162 may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 140.

[0028]

[0046] Communication medium 160 may include a transitory medium, such as a wireless broadcast or a wired network transmission. For example, communication medium 160 may include a radio frequency (RF) spectrum or one or more physical transmission paths (e.g., cables). Communication medium 160 may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet, etc.). In some embodiments, communication medium 160 may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from source device 120 to destination device 140. For example, a network server (not shown) may receive encoded bitstream 162 from source device 120 and provide encoded bitstream 162 to destination device 140 (e.g., via network transmission).

[0029]

[0047] Communication medium 160 may also be in the form of a storage medium (e.g., a non-transitory storage medium), such as a hard disk, a flash drive, a compact disc, a digital video disc, a Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, a computing device at a media production facility, such as a disc stamping facility, may receive the encoded image data from source device 120 and produce a disc containing the encoded video data.

[0030]

[0048] Input interface 142 may include any type of medium or device capable of receiving information from communication medium 160. The received information includes encoded bitstream 162. For example, input interface 142 may include a receiver or transceiver configured to receive encoded bitstream 162 in real time.

[0031]

[0049] The machine vision application 146 includes various hardware and / or software for utilizing the decoded image data generated by the image / video decoder 144. For example, the machine vision application 146 may include a display device that displays the decoded image data to a user and may include any of a wide variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. As another example, the machine vision application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications, such as object recognition and tracking, facial recognition, image matching, image / video search, augmented reality, robotic vision and navigation, autonomous driving, three-dimensional structure construction, stereo correspondence, motion tracking, etc.

[0032]

[0050] Exemplary image data encoding and decoding techniques will now be described with reference to FIGS. 2A-2B and 3A-3B.

[0033]

[0051] FIG. 2A shows a schematic diagram of an exemplary encoding process 200A according to an embodiment of the present disclosure. For example, encoding process 200A may be performed by an encoder, such as image / video encoder 124 of FIG. 1. As shown in FIG. 2A, the encoder may follow process 200A to encode a video sequence 202 into a video bitstream 228. The video sequence 202 may include a set of pictures (referred to as "original pictures") arranged in temporal order. Each original picture of the video sequence 202 may be divided by the encoder into basic processing units, basic processing sub-units, or regions for processing. In some embodiments, the encoder may perform process 200A at the level of basic processing units for each original picture of the video sequence 202. For example, the encoder may perform process 200A in an iterative manner, where the encoder may encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for a region of each original picture of video sequence 202.

[0034]

[0052] In FIG. 2A , an encoder may provide a basic processing unit (referred to as an “original BPU”) of an original picture of a video sequence 202 to a prediction stage 204 to generate prediction data 206 and a prediction BPU 208. The encoder may subtract the prediction BPU 208 from the original BPU to generate a residual BPU 210. The encoder may provide the residual BPU 210 to a transform stage 212 and a quantization stage 214 to generate quantized transform coefficients 216. The encoder may provide the prediction data 206 and the quantized transform coefficients 216 to a binary coding stage 226 to generate a video bitstream 228. Elements 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as a “forward path.” During process 200A, after quantization stage 214, the encoder may provide quantized transform coefficients 216 to inverse quantization stage 218 and inverse transform stage 220 to generate reconstructed residual BPU 222. The encoder may add reconstructed residual BPU 222 to prediction BPU 208 to generate prediction reference 224 used in prediction stage 204 for the next iteration of process 200A. Elements 218, 220, 222, and 224 of process 200A may be referred to as a "reconstruction path." The reconstruction path may be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0035]

[0053] The encoder may perform a process 200A that iterates between encoding each original BPU of an original picture (in the forward path) and generating (in the reconstruction path) a prediction reference 224 for encoding the next original BPU of the original picture. After encoding all original BPUs of an original picture, the encoder may proceed to encode the next picture in the video sequence 202.

[0036]

[0054] Referring to process 200A, an encoder may receive a video sequence 202 generated by a video capture device (e.g., a camera). As used herein, the term "receive" may refer to any act of receiving, inputting, obtaining, retrieving, acquiring, retrieving, accessing, or in any manner for inputting data.

[0037]

[0055] In the prediction stage 204, in the current iteration, the encoder may receive the original BPU and a prediction reference 224, perform a prediction operation, and generate predicted data 206 and a predicted BPU 208. The prediction reference 224 may be generated from a reconstruction path of a previous iteration of the process 200A. The purpose of the prediction stage 204 is to reduce information redundancy by extracting predicted data 206, which may be used to reconstruct the original BPU as a predicted BPU 208 from the prediction data 206 and the prediction reference 224.

[0038]

[0056] Ideally, predicted BPU 208 would be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, predicted BPU 208 typically differs slightly from the original BPU. To record such differences, after generating predicted BPU 208, the encoder may subtract it from the original BPU to generate residual BPU 210. For example, the encoder may subtract pixel values ​​(e.g., grayscale or RGB values) of predicted BPU 208 from corresponding pixel values ​​of the original BPU. Each pixel of residual BPU 210 may have a residual value resulting from such a subtraction between corresponding pixels of the original BPU and predicted BPU 208. Compared to the original BPU, predicted data 206 and residual BPU 210 may have fewer bits, which can be used to reconstruct the original BPU without significant quality degradation. Thus, the original BPU is compressed.

[0039]

[0057] To further compress the residual BPU 210, in the transform stage 212, the encoder may reduce spatial redundancy in the residual BPU 210 by decomposing the residual BPU 210 into a set of two-dimensional "basic patterns," each associated with a "transform coefficient." The base patterns may have the same size (e.g., the size of the residual BPU 210). Each base pattern may represent a variation frequency (e.g., frequency of luminance variation) component of the residual BPU 210. None of the base patterns can be reproduced from any combination (e.g., a linear combination) of any other base patterns. In other words, this decomposition may decompose the variation of the residual BPU 210 into the frequency domain. Such a decomposition is analogous to the discrete Fourier transform of a function, where the base patterns are analogous to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0040]

[0058] Different transform algorithms may use different base patterns. Different transform algorithms (e.g., discrete cosine transform, discrete sine transform, or the like) may be used in transform stage 212. The transform in transform stage 212 is invertible. That is, the encoder may reconstruct residual BPU 210 by inversely operating the transform (referred to as the "inverse transform"). For example, to reconstruct pixels of residual BPU 210, the inverse transform may multiply the values ​​of corresponding pixels of the base pattern by their associated coefficients and add these products to generate a weighted sum. For video coding standards, both the encoder and decoder may use the same transform algorithm (and therefore the same base pattern). Thus, the encoder may record only the transform coefficients, and the decoder may reconstruct residual BPU 210 from the transform coefficients without receiving the base pattern from the encoder. Compared to residual BPU 210, the transform coefficients may have fewer bits, but can be used to reconstruct residual BPU 210 without significant quality degradation. In this way, the residual BPU 210 is further compressed.

[0041]

[0059] The encoder may further compress the transform coefficients in the quantization stage 214. In the transform process, different base patterns may represent different fluctuation frequencies (e.g., luminance fluctuation frequencies). Because the human eye is generally better at perceiving low-frequency fluctuations, the encoder may ignore high-frequency fluctuation information without causing significant quality degradation during decoding. For example, in the quantization stage 214, the encoder may generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (referred to as a "quantization parameter") and rounding the quotient to the nearest integer. After such an operation, some transform coefficients of the high-frequency base pattern may be transformed to zero, and some transform coefficients of the low-frequency base pattern may be transformed to smaller integers. The encoder may ignore zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also invertible, where the quantized transform coefficients 216 may be reconstructed into transform coefficients in an inverse operation of quantization (referred to as "dequantization").

[0042]

[0060] Because the encoder ignores the remainder of such division in rounding operations, quantization stage 214 may be lossy. Typically, quantization stage 214 may contribute the greatest information loss in process 200A. The greater the information loss, the fewer bits the quantized transform coefficients 216 require. To employ different levels of information loss, the encoder may use different values ​​of the quantization parameter or any other parameter of the quantization process.

[0043]

[0061] In binary encoding stage 226, the encoder may encode the prediction data 206 and the quantized transform coefficients 216 by using a binary encoding technique (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm, etc.). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder may encode other information in binary encoding stage 226 (e.g., a prediction mode used in prediction stage 204, parameters of the prediction operation, a transform type in transform stage 212, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), or the like). The encoder may use output data of binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packetized for network transmission.

[0044]

[0062] Referring to the reconstruction path of process 200A, in inverse quantization stage 218, the encoder may perform inverse quantization on quantized transform coefficients 216 to generate reconstructed transform coefficients. In inverse transform stage 220, the encoder may generate reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder may add reconstructed residual BPU 222 to prediction BPU 208 to generate prediction reference 224, which will be used in the next iteration of process 200A.

[0045]

[0063] It should be noted that other variations of process 200A may be used to encode video sequence 202. In some embodiments, the stages of process 200A may be performed in a different order by the encoder. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be split into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may omit one or more stages of FIG. 2A.

[0046]

[0064] 2B shows a schematic diagram of another exemplary encoding process 200B according to an embodiment of the present disclosure. For example, encoding process 200B may be performed by an encoder such as image / video encoder 124 of FIG. 1. Process 200B may be modified from process 200A. For example, process 200B may be used by an encoder conforming to a hybrid video coding standard (e.g., the H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and splits prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filter stage 232 and a buffer 234.

[0047]

[0065] Generally, prediction techniques can be categorized into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-prediction") may use pixels from one or more already-encoded neighboring BPUs within the same picture to predict a current BPU. That is, the prediction reference 224 in spatial prediction may include neighboring BPUs. Spatial prediction may reduce inherent spatial redundancy of a picture. Temporal prediction (e.g., inter-picture prediction or "inter-prediction") may use regions from one or more already-encoded pictures to predict a current BPU. That is, the prediction reference 224 in temporal prediction may include an encoded picture. Temporal prediction may reduce inherent temporal redundancy of a picture.

[0048]

[0066] Referring to process 200B, in the forward path, the encoder performs prediction operations in a spatial prediction stage 2042 and a temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder may perform intra prediction. For an original BPU of a picture being coded, the prediction reference 224 may include one or more neighboring BPUs coded (in the forward path) and reconstructed (in the reconstruction path) within the same picture. The encoder may generate the predicted BPU 208 by extrapolating the neighboring BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, or the like. In some embodiments, the encoder may perform extrapolation at the pixel level, such as by extrapolating the value of a corresponding pixel for each pixel of the predicted BPU 208. The neighboring BPUs used for extrapolation may be located relative to the original BPU from various directions, such as vertically (e.g., above the original BPU), horizontally (e.g., to the left of the original BPU), diagonally (e.g., below-left, below-right, above-left, or above-right of the original BPU), or any direction defined in the video coding standard used. For intra prediction, the prediction data 206 may include, for example, the locations (e.g., coordinates) of the neighboring BPUs used, the sizes of the neighboring BPUs used, parameters of the extrapolation, the orientation of the neighboring BPUs used relative to the original BPU, or the like.

[0049]

[0067] For another example, in the temporal prediction stage 2044, the encoder may perform inter-prediction. For an original BPU of a current picture, the prediction reference 224 may include one or more pictures (referred to as "reference pictures") that have been coded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference pictures may be coded and reconstructed for each BPU. For example, the encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. Once all reconstructed BPUs of the same picture have been generated, the encoder may generate the reconstructed picture as the reference picture. The encoder may perform a "motion estimation" operation to search for a matching region within a certain range (referred to as a "search window") of the reference picture. The location of the search window in the reference picture may be determined based on the location of the original BPU of the current picture. For example, the search window may be centered at a location in the current picture that has the same coordinates in the reference picture as the coordinates of the original BPU, and may extend outward a predetermined distance. When the encoder identifies a region similar to the original BPU within the search window (e.g., by using a pel-recursive algorithm, a block matching algorithm, or the like), the encoder may determine such a region as a matching region. The matching region may have different dimensions than the original BPU (e.g., smaller than, equal to, larger than, or a different shape than the original BPU). Because the reference picture and the current picture are temporally separated in the timeline, the matching region may be considered to "move" toward the location of the original BPU over time. The encoder may record the direction and distance of such movement as a "motion vector." When multiple reference pictures are used, the encoder may search for the matching region for each reference picture and determine its associated motion vector. In some embodiments, the encoder may assign weights to the pixel values ​​of the matching region in each matching reference picture.

[0050]

[0068] Motion estimation may be used to identify various types of motion, such as, for example, translation, rotation, zooming, or the like. For inter prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, a motion vector associated with the matching region, the number of reference pictures, weights associated with the reference pictures, or the like.

[0051]

[0069] To generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. Motion compensation may be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., a motion vector) and the prediction reference 224. For example, the encoder may shift the matching region of the reference picture according to the motion vector, where the encoder may predict the original BPU of the current picture. If multiple reference pictures are used, the encoder may shift the matching region of the reference picture according to each motion vector and average the pixel values ​​of the matching region. In some embodiments, if the encoder assigns weights to the pixel values ​​of the matching region of each matching reference picture, the encoder may add a weighted sum of the pixel values ​​of the shifted matching region.

[0052]

[0070] In some embodiments, inter prediction may be unidirectional or bidirectional. Unidirectional inter prediction may use one or more reference pictures in the same temporal direction relative to the current picture. Unidirectional inter prediction uses a reference picture preceding the current picture. Bidirectional inter prediction may use one or more reference pictures in both temporal directions relative to the current picture.

[0053]

[0071] Still referring to the forward path of process 200B, after spatial prediction stage 2042 and temporal prediction stage 2044, in mode decision stage 230, the encoder may select a prediction mode (e.g., one of intra-prediction or inter-prediction) for the current iteration of process 200B. For example, the encoder may perform a rate-distortion optimization technique, where the encoder may select a prediction mode to minimize the value of a cost function depending on the bitrates of the candidate prediction modes and the distortion of the reconstructed reference picture under the candidate prediction modes. Depending on the selected prediction mode, the encoder may generate a corresponding predicted BPU 208 and predicted data 206.

[0054]

[0072] In the reconstruction path of process 200B, if an intra-prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current BPU coded and reconstructed in the current picture), the encoder may directly provide prediction reference 224 to spatial prediction stage 2042 for later use (e.g., for extrapolation of the next BPU of the current picture). If an inter-prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current picture coded and reconstructed in all BPUs), the encoder may provide prediction reference 224 to loop filter stage 232, where the encoder may apply a loop filter to prediction reference 224 to reduce or remove distortion (e.g., blocking artifacts) introduced by inter prediction. The encoder may apply various loop filter techniques in loop filter stage 232, such as, for example, deblocking, sample adaptive offset, adaptive loop filter, or the like. The loop-filtered reference pictures may be stored in a buffer 234 (or a "decoded picture buffer") for later use (e.g., to be used as inter-predicted reference pictures for future pictures of the video sequence 202). The encoder may store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder may encode loop filter parameters (e.g., loop filter strength) along with the quantized transform coefficients 216, the prediction data 206, and other information in the binary encoding stage 226.

[0055]

[0073] FIG. 3A shows a schematic diagram of an exemplary decoding process 300A according to an embodiment of the present disclosure. For example, the decoding process 300A may be performed by a decoder such as the image / video decoder 144 of FIG. 1. The process 300A may be a decompression process corresponding to the compression process 200A of FIG. 2A. In some embodiments, the process 300A may resemble the reconstruction path of the process 200A. A decoder (e.g., the image / video decoder 144 of FIG. 1) may decode the video bitstream 228 according to the process 300A into a video stream 304. The video stream 304 may be very similar to the video sequence 202. However, due to information loss in the compression and decompression processes (e.g., the quantization stage 214 in FIGS. 2A-2B), the video stream 304 is generally not identical to the video sequence 202. 2A-2B, the decoder may perform process 300A at the level of a basic processing unit (BPU) for each picture encoded in video bitstream 228. For example, the decoder may perform process 300A in an iterative manner (the decoder may decode a basic processing unit in one iteration of process 300A). In some embodiments, the decoder may perform process 300A in parallel for several regions of each picture encoded in video bitstream 228.

[0056]

[0074] In FIG. 3A , a decoder may provide a portion of a video bitstream 228 associated with a basic processing unit (referred to as a “coded BPU”) of a coded picture to a binary decoding stage 302. In the binary decoding stage 302, the decoder may decode this portion into prediction data 206 and quantized transform coefficients 216. The decoder may provide the quantized transform coefficients 216 to an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may provide the prediction data 206 to a prediction stage 204 to generate a predicted BPU 208. The decoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder may provide prediction reference 224 to prediction stage 204 for performing prediction operations in the next iteration of process 300A.

[0057]

[0075] The decoder may perform process 300A to repeat decoding each coded BPU of a coded picture and generating a prediction reference 224 for coding the next coded BPU of the coded picture. After decoding all coded BPUs of a coded picture, the decoder may output the picture to the video stream 304 for display and proceed to decode the next coded picture in the video bitstream 228.

[0058]

[0076] In binary decoding stage 302, the decoder may perform the inverse operation of the binary coding technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to prediction data 206 and quantized transform coefficients 216, the decoder may decode other information in binary decoding stage 302, such as, for example, a prediction mode, parameters of the prediction operation, a transform type, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), or the like. In some embodiments, if video bitstream 228 is transmitted over a network in packets, the decoder may depacketize video bitstream 228 before providing it to binary decoding stage 302.

[0059]

[0077] 3B shows a schematic diagram of another exemplary decoding process 300B according to an embodiment of the present disclosure. For example, the decoding process 300B may be performed by a decoder such as the image / video decoder 144 of FIG. 1. The process 300B may be modified from the process 300A. For example, the process 300B may be used by a decoder compliant with a hybrid video coding standard (e.g., the H.26x series). Compared to the process 300A, the process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filter stage 232 and a buffer 234.

[0060]

[0078] In process 300B, prediction data 206 decoded by the decoder from binary decoding stage 302 for a coded elementary processing unit (referred to as a "current BPU") of a coded picture being decoded (referred to as a "current picture") may include various types of data, depending on which prediction mode was used by the encoder to code the current BPU. For example, if intra prediction is used by the encoder to code the current BPU, prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating intra prediction, parameters of the intra prediction operation, or the like. Parameters of the intra prediction operation may include, for example, locations (e.g., coordinates) of one or more neighboring BPUs used as references, sizes of the neighboring BPUs, parameters of extrapolation, orientations of the neighboring BPUs relative to the original BPU, or the like. For another example, if inter prediction is used by the encoder to code the current BPU, prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating inter prediction, parameters of the inter prediction operation, or the like. Parameters for the inter prediction operation may include, for example, the number of reference pictures currently associated with the BPU, weights associated with each of the reference pictures, locations (e.g., coordinates) of one or more matching regions within each reference picture, one or more motion vectors associated with each of the matching regions, or the like.

[0061]

[0079] Based on the prediction mode indicator, the decoder may determine whether to perform spatial prediction (e.g., intra prediction) in spatial prediction stage 2042 or temporal prediction (e.g., inter prediction) in temporal prediction stage 2044. Details of performing such spatial or temporal prediction are described in FIG. 2B and will not be repeated hereinafter. After performing such spatial or temporal prediction, the decoder may generate a predicted BPU 208. The decoder may add the predicted BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, as described in FIG. 3A.

[0062]

[0080] In process 300B, the decoder may provide a prediction reference 224 to spatial prediction stage 2042 or temporal prediction stage 2044 for performing a prediction operation in a next iteration of process 300B. For example, if the current BPU is decoded by using intra prediction in spatial prediction stage 2042, after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder may provide the prediction reference 224 directly to spatial prediction stage 2042 for later use (e.g., for extrapolation of the next BPU of the current picture). If the current BPU is decoded by using inter prediction in temporal prediction stage 2044, after generating the prediction reference 224 (e.g., the reference picture from which all BPUs are decoded), the encoder may provide the prediction reference 224 to loop filter stage 232 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder may apply a loop filter to the prediction reference 224 in the manner described in FIG. 2B . The loop-filtered reference picture may be stored in a buffer 234 (e.g., a decoded picture buffer in computer memory) for later use (e.g., to be used as an inter-prediction reference picture for a future encoded picture of the video bitstream 228). The decoder may store one or more reference pictures used in the temporal prediction stage 2044 in the buffer 234. In some embodiments, if the prediction mode indicator in the prediction data 206 indicates that inter-prediction was used to encode the current BPU, the prediction data may further include loop filter parameters (e.g., loop filter strength).

[0063]

[0081] Referring back to FIG. 1 , each of the image / video preprocessor 122, the image / video encoder 124, and the image / video decoder 144 may be implemented as any suitable hardware, software, or combination thereof. FIG. 4 is a block diagram of an example device 400 for processing image data according to an embodiment of the present disclosure. For example, the device 400 may be a preprocessor, an encoder, or a decoder. As shown in FIG. 4 , the device 400 may include a processor 402. When the processor 402 executes the instructions described herein, the device 400 may be a dedicated machine for preprocessing, encoding, and / or decoding image data. The processor 402 may be any type of circuitry capable of manipulating or processing information. For example, processor 402 may include any number or combination of a central processing unit (or “CPU”), a graphics processing unit (or “GPU”), a neural processing unit (“NPU”), a microcontroller unit (“MCU”), an optical processor, a programmable logic controller, a microcontroller, a microprocessor, a digital signal processor, an intellectual property (IP) core, a programmable logic array (PLA), a programmable array logic (PAL), a generic array logic (GAL), a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a system on a chip (SoC), an application-specific integrated circuit (ASIC), or the like.In some embodiments, processor 402 may also be a set of processors grouped together as a single logic element. For example, as shown in FIG. 4, processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0064]

[0082] The device 400 may also include a memory 404 configured to store data (e.g., a set of instructions, computer code, intermediate data, or the like). For example, as shown in FIG. 4, the stored data may include program instructions (e.g., program instructions for implementing stages in processes 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). The processor 402 may access the program instructions and data for processing (e.g., via bus 410) and execute the program instructions to perform operations or manipulations on the data for processing. The memory 404 may include a high-speed random-access storage device or a non-volatile storage device. In some embodiments, memory 404 may include any number or combination of random-access memory (RAM), read-only memory (ROM), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, security digital (SD) cards, memory sticks, compact flash (CF) cards, or the like. Memory 404 may also be a collection of memories (not shown in FIG. 4) grouped as a single logical element.

[0065]

[0083] Bus 410 may be a communication device that transfers data between elements internal to apparatus 400, such as an internal bus (e.g., a CPU-memory bus), an external bus (e.g., a universal serial bus port, a peripheral component interconnect express port), or the like.

[0066]

[0084] For ease of explanation and without causing ambiguity, the processor 402 and other data processing circuitry will be collectively referred to in this disclosure as "data processing circuitry." The data processing circuitry may be implemented entirely as hardware or as a combination of software, hardware, or firmware. In addition, the data processing circuitry may be a single, stand-alone module or may be fully or partially coupled to any other element of the device 400.

[0067]

[0085] Device 400 may further include a network interface 406 for providing wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, or the like). In some embodiments, network interface 406 may include any number or combination of a network interface controller (NIC), a radio frequency (RF) module, a transponder, a transceiver, a modem, a router, a gateway, a wired network adapter, a wireless network adapter, a Bluetooth adapter, an infrared adapter, a near-field communication ("NFC") adapter, a cellular network chip, or the like.

[0068]

[0086] In some embodiments, apparatus 400 may optionally further include a peripheral interface 408 for providing connection to one or more peripheral devices. As shown in Figure 4, the peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touchscreen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), a video input device (e.g., a camera, or an input interface coupled to a video archive), etc.

[0069]

[0087] It should be noted that a video codec (e.g., a codec performing process 200A, 200B, 300A, or 300B) may be implemented as any combination of software or hardware modules within apparatus 400. For example, some or all stages of process 200A, 200B, 300A, or 300B may be implemented as one or more software modules of apparatus 400, such as program instructions that may be loaded into memory 404. As another example, some or all stages of process 200A, 200B, 300A, or 300B may be implemented as one or more hardware modules of apparatus 400, such as dedicated data processing circuitry (e.g., FPGA, ASIC, NPU, or the like).

[0070]

[0088] With the development of multimedia processing, transmission, and applications, there has been an explosive growth in visual data. Furthermore, storing and transmitting original visual data is impractical due to the enormous storage and transmission costs of the original visual data. Meanwhile, tremendous redundancy in visual data exists, motivating a compact representation of visual data (making the storage and transmission of visual data possible in real-world applications). Traditionally, the ultimate consumer of visual data is human vision, and thus compact representations of visual data are designed toward human-perceptual quality. However, in recent years, with the unprecedented development of artificial intelligence, there have been many deep-learning-based machine analysis applications for visual data (called machine vision). In machine-vision-oriented applications, machine analysis performance has replaced human-perceptual quality as the final metric for visual data compression (which is outside the scope of visual data compression algorithms designed for human vision). Therefore, there is a need to develop a compact representation of visual data for machine vision. To address this issue, the present disclosure provides an image data compression method using pre-analysis for machine vision. Specifically, pre-analysis for machine vision can extract critical information for machine analysis for better analysis performance and eliminate irrelevant information for machine analysis for a more compact representation. Furthermore, the disclosed pre-analysis algorithm can be compatible with human-visual-oriented compression codecs, meaning that the algorithm is a universal technique for visual data compression for machine vision applications.

[0071]

[0089] The development of image / video compression algorithms is driven by coding standards. Regarding texture compression, a series of standards have been developed to compress visual data, such as JPEG and JPEG2000 for still image compression, and H.264 / AVC, H.265 / HEVC, and VVC (Versatile Video Coding) for video data compression. To further improve compression performance, several algorithms have been developed for future video compression standards, including matrix-weighted intra-prediction, quadtree plus binary, extended coding unit partitioning, and mode-dependent non-separable secondary transform. Meanwhile, various optimization algorithms have been proposed in terms of rate-distortion optimization for both texture and feature quality through encoder optimization. Furthermore, with the unprecedented development of visual data understanding, there is a tremendous challenge in compactly managing thousands of visual data bitstreams and simultaneously transmitting them for further analysis, such as in smart cities and the Internet of Video Things (IoVT). Furthermore, analysis performance can be dramatically affected due to the degradation of feature quality in human visual quality-oriented compression. To address this issue, standards for compact visual feature representation have also been developed by the Moving Picture Experts Group (MPEG) to reduce the representation data size of analysis features, which may facilitate various intelligent tasks through front-end intelligence. Specifically, the Compact Descriptors for Visual Search (CDVS) ​​and Compact Descriptors for Video Analysis (CDVA) standards have been completed, which aim to achieve highly compact descriptors for visual data.Furthermore, standardization of video coding for machine vision has also been launched in an attempt to come up with a complete picture of compact representation of visual data in terms of machine vision.

[0072]

[0090] Furthermore, deep learning can be used in various applications, especially in the visual data representation and understanding domain. In particular, deep neural network-based end-to-end compression frameworks can be used. Recurrent neural networks (RNNs) can be applied to end-to-end learned image representations to achieve comparable performance to JPEG. Motivated by block-based transforms in traditional image / video compression, convolutional neural network (CNN)-based end-to-end image compression models can be combined with discrete cosine transforms (DCTs) to achieve comparable performance to JPEG at low bitrates. Nonlinear transformations are one of the essential properties of neural networks that mimic the human visual system (HVS). From this, generalized divisive normalization (GDN) can be used to optimize end-to-end nonlinear transform codecs for perceptual quality. Based on this, density estimation models can be combined with cascades of GDNs to exceed the compression performance of JPEG2000. The redundancy of latent codes in end-to-end image compression can be further eliminated under an entropy penalty constraint by weight reparameterization implemented by a variational hyper-prior model.To further exploit the correlation of latent representations, an autoregressive model can be used to achieve superior rate-distortion performance compared to the current state-of-the-art image codec BPG in terms of both PSNR and MS-SSIM distortion metrics.To further improve the accuracy of the entropy model for rate estimation, discrete Gaussian Mixture Likelihoods, which have the potential to formulate a more accurate and flexible entropy model, can be used to parameterize the distribution of latent representations and achieve performance comparable to the state-of-the-art compression standard VVC in terms of bitrate-PSNR performance.

[0073]

[0091] Generally, conventional visual data compression is achieved by compactly representing local and low-level information in a patch-like manner. However, this may limit the compact representation capability because high-level information is not fully utilized. To address this issue, pre-analysis can be used to improve representation performance by extracting global and high-level information of visual data. Specifically, a visual attention-based pre-analysis model can be used to optimize coding parameter settings for better compression performance. Furthermore, pre-analysis can also be applied to rate control for real-time video coding. From the perspective of simplifying coding complexity, pre-analysis can also be applied to accelerating integer motion estimation in JEM.

[0074]

[0092] As mentioned above, there have been numerous developments in visual data compression in recent decades. With the progress of various machine analysis tasks, several analysis feature compression algorithms have also been proposed to improve the efficiency of visual data compression for machine vision and accommodate the rapid development of machine analysis applications. However, existing visual data codecs primarily focus on signal fidelity and human visual quality, not machine vision. Although pre-analysis has been investigated to improve the representation efficiency of visual data and accommodate existing codecs, existing pre-analysis methods primarily focus on human perceptual quality, which limits the performance of compact visual data representation for machine vision.

[0075]

[0093] The present disclosure provides a pre-analysis-based compression method that is suitable for machine vision.

[0076]

[0094] FIG. 5 is a schematic diagram illustrating an example compression method 500 for visual data compression for machine vision according to some embodiments consistent with this disclosure. For example, method 500 may be performed by image / video preprocessor 122 (FIG. 1). As shown in FIG. 5, method 500 includes two stages: a machine vision-based pre-analysis stage 510 and a pre-processing stage 520. Specifically, machine vision-based pre-analysis stage 510 performs high-level information extraction using a machine analysis model (e.g., instance segmentation) and detects one or more segmentation masks 512, each corresponding to one or more instances (i.e., objects or foreground). Pre-analysis stage 510 may also execute a classifier (e.g., a neural network classifier) ​​to determine a class (not shown in FIG. 5) associated with each of the segmentation masks 512. The class may be an object class indicating the type of object (e.g., face, tree, animal, building, etc.) associated with the respective segmentation mask. The pre-processing stage 520 includes operations formulated based on the output of the pre-analysis stage 510 (e.g., segment masks and predicted classes) to trade off machine vision performance and representation compactness. The pre-analysis stage 510 may employ machine analysis models such as Faster R-CNN and Mask R-CNN.

[0077]

[0095] For example, Mask R-CNN is a deep learning model that combines object detection and instance segmentation. According to some embodiments, the overall architecture of an exemplary Mask R-CNN may include: (1) a backbone network, (2) a region proposal network (RPN), (3) ROIAlign, and (4) a mask head.

[0078]

[0096] (1) Backbone network: The backbone network can be a pre-trained convolutional neural network such as ResNet or ResNeXt. This backbone processes the input image and generates a corresponding feature map.

[0079]

[0097] (2) Region Proposal Network (RPN): The RPN is responsible for generating region proposals or candidate bounding boxes that may contain objects in the image. The RPN operates on the feature maps generated by the backbone network and proposes possible regions of interest (ROIs).

[0080]

[0098] (3) ROIAlign: The main purpose of ROIAlign is to align features within a region of interest (ROI) with the spatial grid of the output feature map. This alignment can prevent information loss that can occur when quantizing the spatial coordinates of ROIs to nearest integers (as done in ROI pooling).

[0081]

[0099] (4) Mask Head: The Mask Head is an additional branch within Mask R-CNN that is responsible for generating a segmentation mask for each region proposal. The Mask Head uses the aligned features obtained via ROIAlign to predict a binary mask for each object and determines the pixel boundary contour of the instance.

[0082]

[0100] During training, Mask R-CNN models are jointly optimized by using a combination of classification loss, bounding box regression loss, and mask segmentation loss, which allows "Mask R-CNN models to learn to simultaneously detect multiple objects, refine their bounding boxes, and generate fine-grained masks."

[0083]

[0101] As described above, once the pre-analysis stage 510 determines one or more instances in the input picture that belong to a region of interest (ROI), the instances may be further filtered out using the corresponding segment masks 512 by the pre-processing stage 520. Then, the background portions of the input picture may be omitted and will not participate in subsequent processing. Compared to the original input picture, the filtered-out portions, which have less picture information, may offer the potential for better performance for image coding (particularly for machine vision). In the above example, the machine vision pre-analysis model is instance segmentation, and the output of the pre-analysis stage includes segment masks and predicted classes. However, it is contemplated that the disclosed embodiments are not limited to the above example and may use any suitable machine vision pre-analysis and pre-processing techniques to improve the representation efficiency of visual data for machine vision.

[0084]

[0102] Next, details of the pre-processing stage 520 will be described. FIGS. 6A-6E schematically illustrate various ways for pre-processing an input picture based on pre-analysis results, according to some embodiments of the present disclosure. In some embodiments, the pre-processing operation may be performed directly on the segment mask. Specifically, because the compressed visual data is consumed by machine vision, only machine analysis-related information is essential and therefore needs to be preserved. The machine analysis-related information includes the segment mask output by the pre-analysis stage (e.g., pre-analysis stage 510 of FIG. 5). For example, FIG. 6A illustrates an input picture showing multiple objects (e.g., people), each corresponding to multiple instances (i.e., segment masks) identified in the pre-analysis stage. To pre-process the segment masks directly, the segment masks of the multiple instances are combined together to achieve a merged mask in a union manner. For example, as shown in FIG. 6B, segment masks corresponding to objects (i.e., people) may be combined to form a merged mask. Only the information in the merged mask is preserved for further compression. The technical aspects of Figures 6A-6E will be further described in conjunction with Figures 7-15. However, as can be understood by those skilled in the art, although some of these figures are described in conjunction below, it does not necessarily mean that their corresponding embodiments should be realized in a combined manner.

[0085]

[0103] In some embodiments, block-based preprocessing can be used to process the segmentation masks output by the pre-analysis stage. Unlike preserving the instance mask region to extract only the most unique information, the surrounding area of ​​the merged mask is also important to achieve better discrimination of instances. Motivated by this, a block-based preprocessing operation can be used to preserve the visual information surrounding the instance, as shown in FIG. 6C. Specifically, a sliding window s n is performed on the merged mask, and the window size is n×n, where n can be, for example, 128, 192, or 256. The information of the original visual data is filtered through a sliding window s as shown in FIG.n Whenever there is an intersection between the and the merged mask, it may be preserved. Figures 14A and 14B show a schematic representation of the block-based pre-processing described below.

[0086]

[0104] In some embodiments, dilation-based preprocessing may be used to process the segmentation mask output by the pre-analysis stage. Dilation is an important image processing method. Herein, preserving information surrounding the predicted instance involves a preprocessing operation. Specifically, as shown in FIG. 6D, dilation is performed on the merged mask to expand the predicted instance region, and the kernel size is k×k and t iterations (7×7 and 3, respectively). Similarly, information of the original visual data within the corresponding region may be preserved for further compression, as shown in FIG. 6D. FIGS. 15A-15D schematically illustrate the dilation-based preprocessing described below.

[0087]

[0105] In some embodiments, block-based preprocessing and extension-based preprocessing may be combined to process the segment mask output by the pre-analysis stage. The granularity of the block-based preprocessing operation is coarse, and therefore redundant information may be introduced. However, the irregular boundaries of the visual data processed by the extension-based preprocessing operation may increase the representation cost of visual data compression, especially for block-based compression codecs such as HEVC and VVC. To address this issue, extension and block-based preprocessing operations may be used, as shown in FIG. 6E. Specifically, extension-based preprocessing is performed first, and then block-based preprocessing is performed based on the output of the extension-based preprocessing. Here, the sliding window size n s can be smaller (such as 64 and 128) as shown in Figure 6E.

[0088]

[0106] In some embodiments, blur preprocessing may be used to improve the smoothness and representation compactness of the image data. To prevent blurring of the merged mask region (where the object resides) from degrading the performance or accuracy of the machine vision task (e.g., object recognition or tracking), the blur preprocessing output Iblur But, for example, I blur =G k (M bd -M)I+M I, where I is the original image, M is the merged mask, and M bd is the mask after block and / or dilation-based preprocessing, and G k is a Gaussian filter with kernel size k. In this way, in block and / or dilation preprocessed images, blur preprocessing is applied only to image regions outside (i.e., not overlapping with) the merged mask, thereby improving compression efficiency without degrading machine analysis performance.

[0089]

[0107] In some embodiments, to improve compression efficiency, the pre-processing stage may be performed adaptively based on the quantization parameter (QP) and / or the resolution of the input visual data.

[0090]

[0108] Specifically, a lower QP means higher representation cost and better visual data reconstruction quality when the compression codec is a conventional codec such as HEVC or VVC. When the QP is high (e.g., greater than 32), the representation cost is limited, and therefore more coding bits should be allocated to the instance region for better reconstruction quality. Therefore, the sliding window size n in block-based preprocessing and the kernel size in extension-based preprocessing can be made smaller (e.g., 64 and 3×3, respectively). Furthermore, in high QP situations, blur preprocessing can also be applied to the output of block- and / or extension-based preprocessing to further improve compression efficiency. Conversely, when the QP is small (e.g., less than 32), the coding cost is sufficient, and more information about the original visual data can be preserved for better machine analysis performance. For example, the window size and kernel size can be 256 and 9×9, respectively.

[0091]

[0109] Furthermore, the resolution of the input visual data is also important. For visual data with a high resolution (e.g., higher than 1920x1080), the sliding window size of the block-based preprocessing and the kernel size of the expansion-based preprocessing may be larger (e.g., 256 and 7x7, respectively). Conversely, for visual data with a low resolution (e.g., smaller than 1920x1080), the sliding window size of the block-based preprocessing and the kernel size of the expansion-based preprocessing may be smaller (e.g., 128 and 3x3, respectively).

[0092]

[0110] Consistent with embodiments of the present disclosure, adaptive pre-processing may be based on the QP, the resolution of the input picture, or a combination thereof.

[0093]

[0111] 7 is a flowchart of an exemplary image data compression method 700 according to some embodiments of the present disclosure. For example, method 700 may be performed by one or more processors, such as image / video pre-processor 122 (FIG. 1) and / or image / video encoder 124 (FIG. 1). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into apparatus 400 shown in FIG. 4 such that method 700 may be performed by apparatus 400. As shown in FIG. 7, method 700 includes the following steps 710-740:

[0094]

[0112] In step 710, the processor performs instance segmentation of the input picture to generate one or more segment masks.

[0095]

[0113] Specifically, the processor may execute an instance segmentation algorithm to partition the input picture into multiple segments (e.g., multiple sets of pixels, each representing a portion of the input picture). The instance segmentation algorithm may assign a label (i.e., a category) to each set of pixels such that pixels with the same label share certain characteristics. The processor may then group the pixels according to their labels and designate each group as a distinct object (i.e., a segment mask). Based on these labels, the processor may determine the boundaries of each segment mask. These labels do not necessarily belong to different classes. For example, the two people in FIG. 6B may be labeled differently but belong to the same class of object (person). In some embodiments of the present disclosure, each of the two people is an instance and may be detected by the instance segmentation algorithm. For example, the processor may execute the instance segmentation algorithm on the input image as shown in FIGS. 13A-13D. As a result, the pixels belonging to the instances (two people) may be labeled as "Person A" and "Person B." Meanwhile, segment masks 1310 and 1320 are generated accordingly as shown in Figures 13A and 13B, where mask 1310 is generated by grouping pixels of "Person A" and mask 1320 is generated by grouping pixels of "Person B."

[0096]

[0114] In some embodiments, after the segment masks are generated, the processor may also execute a classification algorithm to determine an object class associated with each of the segment masks. For example, the classification algorithm may be implemented by using a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN).

[0097]

[0115] Referring again to FIG. 7, in step 720, the processor combines one or more segment masks to generate a merged mask. Specifically, the merged masks may form a connected region in the input picture. For example, as shown in FIG. 6B, segment masks corresponding to multiple people may be merged to form a connected region. Returning to FIG. 13C, segment mask 1310 and segment mask 1320 are now combined to form merged mask 1330. Combining different segment masks reduces the number of masks, thereby reducing the computational load for image processing. On the other hand, combining different segment masks, together with other morphological techniques described below (e.g., block-based preprocessing or dilation-based preprocessing), reduces the risk of losing part of an instance (which may be a region of interest (ROI) in the input picture) due to mis-segmentation performed in step 710.

[0098]

[0116] In some embodiments, the segment masks generated in step 710 may be isolated from one another. Under this condition, the merged mask combined with the isolated masks will consist of several non-overlapping portions. In some other implementations, the segment masks generated in step 710 may be adjacent to one another. Under this condition, the merged mask will be made up of several connected regions. As shown in Figures 13C and 13D, the merged mask 1330 and corresponding region 1340 are saved in the picture for later processing, while the background region 1350 in the input picture may be ignored and may not participate in later processing.

[0099]

[0117] Referring again to FIG. 7, in step 730, the processor extracts a region from the input picture that contains the merged mask.

[0100]

[0118] In some embodiments, the extracted region includes only the merged mask, for example, as shown in Figure 6B, where only the merged mask is extracted from the input picture.

[0101]

[0119] In some embodiments, the extracted region not only includes the merged mask, but also includes an expanded region surrounding the merged mask. For example, as shown in Figures 6c-6e, block-based preprocessing and expansion-based preprocessing can be used to generate an expanded region surrounding the merged mask.

[0102]

[0120] As already mentioned above, the merged mask may consist of several parts that are not adjacent to each other, which will increase the complexity or reduce the efficiency of the image coding in any case. The morphological processing with the merged mask will introduce parts that may be regions of interest (ROIs) and in some cases will bridge the non-overlapping parts.

[0103]

[0121] When block-based pre-processing is used, the processor applies a sliding window to the image region outside the merged mask to identify one or more blocks that do not overlap with the merged mask. The processor then extracts regions from the input picture that do not contain one or more blocks. Figure 8 illustrates block-based pre-processing, which is described in detail below.

[0104]

[0122] When extension-based pre-processing is used, the processor applies a kernel to the merged mask to determine a first region that extends from the boundary of the merged mask. The processor then extracts the first region and the merged mask from the input picture. Figure 9 illustrates extension-based pre-processing, which is described in detail below.

[0105]

[0123] In some embodiments, block-based pre-processing and augmentation-based pre-processing may be combined. First, the processor augments the merged mask by applying a kernel to the merged mask to determine a first region extending from the boundary of the merged mask. Next, the processor identifies one or more blocks in the input picture that do not overlap with the first region or the merged mask. Finally, the processor extracts a region from the input picture that does not include one or more blocks. Figure 10 illustrates block-based and augmentation-based pre-processing, which will be described in detail below.

[0106]

[0124] In some embodiments, blur preprocessing may be applied to the region containing the merged mask. Specifically, in block and / or dilation-based preprocessed images, blur preprocessing is applied to the stretched region surrounding the merged mask to remove fine details or simple content within the stretched region. Because the stretched region does not contain any objects, blurring the stretched region does not affect the accuracy of the machine vision task. Figure 12 illustrates blur preprocessing applied to the stretched region, which is described in detail below.

[0107]

[0125] In some embodiments, block-based pre-processing, extension-based pre-processing, and / or blur pre-processing may be adaptively performed based on the quantization parameter (QP) for compressing the input picture or the resolution of the input picture. For example, in a high QP situation (e.g., when the QP is greater than a predetermined threshold), the sliding window size in block-based pre-processing and the kernel size in extension-based pre-processing may be made smaller, and blur pre-processing may also be used to blur image regions other than the merged mask. Conversely, in a low QP situation (e.g., when the QP is equal to or less than a predetermined threshold), the sliding window size in block-based pre-processing and the kernel size in extension-based pre-processing may be made larger, and blur pre-processing may be skipped. Figure 12 illustrates QP-based blur pre-processing, which will be described in detail below.

[0108]

[0126] 7, at step 740, the processor compresses image data representing the extracted region. Specifically, the image data may be fed into a codec for compression at a particular QP. The codec may encode and / or decode in accordance with image / video standards such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc.

[0109]

[0127] 8 is a flowchart of an exemplary image data compression method 800 based on method 700 shown in FIG. 7 , according to some embodiments of the present disclosure. For example, method 800 may be performed by one or more processors, such as image / video pre-processor 122 ( FIG. 1 ) and / or image / video encoder 124 ( FIG. 1 ). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into device 400 shown in FIG. 4 , such that method 800 may be performed by device 400. Method 800 applies block-based pre-processing. As shown in FIG. 8 , step 730 is further decomposed into two sub-steps 810 and 820. Other steps inherited from FIG. 7 will not be described again.

[0110]

[0128] 8, at step 810, the processor identifies one or more blocks in the input picture that do not overlap with the merged mask. Here, the one or more blocks may not necessarily be coding blocks used to calculate motion vectors. In some examples, the one or more blocks may be specifically designed to separate the instance and the background of the input picture. As a non-limiting example, the processor may apply a sliding window to the input picture to identify one or more blocks that do not overlap with the merged mask.

[0111]

[0129] As shown in FIG. 14A, the window can slide from the top left to the bottom right of the input picture. The window size can be, for example, 128×128, 192×192, or 256×256 depending on the desired granularity of preprocessing. The window slides through A11, A12, ..., A1n, A21, A22, ..., A2n, ... for pixel-by-pixel traversal. In other words, the sliding step is equal to the window width. In the following, the sliding window in some places is provided as block A11, block A12, ..., block A1n, block A21, A22, ..., block A2n, ... for simplicity.

[0112]

[0130] If any of the blocks fully or partially overlap with the merged mask 1330, it will be marked as an overlapping block and represented by filling it with solid black as shown in Figure 14b. For example, blocks A12, A13, and A14 in Figure 14A each partially overlap with the merged mask 1330, while block A54 completely overlaps with the merged mask 1330. Therefore, these blocks are overlapping blocks and are filled with solid black in Figure 14B. It should be noted that in Figure 14B, the merged mask 1330 is shown above the overlapping block 1410. Once the overlapping block 1410 has been determined, one or more blocks 1420 other than the overlapping block in the input picture (i.e., blocks that do not overlap with the merged mask) may also be identified.

[0113]

[0131] In some embodiments, the size of the sliding window may be determined based on a quantization parameter (QP) for compressing the input picture and / or the resolution of the input picture. When the QP for compressing the input picture is high or the resolution of the input picture is low, the representation cost is limited, and the instance region should be allocated more coding bits for better reconstruction quality. Therefore, the size of the sliding window may be smaller (e.g., 64×64). Conversely, when the QP for compressing the input picture is low or the resolution of the input picture is high, the size of the sliding window may be larger (e.g., 256×256).

[0114]

[0132] 8, in step 820, the processor extracts a region from the input picture that does not include one or more blocks 1420. In other words, the processor may extract a region that includes at least the overlapping block 1410. In a particular example, the region may consist of the overlapping block 1410. As one skilled in the art will appreciate, the region around the merged mask 1330 is also important for achieving better identification of the instance. The method 800 stretches the merged mask to obtain a region that consists at least of the overlapping block that covers the instance with a relatively large probability, so that mis-segmentation present in the merged mask can be mitigated.

[0115]

[0133] 9 is a flowchart of an exemplary image data compression method 900 based on method 700 shown in FIG. 7 , according to some embodiments of the present disclosure. For example, method 900 may be performed by one or more processors, such as image / video pre-processor 122 ( FIG. 1 ) and / or image / video encoder 124 ( FIG. 1 ). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into device 400 shown in FIG. 4 such that method 900 may be performed by device 400. Method 900 applies extension-based pre-processing. As shown in FIG. 9 , step 730 is further decomposed into two sub-steps 910 and 920. Other steps inherited from FIG. 7 will not be described again.

[0116]

[0134] 9, at step 910, the processor dilates the merged mask to determine a first region that extends from a boundary of the merged mask in the input picture. As a non-limiting example, the processor may apply a kernel to the merged mask to determine the first region that extends from a boundary of the merged mask.

[0117]

[0135] As shown in FIGS. 15A-15D , in some embodiments, a kernel having the shape of kernel 1530, kernel 1540, or kernel 1550 may be used to dilate the merged mask to determine a first region 1510 extending from a boundary of the merged mask (not shown for simplicity). In a non-limiting example, kernel 153 having a size of 7×7 may be applied for the dilation. The dilation may be performed by kernel 153 on the merged mask to stretch the predicted instance region, for example, through three iterations. In each iteration, kernel 153 is applied to the input picture for the dilation on a pixel-by-pixel basis. Because the kernel size is smaller than the sliding window, dilation-based preprocessing may have better morphological performance than block-based preprocessing, while its computational load and complexity may be higher.

[0118]

[0136] In some embodiments, the size of the kernel may be determined based on a quantization parameter (QP) for compressing the input picture and / or the resolution of the input picture. If the QP for compressing the input picture is high or the resolution of the input picture is low, the representation cost is limited, and the instance region should be allocated more coding bits for better reconstruction quality. Therefore, the size of the kernel may be smaller (e.g., 3×3). Conversely, if the QP for compressing the input picture is low or the resolution of the input picture is high, the size of the kernel may be larger (e.g., 7×7).

[0119]

[0137] Referring back to Figure 9, in step 920, the processor extracts a first region and an optional dilated mask from the input picture. As can be seen from Figure 15A, a first region 1510 is extracted from the input picture by dilation, while a remaining region 1520 in the input picture other than the first region 1510 can be omitted from subsequent processing. As one skilled in the art can appreciate, the region surrounding the merged mask is also important for achieving better identification of instances. Method 900 dilates the merged mask to obtain a first region that has a relatively greater likelihood of covering the instance, so that mis-segmentation present in the merged mask can be mitigated.

[0120]

[0138] FIG. 10 is a flowchart of an exemplary image data compression method 1000 based on method 700 shown in FIG. 7 , according to some embodiments of the present disclosure. For example, method 1000 may be performed by one or more processors, such as image / video pre-processor 122 ( FIG. 1 ) and / or image / video encoder 124 ( FIG. 1 ). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into device 400 shown in FIG. 4 , such that method 1000 may be performed by device 400. Method 1000 simultaneously applies extension-based pre-processing along with block-based pre-processing. As shown in FIG. 10 , step 730 is further decomposed into three sub-steps 1010, 1020, and 1030. Other steps inherited from FIG. 7 will not be described again.

[0121]

[0139] 10, in step 1010, the processor dilates the merged mask to determine a first region extending from the boundary of the merged mask. The dilation procedure in step 1010 may follow the process described in FIG. 9 and its corresponding embodiments (not described here for brevity).

[0122]

[0140] Next, in step 1020, the processor identifies one or more blocks in the input picture that do not overlap with the first region or merged mask. Similar to the method described with respect to Figure 8, the processor may apply a sliding window to the input picture to identify one or more blocks that do not overlap with the first region or merged mask.

[0123]

[0141] For example, a window may slide from the top left to the bottom right of the input picture for pixel-by-pixel traversal. The window size may be, for example, 128x128, 192x192, or 256x256, with the sliding step being the width of the window. If any of the blocks overlaps, in whole or in part, with the first region or the merged mask, it will be marked as an overlapping block. When the overlapping blocks are determined, one or more blocks other than the overlapping block in the input picture (i.e., blocks that do not overlap with the first region or the merged mask) may be identified as well.

[0124]

[0142] 10, in step 1030, the processor extracts regions from the input picture that do not contain one or more blocks. In other words, the processor extracts regions that contain at least overlapping blocks. In a particular example, the regions may consist of overlapping blocks. As those skilled in the art will appreciate, the area around the merged mask is also important to achieve better instance identification.

[0125]

[0143] Method 1000 stretches the merged mask to obtain a region consisting of at least overlapping blocks that cover the instance with a relatively large probability, so that mis-segmentation present in the merged mask can be mitigated. Compared with methods 800 and 900, method 1000 simultaneously applies expansion-based preprocessing and block-based preprocessing, where, in terms of processing operators (i.e., sliding windows or kernels), block-based preprocessing is performed on a relatively large scale, while expansion-based preprocessing is performed on a relatively small scale. Therefore, method 1000 has better morphological performance than methods 800 or 900, which simply employ block-based preprocessing or expansion-based preprocessing. Here, the sliding window size may be smaller (e.g., 64×64 or 128×128) compared to method 800.

[0126]

[0144] 11 is a flowchart of an exemplary image data compression method 1100 based on method 700 shown in FIG. 7 , according to some embodiments of the present disclosure. For example, method 1100 may be performed by one or more processors, such as image / video pre-processor 122 ( FIG. 1 ) and / or image / video encoder 124 ( FIG. 1 ). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into device 400 shown in FIG. 4 , such that method 1100 may be performed by device 400. Method 1100 applies blur pre-processing. As shown in FIG. 11 , step 730 is further decomposed into two sub-steps 1110 and 1120. Other steps inherited from FIG. 7 will not be described again.

[0127]

[0145] Referring to FIG. 11 , in step 1110, the processor determines a second region within the extracted region that does not overlap with the merged mask. As shown in FIG. 14A , a window may slide from the top left to the bottom right of the input picture. If any of the blocks fully or partially overlap with the merged mask 1330, it will be marked as an overlapping block and represented by filling it with solid black, as shown in FIG. 14B . Note that the merged mask 1330 is shown above the overlapping block 1410. When the overlapping block 1410 is determined, one or more blocks 1420 other than the overlapping block in the input picture may also be identified. The one or more blocks 1420 may be used to form a second region. For example, the processor may determine a second region made up of some or all of the one or more blocks 1420.

[0128]

[0146] 11 , in step 1120, the processor blurs the second region. In some embodiments, the background portion of the input image is not simply ignored. Blur preprocessing may be applied to the second region to improve the smoothness and representational compactness of the image data.

[0129]

[0147] In another example, blur preprocessing may be applied together with block and / or dilation preprocessing. To prevent blurring of the merged mask region (where the object is located) from degrading the performance or accuracy of the machine vision task (e.g., object recognition or tracking), the blur preprocessing output I blur But, for example, I blur =G k (M bd -M)I+M I, where I is the original image, M is the merged mask, and M bd is the mask after block and / or dilation-based preprocessing, and G kis a Gaussian filter with kernel size k. In this way, in block and / or dilation preprocessed images, blur preprocessing is applied only to image regions outside (i.e., not overlapping with) the merged mask, thereby improving compression efficiency without degrading machine analysis performance.

[0130]

[0148] FIG. 12 is a flowchart of an exemplary image data compression method 1200 based on method 700 shown in FIG. 7 , according to some embodiments of the present disclosure. For example, method 1200 may be performed by one or more processors, such as image / video pre-processor 122 ( FIG. 1 ) and / or image / video encoder 124 ( FIG. 1 ). In some embodiments, image / video pre-processor 122 and image / video encoder 124 may be incorporated into device 400 shown in FIG. 4 , such that method 1200 may be performed by device 400. Method 1200 determines whether to perform blur pre-processing based on a quantization parameter (QP). As shown in FIG. 12 , step 730 is further decomposed into two sub-steps 1210 and 1220. Other steps inherited from FIG. 7 will not be described again.

[0131]

[0149] Referring to Figure 12, in step 1210, the processor determines whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture. To further compress the input picture, a portion of the extracted region may be blurred in some way. For example, if the input picture is to be compressed with low quality (higher QP), at least a portion of the extracted region may be blurred. However, if the input picture is to be quantized with more stages (lower QP), the extracted region may not need to be blurred. Blur preprocessing applied in the case of more quantization stages may dramatically reduce the quality of the encoded picture and may not be suitable for machine vision.

[0132]

[0150] Still referring to FIG. 12 , in step 1210, the processor blurs a portion of the extracted region according to a QP greater than a predetermined threshold. Furthermore, blur preprocessing may also be applied to the output of block and / or extension-based preprocessing to further improve compression efficiency. However, the processor may not blur any portion of the extracted region if the QP is less than a predetermined threshold. In some embodiments of the present disclosure, a segment mask or a merged mask may be applied to the input picture to extract an instance within the input picture. As those skilled in the art will understand, for example, when a merged mask is applied to the input picture, a portion of the picture (where the instance exists) will be covered by the merged mask. Any region of the picture that includes an area covered by the merged mask may be referred to as a merged mask-containing region (this region essentially exists in the present disclosure without any additional instructions herein). Or, in other words, the expression "extracting a merged mask-containing region" may also be reproduced as "extracting an area within the input picture that is covered by the applied merged mask." These two expressions possess the same technical contribution to the present disclosure.

[0133]

[0151] 16 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. For example, method 1600 may be performed by one or more processors, such as image / video decoder 144 (FIG. 1). In some embodiments, image / video decoder 144 may be incorporated into device 400 shown in FIG. 4, such that method 1600 may be performed by device 400. Method 1600 may decode a bitstream that uses any of the encoding schemes described above (e.g., the decoding process shown in FIGS. 3A and 3B). As shown in FIG. 16, method 1600 includes the following steps:

[0134]

[0152] In step 1610, the processor receives a bitstream including compressed image data with region information. The compressed image data includes encoded information representing one or more regions extracted from a picture. The process of pre-processing and encoding the extracted regions has been described in the above embodiments. The region information indicates the location(s) of the one or more extracted regions within the picture.

[0135]

[0153] In step 1620, the processor reconstructs a picture by decoding the compressed image data according to the region information. The process for decoding a bitstream was described above in connection with Figures 3A and 3B and is incorporated herein by reference. Because regions other than the extracted region may be blank and no coding bits are assigned to them, the processor may skip this region whenever decoding the compressed image to obtain a picture. In this regard, there is no reference relationship between pixels in the extracted region (indicated by the region information) and pixels in regions other than the extracted region (which may be extrapolated from the region information because the picture size is fixed). The decoder may focus on decoding the compressed image data within the extracted region. Therefore, the decoding pressure of the decoder is reduced compared to decoding the entire picture.

[0136]

[0154] As shown in FIG. 17, the step 1620 of reconstructing a picture by decoding compressed image data according to region information can be further decomposed into two steps 1710 , 1720 and an optional step 1730 .

[0137]

[0155] At step 1710, the processor may generate a first region of the picture based on the decoded image data. The first region is the extracted region that the decoded image data represents. As described above, decoded pixel values ​​from the decoded image data may be used to fill the first region.

[0138]

[0156] In step 1720, the processor may generate a second region of the picture by filling pixels, where the second region is at least a portion of a region other than the first region. As one example, the processor may composite the first region and the second region to form the entire picture. As another example, the processor may ignore the second region, so that the second region is transparent when forming the picture. As another example, the processor may fill the second region in a preset manner. For example, the processor may fill the second region of the picture with "black" or "white" pixels, so that the second region exists as the background of the picture. Here, black pixels are pixels with RGB values ​​(0,0,0), while white pixels are pixels with RGB values ​​(255,255,255). In yet another example, the processor may fill the second region with an existing background (e.g., scenery). For example, the processor may select a scenery picture and then stretch or shrink it to the size of the picture to be decoded. The processor then matches (keys) the scenery picture according to the region information. The remaining portion of the scenery picture after matching can be used as a background to form a second region of the picture. Finally, the processor can merge the first and second regions to form a complete picture.

[0139]

[0157] As an optional step, in step 1730, the processor may restore the location of the instance in the picture according to the region information. In some examples, the decoded picture may be used for other processing (e.g., object identification). The processor may also restore the location of the instance. As a priori assumption, the first region (i.e., the extracted region) may cover one or more instances. By restoring the location of the first region, the location of the instance(s) can be known. Therefore, in subsequent processing (e.g., object identification), emphasis may be placed on the restored location.

[0140]

[0158] It should be noted that the region information described in connection with Figures 16 and 17 is not required to decode the compressed video information. Consistent with the disclosed embodiments, any decision scheme can be used to decode the compressed video information based on the above-described method. For example, the compressed video information can be decoded by a conventional decoder without any special modifications.

[0141]

[0159] It will be understood that some embodiments of the present disclosure may be combined with another embodiment or with some other embodiments.

[0142]

[0160] In some embodiments, a non-transitory computer-readable storage medium storing one or more bitstreams processed according to the above-described methods is also provided. For example, the one or more bitstreams may be generated according to methods 700-1200 (FIGS. 7-12). Specifically, according to some embodiments, the one or more bitstreams include syntax segments encoded by an encoder (e.g., image / video encoder 122 of FIG. 1). For example, the one or more bitstreams may be bitstream 228 encoded according to the encoding process in FIGS. 2A and 2B. Prior to encoding, input pictures of the encoder may be pre-analyzed and pre-processed by a preprocessor (e.g., image / video preprocessor 122 of FIG. 1) according to solutions provided by this disclosure (e.g., the processes described in connection with FIGS. 7-12). Consistent with the disclosed embodiments, the one or more bitstreams may also be decoded by a decoder (e.g., image / video decoder 144 of FIG. 1). For example, the one or more bitstreams may be bitstream 228 decoded according to the decoding process of FIGS. 3A and 3B. Furthermore, the decoded information may be used to generate a reconstructed picture according to the solutions provided by this disclosure (eg, the processes described in connection with FIGS. 16 and 17).

[0143]

[0161] In some embodiments, a non-transitory computer-readable storage medium containing instructions is also provided, and the instructions may be executed by a device (such as the disclosed encoders and decoders) to perform the above-described methods. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cartridge, and networked versions thereof. The device may include one or more processors (CPUs), input / output interfaces, a network interface, and / or memory.

[0144]

[0162] In some embodiments, a computer program product is provided that includes computer program instructions, the computer program instructions enabling a computer to perform the steps of the methods described in any of the embodiments of the present disclosure.

[0145]

[0163] In some embodiments, a computer program is provided that enables a computer to perform the steps of the methods described in any of the embodiments of the present disclosure.

[0146]

[0164] Some embodiments may be further described using the following clauses: 1. A method of processing image data, comprising receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, wherein the encoding comprises: performing instance segmentation of the input pictures to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region. 2. The method of clause 1, wherein the extracted region includes one or more pixels adjacent to the merged mask. 3. The method of clause 1, wherein extracting a region from the input picture that includes the merged mask includes: identifying one or more blocks in the input picture that do not overlap with the merged mask; and extracting a region from the input picture that does not include one or more blocks. 4. The method of clause 3, wherein identifying one or more blocks is performed by using a sliding window, and the size of the sliding window is determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 5. The method of clause 1, wherein extracting a region including the merged mask from the input picture includes: dilating the merged mask to determine a first region extending from a boundary of the merged mask; and extracting the first region and the dilated mask from the input picture. 6. The method of clause 5, wherein expanding the merged mask is performed by applying a kernel to the merged mask, and the size of the kernel is determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 7. The method of clause 1, wherein extracting a region including the merged mask from the input picture includes: extending the merged mask to determine a first region extending from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and extracting a region from the input picture that does not include one or more blocks. 8. The method of clause 1, wherein extracting a region including the merged mask from the input picture further includes: determining a second region within the extracted region that does not overlap with the merged mask; and blurring the second region. 9. The method of clause 1, wherein extracting a region including the merged mask from the input picture further includes: determining whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture; and blurring a portion of the extracted region in response to a QP greater than a predetermined threshold. 10. The method of clause 9, wherein no part of the extracted region overlaps with the merged mask. 11. A non-transitory computer-readable storage medium storing a bitstream generated by a method including: performing instance segmentation of an input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region to generate the bitstream. 12. The media of clause 11, wherein the extracted region includes one or more pixels adjacent to the merged mask. 13. The media of clause 11, wherein extracting a region from the input picture that includes the merged mask includes: identifying one or more blocks in the input picture that do not overlap with the merged mask; and extracting a region from the input picture that does not include the one or more blocks. 14. The medium of clause 13, wherein identifying one or more blocks is performed by using a sliding window, the size of the sliding window being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 15. The media described in clause 11, wherein extracting a region including the merged mask from the input picture includes: expanding the merged mask to determine a first region that extends from a boundary of the merged mask; and extracting the first region and the merged mask from the input picture. 16. The medium of clause 15, wherein expanding the merged mask is performed by applying a kernel to the merged mask, the size of the kernel being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 17. The media described in clause 11, wherein extracting a region including the merged mask from the input picture includes: extending the merged mask to determine a first region that extends from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and extracting a region from the input picture that does not include one or more blocks. 18. The medium of clause 11, wherein extracting a region including the merged mask from the input picture further includes: determining a second region within the extracted region that does not overlap with the merged mask; and blurring the second region. 19. The medium of clause 11, wherein extracting a region including the merged mask from the input picture further includes: determining whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture; and blurring the portion of the extracted region in response to a QP greater than a predetermined threshold. 20. The medium of clause 19, wherein a portion of the extracted region does not overlap with the merged mask. 21. An image data processing apparatus including: a memory storing a set of instructions; and one or more processors, the one or more processors causing the image data processing apparatus to: performing instance segmentation of the input picture to generate one or more segment masks; combining one or more segment masks to generate a merged mask; Extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region An apparatus configured to execute a set of instructions to cause an operation including: 22. The apparatus of clause 21, wherein the extracted region includes one or more pixels adjacent to the merged mask. 23. The apparatus of clause 21, wherein extracting a region from the input picture that includes the merged mask includes: identifying one or more blocks in the input picture that do not overlap with the merged mask; and extracting a region from the input picture that does not include one or more blocks. 24. The apparatus described in clause 23, wherein identifying one or more blocks by using a sliding window and a size of the sliding window is determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 25. The apparatus described in clause 21, wherein extracting a region including the merged mask from the input picture includes: dilating the merged mask to determine a first region extending from a boundary of the merged mask; and extracting the first region and the dilated mask from the input picture. 26. The apparatus described in clause 25, wherein applying a kernel to the merged mask and expanding the merged mask by the size of the kernel are determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 27. The apparatus of clause 21, wherein extracting a region from the input picture that includes the merged mask includes: extending the merged mask to determine a first region that extends from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and extracting a region from the input picture that does not include one or more blocks. 28. The apparatus of clause 21, wherein extracting a region including the merged mask from the input picture further includes: determining a second region within the extracted region that does not overlap with the merged mask; and blurring the second region. 29. The apparatus of clause 21, wherein extracting a region including the merged mask from the input picture further includes: determining whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture; and blurring the portion of the extracted region in response to a QP greater than a predetermined threshold. 30. The apparatus of clause 29, wherein a portion of the extracted region does not overlap with the merged mask. 31. A non-transitory computer-readable storage medium storing a set of instructions that, when executed by a computer, cause the computer to: performing instance segmentation of the input picture to generate one or more segment masks; combining one or more segment masks to generate a merged mask; Extracting a region from the input picture that includes the merged mask; and A non-transitory computer-readable storage medium that causes a method to be performed, the method including compressing image data representing the extracted regions to generate a bitstream. 32. The medium of clause 31, wherein the extracted region includes one or more pixels adjacent to the merged mask. 33. The medium of clause 31, wherein extracting a region from the input picture that includes the merged mask includes: identifying one or more blocks in the input picture that do not overlap with the merged mask; and extracting a region from the input picture that does not include the one or more blocks. 34. The medium of clause 33, wherein identifying one or more blocks is performed by using a sliding window, the size of the sliding window being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 35. The media described in clause 31, wherein extracting a region including the merged mask from the input picture includes: expanding the merged mask to determine a first region that extends from a boundary of the merged mask; and extracting the first region and the merged mask from the input picture. 36. The medium of clause 35, wherein expanding the merged mask is performed by applying a kernel to the merged mask, the size of the kernel being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture. 37. The media of clause 31, wherein extracting a region from the input picture that includes the merged mask includes: extending the merged mask to determine a first region that extends from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and extracting a region from the input picture that does not include one or more blocks. 38. The media of clause 31, wherein extracting a region from the input picture that includes the merged mask further includes: determining a second region within the extracted region that does not overlap with the merged mask; and blurring the second region. 39. The medium of clause 31, wherein extracting a region including the merged mask from the input picture further includes: determining whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture; and blurring the portion of the extracted region in response to a QP greater than a predetermined threshold. 40. The medium of clause 39, wherein a portion of the extracted region does not overlap with the merged mask. 41. A method of processing image data, comprising: receiving a bitstream; and decoding one or more pictures by using encoded information in the bitstream, wherein the decoding comprises: receiving a bitstream containing compressed image data with region information, the region information indicating a region that the compressed image data represents within the picture; and reconstructing the picture by decoding the compressed image data in accordance with the region information. 42. The method of clause 41, wherein reconstructing a picture by decoding compressed image data according to region information includes: generating a first region of the picture based on the decoded image data; and generating a second region of the picture by filling in pixels. 43. The method of clause 41, wherein reconstructing the picture by decoding the compressed image data according to the region information further comprises: restoring the location of the instance within the picture according to the region information. 44. A non-transitory computer-readable storage medium storing a bitstream including compressed image data with region information, the region information indicating a region that the compressed image data represents within a picture, and the bitstream being processed by a method including reconstructing the picture by decoding the compressed image data in accordance with the region information. 45. The medium of clause 44, wherein reconstructing a picture by decoding compressed image data according to region information includes: generating a first region of the picture based on the decoded image data; and generating a second region of the picture by filling in pixels. 46. ​​The medium of clause 44, wherein reconstructing the picture by decoding the compressed image data according to the region information further comprises: restoring the location of the instance within the picture according to the region information. 47. An image data processing device including: a memory storing a set of instructions; and one or more processors, the one or more processors causing the image data processing device to: receiving a bitstream including compressed image data with region information, the region information indicating a region that the compressed image data represents within a picture; and Reconstructing a picture by decoding compressed image data according to region information A processing device configured to execute a set of instructions to cause operations including: 48. The apparatus of clause 47, wherein reconstructing a picture by decoding compressed image data according to region information includes: generating a first region of the picture based on the decoded image data; and generating a second region of the picture by filling in pixels. 49. The apparatus of clause 47, wherein reconstructing the picture by decoding the compressed image data according to the region information further comprises: restoring the location of the instance within the picture according to the region information. 50. A non-transitory computer-readable storage medium storing a set of instructions that, when executed by a computer, cause the computer to: receiving a bitstream including compressed image data with region information, the region information indicating a region that the compressed image data represents within a picture; and Reconstructing a picture by decoding compressed image data according to region information A non-transitory computer-readable storage medium for causing a method including: 51. The medium of clause 50, wherein reconstructing a picture by decoding compressed image data according to region information includes: generating a first region of the picture based on the decoded image data; and generating a second region of the picture by filling in pixels. 52. The medium of clause 50, wherein reconstructing the picture by decoding the compressed image data according to the region information further comprises: restoring the location of the instance within the picture according to the region information.

[0147]

[0165] It should be noted that relational terms herein, such as "first" and "second," are used only to distinguish one entity or operation from another, and thus do not require or imply any actual relationship or order between those entities or operations. Furthermore, the words "comprising," "having," "containing," and "including," and other similar forms, are intended to be equivalents in meaning, and the item or items following any of these words are intended to be open-ended in that they do not imply an exclusive listing of such items or items or that the item or items are not limited to the listed item or items.

[0148]

[0166] As used herein, unless otherwise stated, the term "or" includes all possible combinations unless impracticable. For example, if it is stated that a database may include A or B, the database may include A, B, A and B unless otherwise stated or impracticable. As a second example, if it is stated that a database may include A, B, or C, the database may include A, B, C, A and B, A and C, B and C, A and B and C unless otherwise stated or impracticable.

[0149]

[0167] It is understood that the above-described embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, the software can be stored in the above-described computer-readable medium. The software, when executed by a processor, can perform the disclosed methods. The computing units and other functional units described in this disclosure can be implemented by hardware, or software, or a combination of hardware and software. Those skilled in the art will also understand that multiple of the above-described modules / units can be combined into one module / unit, and that each of the above-described modules / units can be further divided into multiple sub-modules / sub-units.

[0150]

[0168] In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Adaptations and modifications of some of the above-described embodiments may be made. Other embodiments may become apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. It is intended that the specification and examples be considered merely exemplary, with the true scope and spirit of the present disclosure being indicated by the appended claims. It is also intended that the sequence of steps depicted in the figures is for illustrative purposes only, and thus is not intended to be limited to any particular sequence of steps. Thus, one skilled in the art will appreciate that these steps may be performed in different orders while performing the same method.

[0151]

[0169] Illustrative embodiments have been disclosed in the accompanying drawings and herein. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. receiving a video sequence; encoding one or more input pictures associated with the video sequence; and 1. A method of processing image data, comprising generating a bitstream, said encoding comprising: performing instance segmentation of the input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing the extracted region.

2. The method of claim 1 , wherein the extracted region includes one or more pixels adjacent to the merged mask.

3. Extracting the region containing the merged mask from the input picture comprises: identifying one or more blocks in the input picture that do not overlap with the merged mask; and 3. The method of claim 1, further comprising extracting a region from the input picture that does not contain the one or more blocks.

4. 4. The method of claim 3, wherein identifying the one or more blocks is performed using a sliding window, the size of the sliding window being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture.

5. Extracting the region containing the merged mask from the input picture comprises: dilating the merged mask to determine a first region extending from a boundary of the merged mask; and The method of claim 1 or 2, comprising extracting the first region and the dilated mask from the input picture.

6. 6. The method of claim 5, wherein expanding the merged mask is performed by applying a kernel to the merged mask, the size of the kernel being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture.

7. Extracting the region containing the merged mask from the input picture comprises: dilating the merged mask to determine a first region extending from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and 3. The method of claim 1, further comprising extracting a region from the input picture that does not contain the one or more blocks.

8. 3. The method of claim 1, wherein extracting the region containing the merged mask from the input picture further comprises: determining a second region within the extracted region that does not overlap with the merged mask; and blurring the second region.

9. Extracting the region containing the merged mask from the input picture further comprises: determining whether to blur at least a portion of the extracted region based on a quantization parameter (QP) for compressing the input picture; and blurring the portion of the extracted region in response to the QP being greater than a predetermined threshold.

3. The method of claim 1 or 2, comprising:

10. The method of claim 9 , wherein the portion of the extracted region does not overlap with the merged mask.

11. The method according to any one of claims 1 to 10, wherein the instance segmentation is performed by a convolutional neural network (CNN).

12. 1. A non-transitory computer-readable storage medium storing a bitstream generated by a method, the method comprising: performing instance segmentation of the input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing the image data representing the extracted region to generate a bitstream.

1. A non-transitory computer-readable storage medium comprising:

13. The medium of claim 12 , wherein the extracted region includes one or more pixels adjacent to the merged mask.

14. Extracting the region containing the merged mask from the input picture comprises: identifying one or more blocks in the input picture that do not overlap with the merged mask; and 14. A medium as claimed in claim 12 or 13, comprising extracting a region from the input picture that does not contain the one or more blocks.

15. 15. The medium of claim 14, wherein identifying the one or more blocks is performed using a sliding window, the size of the sliding window being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture.

16. Extracting the region containing the merged mask from the input picture comprises: dilating the merged mask to determine a first region extending from a boundary of the merged mask; and The medium of claim 12 or 13, comprising extracting the first region and the merged mask from the input picture.

17. 17. The medium of claim 16, wherein expanding the merged mask is performed by applying a kernel to the merged mask, the size of the kernel being determined based on at least one of a quantization parameter (QP) for compressing the input picture or a resolution of the input picture.

18. Extracting the region containing the merged mask from the input picture comprises: dilating the merged mask to determine a first region extending from a boundary of the merged mask; identifying one or more blocks in the input picture that do not overlap with the first region or the merged mask; and 14. A medium as claimed in claim 12 or 13, comprising extracting a region from the input picture that does not contain the one or more blocks.

19. Extracting the region containing the merged mask from the input picture further comprises: determining a second region within the extracted region that does not overlap with the merged mask; and 14. The medium of claim 12 or 13, comprising blurring the second region.

20. 1. An image data processing apparatus comprising: a memory storing a set of instructions; and one or more processors, the one or more processors causing the image data processing apparatus to: performing instance segmentation of the input picture to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting a region from the input picture that includes the merged mask; and compressing image data representing said extracted region; an apparatus configured to execute said set of instructions to cause an apparatus to perform operations including:

21. A computer program product comprising computer program instructions enabling a computer to carry out the method according to any one of claims 1 to 11.

22. A computer program enabling a computer to carry out the method according to any one of claims 1 to 11.

Images