Image compression method based on pre-analysis

Abstract

The present disclosure provides a pre-analysis based method for adaptively compressing image data processed by machine vision tasks. An exemplary method includes receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream wherein the encoding comprises: performing an instance segmentation on an input picture to generate one or more segmentation masks; combining the one or more split masks to generate a combined mask; extracting a region containing the combined mask from the input picture; and compressing image data characterizing the extraction region.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims priority to U.S. Provisional Application No. 63 / 378,888, filed on October 10, 2022, and priority to U.S. Patent Application No. 18 / 478,680, filed on September 29, 2023, entitled “Pre-Analysis-Based Image Compression Method,” and the entire contents of the foregoing two applications are incorporated into this application by reference. Technical Field

[0002] The present disclosure relates generally to video processing and, more particularly, to methods and apparatus for performing pre-analysis based image / video compression for machine vision tasks. Background Art

[0003] With the rise of machine learning technology and machine vision applications, the amount of videos and images processed by machines (collectively referred to as "image data") has grown rapidly. Typical use cases include autonomous driving, smart transportation, smart cities, smart content management, etc., which include machine vision tasks such as object detection, instance segmentation, and object tracking.

[0004] Since machine vision tasks require a large amount of image data, the image data must be compressed for efficient transmission and storage. However, traditional image / video compression technology has been focused on ensuring image / video quality perceived by humans, while machines process and understand visual data differently from human vision. Therefore, image / video compression technology adapted to machine vision may be different from traditional technology. Therefore, new compression technology is needed to achieve performance optimization for machine use. Summary of the invention

[0005] The present disclosure provides a method for adaptively compressing image data processed by a machine vision task based on pre-analysis. In some embodiments, an exemplary image data processing method includes: receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, wherein the encoding includes: performing instance segmentation on an input picture to generate one or more segmentation masks; combining one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input picture; and compressing image data representing the extracted region.

[0006] In some embodiments, a non-transitory computer-readable storage medium stores a bitstream generated by the following method: performing instance segmentation on an input picture to generate one or more segmentation masks; combining one or more segmentation masks to generate a merge mask; extracting a region including the merge mask from the input picture; and compressing image data representing the extracted region to generate the bitstream.

[0007] In some embodiments, an image data processing device includes: a memory storing an instruction set; and one or more processors configured to execute the instruction set to cause the device to perform operations including: performing instance segmentation on an input image to generate one or more segmentation masks; combining one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region.

[0008] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, causes the computer to perform a method comprising: performing instance segmentation on an input image to generate one or more segmentation masks; combining one or more segmentation masks to generate a merge mask; extracting a region containing the merge mask from the input image; and compressing image data representing the extracted region to generate a bitstream.

[0009] In some embodiments, an image data processing method includes: receiving a bitstream; and decoding one or more pictures using the encoded information of the bitstream, wherein the decoding includes: receiving a bitstream, the bitstream including compressed image data with region information, wherein the region information indicates the region represented by the compressed image data in a picture; and reconstructing the picture by decoding the compressed image data according to the region information.

[0010] In some embodiments, a non-transitory computer-readable storage medium stores a bitstream, the bitstream including compressed image data with region information, wherein the region information indicates a region represented by the compressed image data in a picture, and the bitstream is processed by a method comprising: decoding the compressed image data according to the region information to reconstruct the picture.

[0011] In some embodiments, an image data processing device includes: a memory storing an instruction set; and one or more processors configured to execute the instruction set so that the device performs operations, the operations including: receiving a bit stream including compressed image data with region information, wherein the region information indicates the region represented by the compressed image data in a picture; and decoding the compressed image data according to the region information to reconstruct the picture.

[0012] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions that, when executed by a computer, causes the computer to perform a method, including: receiving a bitstream, the bitstream including compressed image data with region information, wherein the region information indicates the region represented by the compressed image data in a picture; and decoding the compressed image data according to the region information to reconstruct the picture.

[0013] In some embodiments, a computer program product is provided, the computer program product comprising: computer program instructions, and the computer program instructions enable a computer to execute the image data processing method according to the above-mentioned image data processing method embodiment.

[0014] In some embodiments, a computer program is provided, which enables a computer to execute the image data processing method according to the above-mentioned image data processing method embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments and aspects of the present disclosure are shown in the following detailed description and accompanying drawings. The various features shown in the drawings are not drawn to scale.

[0016] Figure 1 A schematic diagram of an exemplary system for preprocessing and encoding image data according to some embodiments of the present disclosure is shown.

[0017] Figure 2A A schematic diagram showing an exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present disclosure is shown.

[0018] Figure 2B A schematic diagram showing another exemplary encoding process of a hybrid video encoding system consistent with an embodiment of the present disclosure is shown.

[0019] Figure 3A A schematic diagram showing an exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present disclosure is shown.

[0020] Figure 3B A schematic diagram showing another exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present disclosure is shown.

[0021] Figure 4 is a block diagram showing an exemplary apparatus for preprocessing or encoding image data according to some embodiments of the present disclosure.

[0022] Figure 5 A schematic diagram of an exemplary compression method for visual data compression for machine vision according to some embodiments of the present disclosure is shown.

[0023] Figure 6A-6E schematically illustrates various methods for pre-processing an input image based on pre-analysis results according to some embodiments of the present disclosure.

[0024] Figure 7 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0025] Figure 8 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0026] Fig. 9 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0027] Fig.10 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0028] Fig.11 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0029] Fig.12 is a flowchart of an exemplary image data compression method according to some embodiments of the present disclosure.

[0030] Fig.13 A-13D schematically illustrates pre-processing of an input image based on a pre-analysis result according to some embodiments of the present disclosure.

[0031] Fig.14 A and Fig.14 B schematically shows that according to some embodiments of the present disclosure, an input picture is preprocessed based on block preprocessing.

[0032] Fig.15 A-15D schematically illustrates preprocessing an input image based on dilation preprocessing according to some embodiments of the present disclosure.

[0033] Fig.16 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure.

[0034] Fig.17 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings. The following description refers to the accompanying drawings, wherein the same reference numerals in different drawings represent the same or similar elements unless otherwise specified. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with the aspects related to the present disclosure as recorded in the attached claims. Specific aspects of the present disclosure are described in more detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0036] The present disclosure relates to machine video coding (VCM), which is intended to compress input videos and images or feature maps for machine vision tasks. Specifically, according to an exemplary embodiment, before encoding, the input picture is pre-analyzed to identify a segmentation mask corresponding to an object or foreground in the input picture. The segmentation mask is further pre-processed to form a merge mask, which is then encoded (i.e., compressed). In some embodiments, the pre-processing can also identify an extended area around the merge mask. The extended area is also compressed by the encoder.

[0037] Consistent with the disclosed embodiments, the pre-analysis and pre-processing may be performed by an image data pre-processor separate from the encoder. Alternatively, the pre-analysis may be performed by the encoder itself. The present disclosure does not limit the hardware or software architecture used to implement the image data pre-analysis.

[0038] The disclosed technology is applicable to compressing image data used by any machine vision task, such as object recognition and tracking, facial recognition, image / video search, mobile augmented reality (MAR), autonomous vehicles, Internet of Things (IoT), image matching, 3D structure construction, stereo correspondence, motion tracking, etc.

[0039] Figure 1 A block diagram of a system 100 for preprocessing and encoding image data according to some disclosed embodiments is shown. The image data may include images (also referred to as "pictures" or "frames"), multiple images, or videos. Images are static pictures. Multiple images may be related or unrelated in space or time. A video is a set of images arranged in time order.

[0040] like Figure 1As shown, system 100 includes a source device 120 that provides encoded video data that is subsequently decoded by a target device 140. Consistent with the disclosed embodiments, each of source device 120 and target device 140 may include any of a variety of devices, including: a desktop computer, a notebook (e.g., laptop) computer, a server, a tablet, a set-top box, a mobile phone, a vehicle, a camera, an image sensor, a robot, a television, a wearable device (e.g., a smart watch or a wearable camera), a display device, a digital media player, a video game console, a video streaming device, and the like. Source device 120 and target device 140 may be configured for wireless or wired communication.

[0041] refer to Figure 1 , the source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. The target device 140 may include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. The image / video preprocessor 122 preprocesses image data, i.e., one or more images or one or more videos, and generates an input bit stream for the image / video encoder 124. The image / video encoder 124 encodes the input bit stream and outputs an encoded bit stream 162 via the output interface 126. The encoded bit stream 162 is sent via the communication medium 160 and is received by the input interface 142. The image / video decoder 144 then decodes the encoded bit stream 162 to generate decoded data, which can be utilized by the machine vision application 146.

[0042] More specifically, the source device 120 may further include various devices (not shown) for providing source image data to be preprocessed by the image / video preprocessor 122. The device for providing source image data may include an image / video capture device, such as a camera, an image / video archiving or storage device containing previously captured images / videos, or an image / video feed interface for receiving images / videos from an image / video content provider.

[0043] The image / video encoder 124 and the image / video decoder 144 can each be implemented in any of a variety of suitable encoder or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When encoding or decoding is partially implemented in software, the image / video encoder 124 or the image / video decoder 144 can store multiple instructions for the software in a suitable, non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform techniques consistent with the present disclosure. Each of the image / video encoder 124 or the image / video decoder 144 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (CODEC) in a corresponding device.

[0044] The image / video encoder 124 and the image / video decoder 144 may operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc. Alternatively, the image / video encoder 124 and the image / video decoder 144 may be custom devices that do not conform to existing standards. Figure 1 Not shown, but in some embodiments, the image / video encoder 124 and the image / video decoder 144 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units (MUX-DEMUX), or other hardware and software, to handle the encoding of audio and video in a common data stream or in separate data streams.

[0045] Output interface 126 may include any type of medium or device capable of transmitting encoded bitstream 162 from source device 120 to target device 140. For example, output interface 126 may include a transmitter or transceiver configured to transmit encoded bitstream 162 directly from source device 120 to target device 140 in real-time. Encoded bitstream 162 may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to target device 140.

[0046] The communication medium 160 may include a transient medium, such as a wireless broadcast or a wired network transmission. For example, the communication medium 160 may include a radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). The communication medium 160 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. In some embodiments, the communication medium 160 may include a router, a switch, a base station, or any other device that can be used to facilitate communication from the source device 120 to the target device 140. For example, a network server (not shown) may receive an encoded bit stream 162 from the source device 120 and provide the encoded bit stream 162 to the target device 140, such as via a network transmission.

[0047] Communication medium 160 may also take the form of a storage medium (e.g., a non-transitory storage medium) such as a hard disk, a flash drive, an optical disk, a digital video disk, a Blu-ray disk, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, a computer device of a media generation device (e.g., an optical disk pressing device) may receive the encoded image data from source device 120 and generate a disk containing the encoded video data.

[0048] Input interface 142 may include any type of medium or device capable of receiving information from communication medium 160. The received information includes encoded bitstream 162. For example, input interface 142 may include a receiver or transceiver configured to receive encoded bitstream 162 in real time.

[0049] The machine vision application 146 includes various hardware and / or software for utilizing the decoded image data generated by the image / video decoder 144. For example, the machine vision application 146 may include a display device that displays the decoded image data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices. As another example, the machine vision application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications, such as object recognition and tracking, facial recognition, image matching, image / video search, augmented reality, robotic vision and navigation, autonomous driving, three-dimensional structure construction, stereo correspondence, motion tracking, etc.

[0050] Next, combine Figure 2A-2B and Figure 3A-3B To describe exemplary image data encoding and decoding techniques.

[0051] Figure 2ASchematic diagram of an example encoding process 200A consistent with an embodiment of the present disclosure is shown. For example, the encoding process 200A may be performed by Figure 1 The image / video encoder 124 in FIG. Figure 2A As shown, the encoder can encode the video sequence 202 into a video bit stream 228 according to process 200A. The video sequence 202 may include a set of pictures (referred to as "original pictures") arranged in time order. Each original picture of the video sequence 202 may be divided by the encoder into a plurality of basic processing units, a plurality of basic processing sub-units, or a plurality of regions for processing. In some embodiments, the encoder may perform process 200A at the basic processing unit level of each original picture of the video sequence 202. For example, the encoder may perform process 200A in an iterative manner, wherein the encoder may encode the basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for a plurality of regions of each original picture of the video sequence 202.

[0052] exist Figure 2A 2, an encoder may feed a basic processing unit (referred to as an "original BPU") of an original picture of a video sequence 202 to a prediction stage 204 to produce prediction data 206 and a prediction BPU 208. The encoder may subtract the prediction BPU 208 from the original BPU to produce a residual BPU 210. The encoder may feed the residual BPU 210 to a transform stage 212 and a quantization stage 214 to produce quantized transform coefficients 216. The encoder may feed the prediction data 206 and the quantized transform coefficients 216 to a binary encoding stage 226 to produce a video bitstream 228. The components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as a "forward path." During process 200A, after the quantization stage 214, the encoder may feed the quantized transform coefficients 216 to an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224, which is used in the prediction stage 204 for the next iteration of the process 200A. The components 218, 220, 222, and 224 of the process 200A may be referred to as a "reconstruction path." The reconstruction path may be used to ensure that both the encoder and the decoder use the same reference data for prediction.

[0053] The encoder may iteratively perform process 200A to encode each original BPU of the original picture (in the forward path) and generate a prediction reference 224 for encoding the next original BPU of the original picture (in the reconstruction path). After encoding all the original BPUs of the original picture, the encoder may proceed to encode the next picture in the video sequence 202.

[0054] Referring to process 200A, the encoder may receive a video sequence 202 generated by a video capture device (e.g., a camera). As used herein, the term "receiving" may refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or in any way inputting data.

[0055] In the prediction phase 204, in the current iteration, the encoder may receive an original BPU and a prediction reference 224, and perform a prediction operation to generate prediction data 206 and a prediction BPU 208. The prediction reference 224 may be generated from a reconstruction path in a previous iteration of the process 200A. The purpose of the prediction phase 204 is to reduce information redundancy by extracting prediction data 206 that can be used to reconstruct the original BPU into a prediction BPU 208 from the prediction data 206 and the prediction reference 224.

[0056] Ideally, the predicted BPU 208 can be the same as the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is usually slightly different from the original BPU. In order to record such differences, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate a residual BPU 210. For example, the encoder can subtract the pixel value (e.g., grayscale value or RGB value) of the predicted BPU 208 from the pixel value corresponding to the original BPU. Each pixel value of the residual BPU 210 can have a residual value as a result of this subtraction between the corresponding pixel values ​​of the original BPU and the predicted BPU 208. Compared with the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without obvious quality degradation. Thus, the original BPU is compressed.

[0057] To further compress the residual BPU 210, in the transform stage 212, the encoder can reduce the spatial redundancy of the residual BPU 210 by decomposing the residual BPU 210 into a set of two-dimensional "basic patterns", each of which is associated with a "transform coefficient". The basic patterns can have the same size (e.g., the size of the residual BPU 210). Each basic pattern can represent a changing frequency (e.g., the frequency of brightness change) component of the residual BPU 210. No basic pattern can be reproduced from any combination (e.g., linear combination) of any other basic patterns. In other words, the decomposition can decompose the multiple changes of the residual BPU 210 into the frequency domain. This decomposition is similar to the discrete Fourier transform of a function, where the basic patterns are similar to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are similar to the coefficients associated with the basis functions.

[0058] Different transform algorithms may use different basic modes. Various transform algorithms may be used in the transform stage 212, such as discrete cosine transform, discrete sine transform, and the like. The transform at the transform stage 212 is reversible. That is, the encoder may restore the residual BPU 210 by an inverse operation of the transform (referred to as an "inverse transform"). For example, in order to restore a certain pixel of the residual BPU 210, the inverse transform may be to multiply the corresponding pixel values ​​of a plurality of basic modes by their respective associated coefficients, and to add the above products to produce a weighted sum. For video coding standards, both the encoder and the decoder may use the same transform algorithm (and therefore have the same basic mode). Therefore, the encoder may only record the transform coefficients, and the decoder may reconstruct the residual BPU 210 based on these transform coefficients without receiving the basic mode from the encoder. Compared with the residual BPU 210, the transform coefficients may have fewer bits, but they may be used to reconstruct the residual BPU 210 without significant quality degradation. Therefore, the residual BPU 210 is further compressed.

[0059] The encoder may further compress the transform coefficients in the quantization stage 214. During the transform process, different basic patterns may represent different frequencies of change (e.g., frequencies of brightness changes). Since the human eye is generally better at recognizing low-frequency changes, the encoder may ignore information about high-frequency changes without causing a significant decrease in decoding quality. For example, in the quantization stage 214, the encoder may generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to its nearest integer. After such an operation, some of the transform coefficients of the high-frequency basic patterns may be converted to zero, while the transform coefficients of the low-frequency basic patterns may be converted to smaller integers. The encoder may ignore the zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed as transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0060] Because the encoder ignores the remainder of this division in the rounding operation, the quantization stage 214 may be lossy. Generally, the quantization stage 214 may cause the greatest information loss in the process 200A. The greater the information loss, the fewer bits may be required to quantize the transform coefficients 216. To achieve different levels of information loss, the encoder may use different values ​​for the quantization parameter or any other parameter in the quantization process.

[0061] At the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantized transform coefficients 216 using a binary encoding technique, such as entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder may also encode other information at the binary encoding stage 226, such as the prediction mode used at the prediction stage 204, the parameters of the prediction operation, the transform type at the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., rate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packaged for network transmission.

[0062] Referring to the reconstruction path of process 200A, at the inverse quantization stage 218, the encoder may perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. At the inverse transform stage 220, the encoder may generate a reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0063] It should be noted that other variations of process 200A may be used to encode video sequence 202. In some embodiments, the stages of process 200A may be performed by the encoder in a different order. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may omit Figure 2A one or more stages in a process.

[0064] Figure 2B 2 shows a schematic diagram of another exemplary encoding process 200B consistent with an embodiment of the present disclosure. For example, the encoding process 200B may be composed of, for example Figure 1 200B can be modified from process 200A. For example, process 200B can be used by an encoder that complies with a hybrid video coding standard (e.g., H.26x series). Compared with process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filtering stage 232 and a buffer 234.

[0065] In general, prediction techniques can be divided into two categories: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-frame prediction") can use pixels from one or more encoded adjacent BPUs in the same picture to predict the current BPU. That is, the prediction reference 224 in the spatial prediction may include the multiple adjacent BPUs. The spatial prediction can reduce the spatial redundancy inherent in the picture. Temporal prediction (e.g., inter-picture prediction or "inter-frame prediction") can use multiple regions from one or more encoded pictures to predict the current BPU. That is, the prediction reference 224 in the temporal prediction may include multiple encoded pictures. Temporal prediction can reduce the temporal redundancy inherent in the multiple pictures.

[0066] Referring to process 200B, in the forward path, the encoder performs the prediction operation in the spatial prediction stage 2042 and the temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder may perform intra-frame prediction. For a certain original BPU of a certain picture being encoded, the prediction reference 224 may include one or more neighboring BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder may generate a predicted BPU 208 by extrapolating multiple neighboring BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform the extrapolation at the pixel level, for example, by extrapolating to obtain the corresponding pixel value of each pixel of the predicted BPU 208. The multiple neighboring BPUs used for extrapolation can be positioned from various directions relative to the original BPU, for example, in the vertical direction (for example, at the top of the original BPU), the horizontal direction (for example, on the left side of the original BPU), the diagonal direction (for example, the lower left, lower right, upper left or upper right of the original BPU), or any direction defined in the video coding standard used. For intra-frame prediction, the prediction data 206 may include, for example, the positions (for example, coordinates) of the multiple neighboring BPUs used, the sizes of the multiple neighboring BPUs used, the parameters of the extrapolation, the direction of the neighboring BPUs used relative to the original BPU, and the like.

[0067] For another example, in the temporal prediction stage 2044, the encoder may perform the inter-frame prediction. For a certain original BPU of the current picture, the prediction reference 224 may include one or more pictures (referred to as "reference pictures") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference picture may be encoded and the BPU may be reconstructed by the BPU. For example, the encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. After all reconstructed BPUs of the same picture are generated, the encoder may generate a reconstructed picture as a reference picture. The encoder may perform a "motion estimation" operation to search for a matching area within a range of the reference picture (referred to as a "search window"). The position of the search window in the reference picture may be determined based on the position of the original BPU in the current picture. For example, the search window may be centered at a position in the reference picture having the same coordinates as the original BPU in the current picture, and may extend outward by a predetermined distance. When the encoder identifies an area similar to the original BPU in the search window (e.g., by using a pixel recursive algorithm, a block matching algorithm, etc.), the encoder may determine such an area as a matching area. The matching area may have a different size than the original BPU (e.g., smaller than, equal to, larger than, or having a different shape). Since the reference picture and the current picture are temporally separated on the time axis, it can be considered that the matching area "moves" to the position of the original BPU over time. The encoder may record the direction and distance of this movement as a "motion vector". When multiple reference pictures are used, the encoder may search for a matching area and determine the associated motion vector for each reference picture. In some embodiments, the encoder may assign weights to the pixel values ​​of the matching areas of each matching reference picture.

[0068] The motion estimation may be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, a plurality of motion vectors associated with the matching region, the number of reference pictures, a plurality of weights associated with the plurality of reference pictures, and the like.

[0069] In order to generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. The motion compensation may be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., the motion vector) and the prediction reference 224. For example, the encoder may move the matching area of ​​the reference picture according to the motion vector to predict the original BPU of the current picture. When multiple reference pictures are used, the encoder may move the matching areas of the multiple reference pictures according to the respective motion vectors and average pixel values ​​of the matching areas. In some embodiments, if the encoder has assigned weights to the pixel values ​​of the matching areas of the respective matching reference pictures, the encoder may add the weighted sum of the pixel values ​​of the multiple moved matching areas.

[0070] In some embodiments, inter prediction can be unidirectional or bidirectional. Unidirectional inter prediction can use one or more reference pictures in the same temporal direction relative to the current picture. Unidirectional inter prediction uses a reference picture before the current picture. Bidirectional inter prediction can use one or more reference pictures in both temporal directions relative to the current picture.

[0071] Still referring to the forward path of process 200B, after spatial prediction 2042 and temporal prediction stage 2044, at mode decision stage 230, the encoder may select a prediction mode (e.g., one of intra prediction or inter prediction) for the current iteration of process 200B. For example, the encoder may perform rate-distortion optimization techniques, wherein the encoder may select a prediction mode to minimize a value of a cost function based on the bit rate of a candidate prediction mode and the distortion of reconstructing a reference picture under the candidate prediction mode. Based on the selected prediction mode, the encoder may generate a corresponding prediction BPU 208 and prediction data 206.

[0072] In the reconstruction path of process 200B, if the intra-frame prediction mode has been selected in the forward path, after generating the prediction reference 224 (e.g., the current BPU that has been encoded and reconstructed in the current picture), the encoder can directly feed the prediction reference 224 to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). If the inter-frame prediction mode has been selected in the forward path, after generating the prediction reference 224 (e.g., in the current picture, all BPUs have been encoded and reconstructed), the encoder can feed the prediction reference 224 to the loop filtering stage 232, where the encoder can apply loop filtering to the prediction reference 224 to reduce or eliminate the distortion (e.g., blocking effects) introduced by the inter-frame prediction. The encoder can apply various loop filtering techniques in the loop filtering stage 232, such as deblocking, sample adaptive offset, adaptive loop filtering, etc. The loop-filtered reference pictures may be stored in a buffer 234 (or "decoded picture buffer") for subsequent use (e.g., as an inter-prediction reference picture for a future picture of the video sequence 202). The encoder may store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder may encode the parameters of the loop filter (e.g., loop filter strength) as well as the quantized transform coefficients 216, the prediction data 206, and other information in a binary encoding stage 226.

[0073] Figure 3A Schematic diagram of an exemplary decoding process 300A consistent with an embodiment of the present disclosure is shown. For example, the decoding process 300A may be performed by, for example Figure 1 Process 300A may be performed by a decoder such as the image / video decoder 144 in FIG. Figure 2A In some embodiments, process 300A may be similar to the reconstruction path of process 200A. A decoder (e.g., Figure 1 The image / video decoder 144 in the embodiment of the present invention can decode the video bitstream 228 into a video stream 304 according to the process 300A. The video stream 304 can be very similar to the video sequence 202. However, due to the compression and decompression process (e.g., Figure 2A-2B The information in the quantization stage 214 is lost, and typically, the video stream 304 is not identical to the video sequence 202. Figure 2A-2B200A and 200B in the decoder, the decoder may perform process 300A at a basic processing unit (BPU) level for each picture encoded in the video bitstream 228. For example, the decoder may perform process 300A in an iterative manner, wherein the decoder may decode a basic processing unit in one iteration of process 300A. In some embodiments, the decoder may perform process 300A in parallel for regions of each picture encoded in the video bitstream 228.

[0074] exist Figure 3A In the process 300A, the decoder may feed a portion of the video bitstream 228 associated with a basic processing unit (referred to as an "encoded BPU") of an encoded picture to the binary decoding stage 302. In the binary decoding stage 302, the decoder may decode the portion into prediction data 206 and quantized transform coefficients 216. The decoder may feed the quantized transform coefficients 216 to the inverse quantization stage 218 and the inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may feed the prediction data 206 to the prediction stage 204 to generate the prediction BPU 208. The decoder may add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224. In some embodiments, the prediction reference 224 may be stored in a buffer (e.g., a decoded picture buffer in a computer memory). The decoder may feed the prediction reference 224 to the prediction stage 204 for performing a prediction operation in the next iteration of the process 300A.

[0075] The decoder may iteratively perform process 300A to decode each coded BPU of the coded picture and generate a prediction reference 224 for encoding the next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder may output the picture to a video stream 304 for display and continue decoding the next coded picture in the video bitstream 228.

[0076] In the binary decoding stage 302, the decoder may perform the inverse of the binary coding technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the decoder may also decode other information in the binary decoding stage 302, such as the prediction mode, parameters of the prediction operation, the transform type, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., rate control parameters), etc. In some embodiments, if the video bitstream 228 is transmitted in the form of packets over the network, the decoder may unpack the video bitstream 228 before feeding the video bitstream 228 to the binary decoding stage 302.

[0077] Figure 3B FIG. 3 is a schematic diagram of another exemplary decoding process 300B consistent with an embodiment of the present disclosure. For example, the decoding process 300B may be performed by, for example Figure 1 2042 and 2044. The process 300B may be modified from the process 300A. For example, the process 300B may be used by a decoder that complies with a hybrid video coding standard (e.g., H.26x series). Compared to the process 300A, the process 300B further divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and also includes a loop filtering stage 232 and a buffer 234.

[0078] In process 300B, for a coding basic processing unit (referred to as "current BPU") of an encoded picture (referred to as "current picture") being decoded, the prediction data 206 decoded by the decoder from the binary decoding stage 302 may contain various types of data, depending on what prediction mode the encoder uses to encode the current BPU. For example, if the encoder uses intra-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating that intra-frame prediction is used, parameters of the intra-frame prediction operation, and the like. The parameters of the intra-frame prediction operation may include, for example, the position (e.g., coordinates) of one or more neighboring BPUs used as references, the sizes of multiple neighboring BPUs, extrapolation parameters, directions of multiple neighboring BPUs relative to the original BPU, and the like. For another example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating that inter-frame prediction is used, parameters of the inter-frame prediction operation, and the like. The parameters of the inter-frame prediction operation may include, for example, the number of multiple reference pictures associated with the current BPU, the weights respectively associated with the multiple reference pictures, the positions (e.g., coordinates) of one or more matching regions in the respective reference pictures, one or more motion vectors respectively associated with the matching regions, and the like.

[0079] Based on the prediction mode indicator, the decoder can decide whether to perform spatial prediction (e.g., intra prediction) in the spatial prediction stage 2042 or temporal prediction (e.g., inter prediction) in the temporal prediction stage 2044. Details of performing such spatial prediction or temporal prediction are described in Figure 2B After performing such spatial prediction or temporal prediction, the decoder may generate a prediction BPU 208. The decoder may add the prediction BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, such as Figure 3A as shown in .

[0080] In process 300B, the decoder may feed the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 for performing a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intra-frame prediction in the spatial prediction stage 2042, then after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder may feed the prediction reference 224 directly to the spatial prediction stage 2042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). If the current BPU is decoded using inter-frame prediction in the temporal prediction stage 2044, then after generating the prediction reference 224 (e.g., in a reference picture, all BPUs have been decoded), the encoder may feed the prediction reference 224 to the loop filtering stage 232 to reduce or eliminate distortion (e.g., blocking effects). The decoder may perform the following steps: Figure 2B In-loop filtering is applied to the prediction reference 224 in the manner described in . The loop-filtered reference picture may be stored in a buffer 234 (e.g., a decoded picture buffer in a computer memory) for subsequent use (e.g., as an inter-prediction reference picture for a subsequent encoded picture of the video bitstream 228). The decoder may store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, when the prediction mode indicator of the prediction data 206 indicates that inter-prediction is used to encode the current BPU, the prediction data may further include parameters of the loop filtering (e.g., loop filtering strength).

[0081] Return to reference Figure 1 The image / video preprocessor 122, the image / video encoder 124 and the image / video decoder 144 may be implemented using any suitable hardware, software or a combination thereof. Figure 4 4 is a block diagram of an example apparatus 400 for processing image data consistent with an embodiment of the present disclosure. For example, the apparatus 400 may be a preprocessor, an encoder, or a decoder. Figure 4As shown, the device 400 may include a processor 402. When the processor 402 executes the instructions described herein, the device 400 can become a special-purpose machine for preprocessing, encoding and / or decoding image data. The processor 402 can be any type of circuit capable of manipulating or processing information. For example, the processor 402 may include any number of CPU processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property cores (IP cores), programmable logic arrays (PLAs), programmable array logic (PALs), general array logic (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), systems on chip (SoCs), application-specific integrated circuits (ASICs), or other similar combinations. In some embodiments, the processor 402 may also be a group of processors grouped into a single logical component. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0082] The apparatus 400 may also include a memory 404 configured to store data (eg, instruction sets, computer codes, intermediate data, etc.). Figure 4 As shown, the stored data may include program instructions (e.g., program instructions for implementing the stages in process 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video bitstream 228, or video stream 304). Processor 402 may access the program instructions and data for processing (e.g., via bus 410), and execute the program instructions to perform operations or manipulations on the data for processing. Memory 404 may include a high-speed random access storage device or a non-transitory storage device. In some embodiments, memory 404 may include any number of random access memories (RAMs), read-only memories (ROMs), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, etc. or any combination of the like. Memory 404 may also be a group of memories (e.g., multiple memory cards) grouped into a single logical component. Figure 4 not shown).

[0083] The bus 410 may be a communication device that transmits data between components inside the apparatus 400 , such as an internal bus (eg, a CPU-memory bus), an external bus (eg, a universal serial bus port, a peripheral component interconnect express port), and the like.

[0084] For ease of explanation and without ambiguity, the processor 402 and other data processing circuits are collectively referred to as "data processing circuits" in this disclosure. The data processing circuits may be implemented entirely as hardware, or as a combination of software, hardware, or firmware. In addition, the data processing circuits may be a single independent module or may be fully or partially combined into any other component of the device 400.

[0085] The device 400 may further include a network interface 406 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 406 may include any number of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, or any combination of other similar items.

[0086] In some embodiments, the apparatus 400 may optionally further include a peripheral interface 408 to provide a connection to one or more peripheral devices. Figure 4 As shown, the peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), a video input device (e.g., a camera or input interface coupled to a video archive), and the like.

[0087] It should be noted that the video codec (e.g., a codec that performs process 200A, 200B, 300A, or 300B) can be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of process 200A, 200B, 300A, or 300B can be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. For another example, some or all stages of process 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of device 400, such as dedicated data processing circuits (e.g., FPGA, ASIC, NPU, etc.).

[0088] With the development of multimedia processing, transmission and application, visual data has shown explosive growth. In addition, due to the huge storage and transmission costs of raw visual data, it is impractical to store and transmit raw visual data. At the same time, there is huge redundancy in visual data, which has prompted the compressed representation of visual data, which makes it possible to store and transmit visual data in practical applications. Usually, the end user of visual data is human vision, and the compressed representation of visual data is designed for human perception quality. However, in recent years, with the great progress of artificial intelligence, many machine analysis applications for visual data based on deep learning have emerged, called machine vision. In applications oriented to machine vision, machine analysis performance has replaced human perception quality as the ultimate indicator of visual data compression, which is beyond the scope of visual data compression algorithms designed for human vision. Therefore, it is necessary to develop a compressed representation of visual data for machine vision. In order to solve this problem, the present disclosure provides an image data compression method for machine vision using pre-analysis. Specifically, the pre-analysis for machine vision can extract key information of machine analysis to obtain better analysis performance, and eliminate information irrelevant to machine analysis to obtain a more compact representation. Furthermore, the disclosed pre-analysis algorithm can be adapted to compression codecs oriented toward human vision, which means it is a general technique for visual data compression for machine vision applications.

[0089] The development of image / video compression algorithms is driven by coding standards. For texture compression, a series of standards have been developed to compress visual data, such as JPEG and JPEG 2000 for static image compression, and H.264 / AVC, H.265 / HEVC, and VVC (Versatile Video Coding) for video data compression. To further improve compression performance, many algorithms have been developed for future video compression standards, including matrix-weighted intra prediction, quadtree plus binary, extended coding unit partitioning, and pattern-dependent non-separable quadratic transform. At the same time, for encoder optimization, various optimization algorithms have been proposed to achieve rate-distortion optimization of texture and feature quality. In addition, with the great development of visual data understanding, it is a huge challenge to compactly manage thousands of visual data bit streams and transmit them simultaneously for further analysis, such as smart cities and video Internet of Things (IoVT). In addition, due to the degradation of feature quality in compression for human visual quality, the analysis performance may be significantly affected. To address this issue, the Moving Picture Experts Group (MPEG) has also developed a compressed visual feature representation standard to reduce the representation data size of the analyzed features, so that various intelligent tasks can be promoted through front-end intelligence. Specifically, the standards for Compressed Descriptors for Visual Search (CDVS) ​​and Compressed Descriptors for Video Analytics (CDVA) have been finalized, aiming to obtain very compact descriptors for visual data. In addition, the standardization of video coding for machines has also been initiated, striving to paint a complete picture of compressed representation of visual data in terms of machine vision.

[0090] In addition, deep learning can be used for various applications, especially in the field of visual data representation and understanding. In particular, an end-to-end compression framework based on deep neural networks can be used. Recurrent neural networks (RNNs) can be applied to end-to-end learned image representation to achieve performance comparable to JPEG. Inspired by block-based transforms in traditional image / video compression, an end-to-end image compression model based on convolutional neural networks (CNNs) can be combined with discrete cosine transforms (DCTs) to achieve performance comparable to JPEG at low bit rates. Nonlinear transformation is one of the fundamental properties of neural networks, which is consistent with the human visual system (HVS). Therefore, the end-to-end nonlinear transformation codec can be optimized using generalized segmentation normalization (GDN) to improve the perceptual quality. On this basis, a density estimation model can be combined with a cascaded GDN to surpass the compression performance of JPEG2000. In end-to-end image compression, the redundancy of the latent coding can be further eliminated by weight re-parameterization under the entropy penalty constraint, which is achieved through a variational hyper-prior model. To further exploit the relevance of the latent representation, in terms of PSNR and MS-SSIM distortion metrics, in order to obtain excellent rate-distortion performance comparable to the current state-of-the-art image codec BPG, an autoregressive model can be used. To further improve the accuracy of the entropy model in bitrate estimation, the distribution of the latent representation can be parameterized using discrete Gaussian mixture likelihood, which can formulate a more accurate and flexible entropy model and achieve comparable performance to the latest compression standard VVC in terms of bitrate-PSNR performance.

[0091] Typically, traditional visual data compression is achieved by compactly representing local and low-level information in a block-by-block manner. However, this may limit the compression representation capability because high-level information is not fully utilized. To address this issue, pre-analysis can be used to improve the representation performance because it can extract global and high-level information of the visual data. Specifically, a visual attention-based pre-analysis model can be used to optimize the encoding parameter settings to obtain better compression performance. In addition, pre-analysis can also be applied to rate control for real-time video coding. From the perspective of simplifying encoding complexity, pre-analysis can also be applied to the acceleration of integer motion estimation in JEM.

[0092] As mentioned above, there have been many developments in visual data compression in recent decades. With the advancement of various machine analysis tasks, some analysis feature compression algorithms have also been proposed to improve the visual data compression efficiency of machine vision and adapt to the rapid development of machine analysis applications. However, existing visual data codecs mainly focus on signal fidelity and human visual quality rather than machine vision. In order to improve the representation efficiency of visual data and adapt to existing codecs, pre-analysis has been studied, but existing pre-analysis methods mainly focus on human perceptual quality, which limits the compression representation performance of visual data for machine vision.

[0093] The present disclosure provides a pre-analysis based compression method suitable for machine vision.

[0094] Figure 5 is a schematic diagram illustrating an exemplary compression method 500 for visual data compression for machine vision according to some embodiments consistent with the present disclosure. For example, the method 500 may be performed by the image / video preprocessor 122 ( Figure 1 ) to execute. Figure 5 As shown, the method 500 includes two stages: a machine vision-based pre-analysis stage 510 and a pre-processing stage 520. Specifically, the machine vision-based pre-analysis stage 510 performs high-level information extraction, such as instance segmentation, through a machine analysis model, and detects one or more segmentation masks 512 corresponding to one or more instances (i.e., objects or foregrounds). The pre-analysis stage 510 can also run a classifier (e.g., a neural network classifier) ​​to determine a class ( Figure 5 ). The class may be an object class that indicates a certain type of object associated with each segmentation mask (e.g., face, tree, animal, building, etc.). The preprocessing stage 520 includes operations formulated based on the output of the pre-analysis stage 510 (e.g., segmentation masks and predicted categories) to balance between machine vision performance and compressibility of the representation. The pre-analysis stage 510 may employ machine analysis models such as Faster Region Convolutional Neural Network (Faster R-CNN) and Mask Region Convolutional Neural Network (which may be referred to as Mask R-CNN, or Mask R-CNN).

[0095] For example, Mask R-CNN is a deep learning model that combines object detection and instance segmentation. According to some embodiments, the overall architecture of an exemplary Mask R-CNN may include: (1) a backbone network, (2) a region proposal network (RPN), (3) a region of interest alignment (ROIAlign, Region of Interest Align), and (4) a mask branch (MaskHead).

[0096] (1) Backbone network: The backbone network can be a pre-trained convolutional neural network, such as a residual network (ResNet) or ResNeXt (Residual Next Step Network). The backbone network processes the input image and generates a corresponding feature map.

[0097] (2) Region Proposal Network (RPN): The RPN is responsible for generating region proposals or candidate bounding boxes that may contain objects within the image. It operates on the feature maps generated by the backbone network and recommends potential regions of interest (ROIs).

[0098] (3) ROIAlign: The main purpose of ROIAlign is to align the features within the region of interest (ROI) with the spatial grid of the output feature map. This alignment prevents the information loss that may occur when quantizing the spatial coordinates of the ROI to the nearest integer (as done in ROI pooling).

[0099] (4) Mask Head: Mask Head is an additional branch in Mask R-CNN that is responsible for generating segmentation masks for each region proposal. The branch uses the aligned features obtained through ROIAlign to predict a binary mask for each object, depicting the pixel-level boundaries of the instance.

[0100] During training, the Mask R-CNN model uses a combination of classification loss, bounding box regression loss, and mask segmentation loss for joint optimization. This enables the Mask R-CNN model to learn to simultaneously detect objects, refine their bounding boxes, and generate accurate segmentation masks.

[0101] As described above, once the pre-analysis stage 510 determines that one or more instances in the input image belong to a region of interest (ROI), the pre-processing stage 520 can further filter out these instances through the corresponding segmentation mask 512. The background portion of the input image can then be omitted and does not participate in subsequent processing. Compared with the original input image, the filtered portion with less image information is more likely to obtain better image encoding performance, especially for machine vision. In the above example, the machine vision pre-analysis model is instance segmentation, and the output of the pre-analysis stage includes multiple segmentation masks and predicted categories. However, it is expected that the disclosed embodiments are not limited to the above examples, and any suitable machine vision pre-analysis and pre-processing techniques can be used to improve the efficiency of characterization of visual data for machine vision.

[0102] Next, the details of the preprocessing stage 520 are described. Figure 6A-6E schematically illustrates various ways to pre-process an input image based on pre-analysis results according to some embodiments of the present disclosure. In some embodiments, the pre-processing operation can be performed directly on the segmentation mask. Specifically, since the compressed visual data is processed by machine vision, only information related to machine analysis is necessary and needs to be retained. The machine analysis related information includes the information generated by the pre-analysis stage (e.g., Figure 5 The segmentation mask output by the pre-analysis stage 510 in FIG. Figure 6 A shows an input image showing a plurality of objects (e.g., people) corresponding to a plurality of instances (i.e., segmentation masks) identified in the pre-analysis stage. In order to directly pre-process the segmentation masks, the segmentation masks of the plurality of instances are combined together to obtain a merged mask in a union manner. For example, Figure 6 As shown in B, the segmentation masks corresponding to the objects (i.e., people) can be combined to form a merge mask. Only the information within the merge mask is retained for further compression. Figure 7-15 Further description Figure 6 Technical aspects of A-6E. However, as those skilled in the art will appreciate, although some of these figures may be described in conjunction with each other below, this does not necessarily mean that their corresponding embodiments should be implemented in combination.

[0103] In some embodiments, block-based preprocessing can be used to process the segmentation mask output by the pre-analysis stage. Different from retaining only the instance mask area to extract the most essential information, the area around the merged mask is also important for achieving better instance discrimination. Inspired by this, Figure 6 As shown in C, a block-based preprocessing operation can be used to preserve the visual information around the instance. Specifically, a sliding window operation S is performed on the merged mask. n , the window size is n×n, where n can be, for example, 128, 192 or 256. As long as the sliding window S n There is an intersection between and the merge mask, so the information of the original visual data can be retained, such as Figure 6 As shown in C. Fig.14 A and Fig.14 B schematically illustrates block-based preprocessing, which will be described below.

[0104] In some embodiments, dilation-based preprocessing can be used to process the segmentation mask output by the pre-analysis stage. Dilation is an important image processing method. In this document, it is used in conjunction with the preprocessing operation to preserve information around the predicted instance. Specifically, Figure 6As shown in D, the merge mask is expanded to expand the predicted instance region, the convolution kernel size is k×k, and the number of iterations is t, for example, 7×7 and 3 respectively. Similarly, the information of the original visual data in the corresponding region can be retained for further compression, as shown in Figure 6 As shown in D. Fig.15 A-15D schematically illustrates expansion-based preprocessing, which will be described below.

[0105] In some embodiments, the block-based preprocessing and dilation-based preprocessing can be combined to process multiple segmentation masks output by the preprocessing stage. The granularity of the block-based preprocessing operation is coarse and may introduce redundant information. However, the irregular boundaries of visual data processed using the dilation-based preprocessing operation increase the representation overhead of visual data compression, especially for block-based compression codecs such as HEVC and VVC. To address this issue, Figure 6 As shown in FIG. 1 , a dilation-based and block-based preprocessing operation can be used. Specifically, a dilation-based preprocessing is first performed, and a block-based preprocessing is performed based on the output of the dilation-based preprocessing. Here, the sliding window size n s Can be smaller, such as 64 and 128, such as Figure 6 As shown in E.

[0106] In some embodiments, blur preprocessing can be used to improve the smoothness and representation compactness of image data. In order to prevent the merge mask region where the object is located from becoming blurred, thereby reducing the performance or accuracy of machine vision tasks (such as object recognition or tracking), the output of blur preprocessing I blur It can be expressed as a formula, for example, I blur =G k (M bd -M)I+MI, where I is the original image, M is the merge mask, and M bd is the mask after block-based and / or dilation preprocessing, and G k is a Gaussian filter with a convolution kernel size of k. Thus, in the block and / or dilation preprocessed image, the blur preprocessing is only applied to the image area outside the merge mask (ie, non-overlapping), thereby improving the compression efficiency without impairing the machine analysis performance.

[0107] In some embodiments, to improve compression efficiency, the pre-processing stage may be adaptively performed based on a quantization parameter (QP) and / or the resolution of the input visual data.

[0108] Specifically, when the compression codec is a traditional codec (such as HEVC and VVC), a lower QP means a higher representation overhead and better visual data reconstruction quality. When the QP is high (for example, greater than 32), the representation overhead is limited, and more coding bits should be allocated to the instance area to obtain better reconstruction quality. In this way, the sliding window size n in the block-based preprocessing and the convolution kernel size in the expansion-based preprocessing can become smaller, for example, 64 and 3×3, respectively. In addition, in the case of high QP, blur preprocessing can also be applied to the output of the block and / or expansion-based preprocessing to further improve the compression efficiency. On the contrary, when the QP is small (for example, less than 32), the coding overhead is sufficient and more original visual data information can be retained to obtain better machine analysis performance. For example, the window size and convolution kernel size can be 256 and 9×9, respectively.

[0109] In addition, the definition of the input visual data is also important. For high-definition (e.g., greater than 1920×1080) visual data, the sliding window size of block-based preprocessing and the kernel size of dilation-based preprocessing can be set larger, such as 256 and 7×7, respectively. On the contrary, for low-definition (e.g., less than 1920×1080) visual data, the sliding window size of block-based preprocessing and the convolution kernel size of dilation-based preprocessing can be set smaller, such as 128 and 3×3, respectively.

[0110] Consistent with the disclosed embodiments, the adaptive pre-processing may be based on QP, resolution of the input picture, or a combination thereof.

[0111] Figure 7 700 is a flowchart of an exemplary image data compression method 700 according to some embodiments of the present disclosure. For example, the method 700 may be performed by one or more processors, such as the image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 4 , the method 700 can be performed by the device 400. Figure 7 As shown, method 700 includes the following steps 710-740.

[0112] At step 710 , the processor performs instance segmentation on the input image to generate one or more segmentation masks.

[0113] Specifically, the processor may execute an instance segmentation algorithm to divide an input image into a plurality of parts, for example, sets of pixels representing a portion of the input image. The instance segmentation algorithm may assign a label (i.e., a category) to each group of pixels so that pixels with the same label share certain common features. The processor may then group the pixels according to the labels of the pixels and designate each group as a specific object, i.e., a segmentation mask. Based on the labels, the processor may determine the boundaries of each segmentation mask. The labels do not necessarily belong to different categories. For example, Figure 6 The two persons in B may be labeled with different labels, but they may belong to the same class of objects, namely persons. In some embodiments of the present disclosure, each of the two persons is an instance and can be detected by an instance segmentation algorithm. For example, the processor can perform an instance segmentation algorithm on the input image, such as Fig.13 As shown in FIG. 13A-13D , the pixels belonging to the example (two persons) can be labeled as “person-A” and “person-B”. At the same time, segmentation masks 1310 and 1320 are generated accordingly, as shown in FIG. Fig.13 A and 13B, where a mask 1310 is generated by grouping pixels of "person-A", and a mask 1320 is generated by grouping pixels of "person-B".

[0114] In some embodiments, after generating the segmentation masks, the processor may further execute a classification algorithm to determine the object category associated with each segmentation mask. For example, the classification algorithm may be implemented using a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN).

[0115] Return to reference Figure 7 In step 720, the processor combines one or more segmentation masks to generate a merge mask. Specifically, the merge mask can form a connected region in the input image. For example, Figure 6 As shown in B, the segmentation masks corresponding to multiple persons can be merged to form a connected region. Fig.13 C, where the segmentation mask 1310 and the segmentation mask 1320 are combined to form a merged mask 1330. The combination of different segmentation masks will reduce the number of masks, thereby reducing the computational burden of image processing. On the other hand, the combination of different segmentation masks, as well as other morphological techniques described below (e.g., block-based preprocessing or dilation-based preprocessing), will reduce the risk of missing some instances due to missed segmentation at the execution step 710, and the lost some instances may also be the region of interest (ROI) of the input image.

[0116] In some embodiments, the multiple segmentation masks generated in step 710 may be isolated from each other. In this case, the merged mask combined with the multiple isolated masks consists of multiple non-overlapping parts. In other embodiments, the multiple segmentation masks generated in step 710 may be adjacent to each other. In this case, the merged mask consists of multiple connected regions. Fig.13 C and Fig.13 As shown in D, the merge mask 1330 and the corresponding area 1340 can be retained in the picture for subsequent processing, while the background area 1350 in the input picture can be discarded and may not participate in the subsequent processing.

[0117] Return to reference Figure 7 , in step 730 , the processor extracts a region including the merge mask from the input image.

[0118] In some embodiments, the extracted region includes only a merge mask. Figure 6 As shown in B, only the merge mask is extracted from the input image.

[0119] In some embodiments, the extraction area includes not only the merge mask, but also an extended area around the merge mask. Figure 6 As shown in c-6e, block-based preprocessing and dilation-based preprocessing can be used to create an extended region around the merge mask.

[0120] As mentioned above, the merge mask may consist of several parts that are not adjacent to each other, which may increase the complexity or reduce the efficiency of image coding to some extent. The morphological processing combined with the merge mask will incorporate parts that may be regions of interest (ROIs), and in some cases, it will bridge non-overlapping parts into a unified whole.

[0121] When block-based pre-processing is used, the processor applies a sliding window to the image area outside the merge mask to identify one or more blocks that do not overlap with the merge mask. The processor then extracts a region from the input picture that does not include the one or more blocks. Figure 8 Block-based preprocessing is shown, which will be described in detail below.

[0122] When using dilation-based preprocessing, the processor applies a convolution kernel to a merge mask to determine a first region extending from a boundary of the merge mask. The processor then extracts the first region and the merge mask from an input picture. Fig. 9 A dilation-based preprocessing is shown, which will be described in detail below.

[0123] In some embodiments, the block-based preprocessing and the dilation-based preprocessing may be used in combination. First, the processor dilates the merge mask by applying a convolution kernel to the merge mask to determine a first region extending from a boundary of the merge mask. Next, the processor identifies one or more blocks in the input picture that do not overlap with the first region or the merge mask. Finally, the processor extracts a region from the input picture that does not include the one or more blocks. Fig.10 Block-based and dilation-based pre-processing are shown, which will be described in detail below.

[0124] In some embodiments, blur preprocessing can be applied to the area including the merge mask. Specifically, in the block and / or dilation preprocessed image, the blur preprocessing is applied to the extended area around the merge mask to remove details or simple content in the extended area. Since the extended area does not contain an object, blurring it will not affect the accuracy of the machine vision task. Fig.12 A blur pre-processing applied to the extended region is shown, which will be described in detail below.

[0125] In some embodiments, the block-based preprocessing, the dilation-based preprocessing and / or the blur preprocessing may be adaptively performed based on a quantization parameter (QP) used to compress the input picture or the clarity of the input picture. For example, in a high QP case (e.g., when the QP is greater than a predetermined threshold), the sliding window size in the block-based preprocessing and the convolution kernel size in the dilation-based preprocessing may be made smaller, and the blur preprocessing may also be used to blur image areas other than the merge mask. In contrast, in a low QP case (e.g., when the QP is lower than or equal to a predetermined threshold), the sliding window size in the block-based preprocessing and the convolution kernel size in the dilation-based preprocessing may be made larger, and the blur preprocessing may be skipped. Fig.12 QP-based blur preprocessing is shown, which will be described in detail below.

[0126] Return to reference Figure 7 In step 740, the processor compresses the image data representing the extracted area. Specifically, the image data may be fed into a codec for compression using the specific QP. The codec may perform encoding and / or decoding according to an image / video standard such as advanced video, for example, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc.

[0127] Figure 8According to some embodiments of the present disclosure, Figure 7 800 is a flowchart of an exemplary image data compression method 800 of the method 700 shown in FIG. 800. For example, the method 800 may be performed by one or more processors, such as the image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 4 , the method 800 can be performed by the device 400. The method 800 applies block-based preprocessing. Figure 8 As shown, step 730 is further decomposed into two sub-steps 810 and 820. Figure 7 The other steps of inheritance will not be described.

[0128] refer to Figure 8 At step 810, the processor identifies one or more blocks in the input picture that do not overlap with the merge mask. The one or more blocks here are not necessarily coded blocks used to calculate the motion vector. In some examples, one or more blocks may be specifically designed to separate instances and background of the input picture. As a non-limiting example, the processor may apply a sliding window to the input picture to identify one or more blocks that do not overlap with the merge mask.

[0129] like Fig.14 As shown in A, the window can slide from the upper left to the lower right of the input image. Depending on the desired preprocessing granularity, the window size can be, for example, 128×128, 192×192, or 256×256. The window slides through A11, A12, ..., A1n, A21, A22, ...A2n, ... for pixel-level traversal. In other words, the sliding step is equal to the width of the window. In the following, for simplicity, the sliding window in some places is represented as block A11, block A12, ..., block A1n, block A21, block A22, ... block A2n, ...

[0130] When any block fully or partially overlaps with the merge mask 1330, it will be marked as an overlapping block and represented by being filled with solid black, as shown in Fig.14 b. For example, Fig.14 Blocks A12, A13, and A14 in A partially overlap with the merge mask 1330, while block A54 completely overlaps with the merge mask 1330. Therefore, these blocks are overlapping blocks and are in Fig.14 B is filled with pure black. It should be noted that Fig.14In FIG. 1B , the merge mask 1330 is shown on top of the overlapping block 1410. When determining the overlapping block 1410, one or more blocks 1420 in the input picture other than the overlapping block (ie, blocks that do not overlap with the merge mask) may also be identified.

[0131] In some embodiments, the size of the sliding window can be determined based on the quantization parameter (QP) used to compress the input picture and / or the definition of the input picture. When the QP used to compress the input picture is high or the definition of the input picture is low, the representation overhead is limited, and more coding bits should be allocated to the instance area to obtain better reconstruction quality. In this way, the size of the sliding window can be smaller, such as 64×64. On the contrary, when the QP used to compress the input picture is low or the definition of the input picture is high, the size of the sliding window can be larger, such as 256×256.

[0132] Return to reference Figure 8 , in step 820, the processor extracts a region from the input image that does not include one or more blocks 1420. In other words, the processor may extract a region that includes at least the overlapping blocks 1410. In a specific example, the region may consist of the overlapping blocks 1410. As can be appreciated by those skilled in the art, the region around the merge mask 1330 is also important for achieving better distinction of instances. The method 800 extends from the merge mask to obtain a region consisting of at least the overlapping blocks, which covers instances with relatively high probability, thereby reducing missed segmentation in the merge mask.

[0133] Fig. 9 According to some embodiments of the present disclosure, Figure 7 Flowchart of an exemplary image data compression method 900 of method 700 shown in FIG. 900. For example, method 900 may be performed by one or more processors, such as image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 4 , the method 900 can be performed by the device 400. The method 900 applies a preprocessing based on expansion. Fig. 9 As shown, step 730 is further decomposed into two sub-steps 910 and 920. Figure 7 The other steps of inheritance will not be described.

[0134] Reference Fig. 9In step 910, the processor dilates the merge mask to determine a first region extending from a boundary of the merge mask in the input image. As a non-limiting example, the processor may apply a convolution kernel to the merge mask to determine a first region extending from a boundary of the merge mask.

[0135] like Fig.15 As shown in A-15D, in some embodiments, a convolution kernel having a shape of convolution kernel 1530, convolution kernel 1540, or convolution kernel 1550 may be used to dilate the merge mask to determine a first region 1510 (not shown for simplicity) extending from the boundary of the merge mask. In a non-limiting example, a convolution kernel 153 of size 7×7 may be applied for dilation. For example, the merge mask may be dilated using convolution kernel 153 to expand the predicted instance region through 3 iterations. In each iteration, convolution kernel 153 is applied to the input image to dilate in a pixel-level manner. Since the convolution kernel size is smaller than the sliding window, the dilation-based preprocessing may have better morphological performance than the block-based preprocessing, but its computational burden and complexity may be higher.

[0136] In some embodiments, the convolution kernel size can be determined based on the quantization parameter (QP) used to compress the input picture and / or the clarity of the input picture. When the QP used to compress the input picture is high or the clarity of the input picture is low, the representation overhead is limited, and more coding bits should be allocated to the instance area to obtain better reconstruction quality. Therefore, the convolution kernel size can be smaller, such as 3×3. On the contrary, when the QP of the compressed input picture is low or the clarity of the input picture is high, the convolution kernel size can be larger, such as 7×7.

[0137] Return to reference Fig. 9 At step 920, the processor extracts a first region and an optional dilation mask from the input image. Fig.15 As can be seen from A, the first region 1510 is extracted from the input image by dilation, and the remaining region 1520 in the input image except the first region 1510 can be ignored in subsequent processing. As can be appreciated by those skilled in the art, the region around the merge mask is also important for achieving better instance differentiation. Method 900 extends from the merge mask to obtain a first region covering a relatively more likely instance, thereby reducing missed segmentation in the merge mask.

[0138] Fig.10 According to some embodiments of the present disclosure, Figure 71000 is a flowchart of an exemplary image data compression method 1000 of method 700 shown in FIG. 1000. For example, method 1000 may be performed by one or more processors, such as image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 4 , the method 1000 can be performed by the device 400. The method 1000 applies both dilation-based preprocessing and block-based preprocessing. Fig.10 As shown, step 730 is further decomposed into three sub-steps 1010, 1020 and 1030. Figure 7 The other steps of inheritance will not be described.

[0139] Reference Fig.10 In step 1010, the processor dilates the merge mask to determine a first region extending from a boundary of the merge mask. The dilation step of step 1010 may follow Fig. 9 For the sake of brevity, the processes described in the corresponding embodiments are not repeated here.

[0140] Next, at step 1020, the processor identifies one or more blocks that do not overlap with the first region in the input picture or the merge mask. Figure 8 In the described method, the processor may apply a sliding window to the input picture to identify one or more blocks that do not overlap with the first region or the merge mask.

[0141] For example, the window can be slid from the upper left corner of the input picture to the lower right for pixel-level traversal. The window size can be, for example, 128×128, 192×192, or 256×256, and the sliding step is the width of the window. When any of the blocks completely or partially overlaps the first region or the merge mask, it will be marked as an overlapping block. When the overlapping block is determined, one or more blocks in the input picture other than the overlapping block (i.e., blocks that do not overlap with the first region or the merge mask) can also be identified.

[0142] Still reference Fig.10 In step 1030, the processor extracts a region from the input image that does not include one or more blocks. In other words, the processor extracts a region that includes at least a plurality of the overlapping blocks. In a specific example, the region may be composed of a plurality of overlapping blocks. As can be appreciated by those skilled in the art, the region around the merge mask is also important for achieving better instance differentiation.

[0143] Method 1000 extends from the merge mask to obtain a region consisting of at least overlapping blocks, which overlap blocks cover instances with relatively large probability, thereby reducing missed segmentations in the merge mask. Compared with methods 800 and 900, method 1000 simultaneously applies dilation-based preprocessing and block-based preprocessing, wherein, in terms of processing operators (i.e., the sliding window or the convolution kernel), the block-based preprocessing is performed at a relatively large scale, while the dilation-based preprocessing is performed at a relatively small scale. Therefore, method 1000 has better morphological performance than method 800 or method 900 that only uses block-based preprocessing or only uses dilation-based preprocessing. Here, the sliding window size can be smaller than method 800, such as 64×64 or 128×128.

[0144] Fig.11 According to some embodiments of the present disclosure, Figure 7 1. The method 1100 of the exemplary image data compression method 1100 of the method 700 shown in FIG. 1100 may be performed by one or more processors, such as the image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 1 , the method 1100 can be performed by the device 400. The method 1100 applies fuzzy preprocessing. Fig.11 As shown, step 730 is further decomposed into two sub-steps 1110 and 1120. Figure 7 The other steps of inheritance will not be described.

[0145] Reference Fig.11 In step 1110, the processor determines a second region in the extracted region that does not overlap with the merge mask. Fig.14 As shown in A, the window can slide from the upper left to the lower right of the input image. When any block completely or partially overlaps with the merge mask 1330, it will be marked as an overlapping block and characterized by filling it with solid black, as shown in FIG. Fig.14 B. It should be noted that the merge mask 1330 is shown above the overlapping block 1410. When the overlapping block 1410 is determined, one or more blocks 1420 other than the overlapping block in the input picture may also be identified. The one or more blocks 1420 may be used to form a second region. For example, the processor may determine a second region consisting of some or all of the one or more blocks 1420.

[0146] Return to reference Fig.11In step 1120, the processor blurs the second region. In some embodiments, the background portion of the input image is not simply discarded. The second region may be pre-blurred to improve the smoothness and representation compactness of the image data.

[0147] In a further example, blur preprocessing may be applied in conjunction with block and / or dilation preprocessing. To prevent blurring of the merge mask region where the object is located from affecting the performance or accuracy of a machine vision task (e.g., object recognition or tracking), the output of the blur preprocessing may be blur It can be expressed as, for example, I blur =G k (M bd -M)I+MI, where I is the original image, M is the merge mask, and M bd is the mask after the block-based and / or dilated preprocessing, and G k is a Gaussian filter with a convolution kernel size of k. Thus, in the block and / or dilation preprocessed image, the blur preprocessing is only applied to the image area outside the merge mask (ie, non-overlapping), thereby improving the compression efficiency without impairing the machine analysis performance.

[0148] Fig.12 According to some embodiments of the present disclosure, Figure 7 1200 is a flowchart of an exemplary image data compression method 1200 of method 700. For example, method 1200 may be performed by one or more processors, such as image / video preprocessor 122 ( Figure 1 ) and / or image / video encoder 124 ( Figure 1 In some embodiments, the image / video preprocessor 122 and the image / video encoder 124 may be integrated into Figure 4 In the device 400 shown in FIG. 1 , the method 1200 can be performed by the device 400. The method 1200 determines whether to perform blur preprocessing based on a quantization parameter (QP). Fig.12 As shown, step 730 is further decomposed into two sub-steps 1210 and 1220. Figure 7 The other steps of inheritance will not be described.

[0149] Reference Fig.12In step 1210, the processor determines whether to blur at least a portion of the extracted region based on a quantization parameter (QP) used to compress the input picture. In order to further compress the input picture, a portion of the extracted region may be blurred in some manner. For example, if the input picture is to be compressed at a lower quality (higher QP), at least a portion of the extracted region may be blurred. However, if the input picture is to be quantized with more stages (lower QP), the extracted region may not be blurred. Blurring pre-processing applied in the case of more quantization stages may significantly reduce the quality of the encoded picture, which may not be suitable for machine vision.

[0150] Still refer to Fig.12 , in step 1210, the processor blurs the portion of the extraction area in response to the QP being greater than a predetermined threshold. In addition, the blur preprocessing may also be applied to the output of the block-based and / or dilation-based preprocessing to further improve compression efficiency. However, if the QP is less than a predetermined threshold, the processor may not blur any portion of the extraction area. In an embodiment of the present disclosure, the segmentation mask or the merge mask may be applied to the input picture to extract instances in the input picture. As can be understood by those skilled in the art, for example, when the merge mask is applied to the input picture, a portion of the picture (where the instance exists) will be covered by the merge mask. Any area in the picture including an area covered by the merge mask may be referred to as an area including the merge mask, which itself exists in the present disclosure even if there is no additional description here. Or in other words, the expression "extracting an area including the merge mask" may also be restated as "extracting an area in the input picture covered by the applied merge mask". Both expressions have the same technical contribution to the present invention.

[0151] Fig.16 1600 is a flowchart of an exemplary image data decoding method according to some embodiments of the present disclosure. For example, method 1600 may be performed by one or more processors, such as image / video decoder 144 ( Figure 1 In some embodiments, the image / video decoder 144 may be integrated into Figure 4 In the device 400 shown in FIG. 1 , the method 1600 can be performed by the device 400. The method 1600 can decode a bit stream using any of the above-mentioned coding schemes, for example, Figure 3A and Figure 3B The decoding process shown in . Fig.16 As shown, method 1600 includes the following steps.

[0152] At step 1610, the processor receives a bitstream, the bitstream comprising compressed image data with region information. The compressed image data comprises encoding information representing one or more regions extracted from a picture. The process of preprocessing and encoding the extracted regions is described in the above embodiment. The region information indicates one or more locations of the one or more extracted regions in the picture.

[0153] In step 1620, the processor reconstructs a picture by decoding the compressed image data according to the region information. The process of decoding the bit stream is described in detail in the previous section. Figure 3A and Figure 3B is described and incorporated herein. Since the area other than the extraction area may be blank and no coding bits are allocated to the area, the processor can skip the area whenever the compressed image is decoded to obtain a picture. In this regard, there is no reference relationship between the pixels in the extraction area (indicated by the area information) and the pixels in the area outside the extraction area (since the picture size is fixed, the area can be extrapolated from the area information). The decoder can focus on decoding the compressed image data within the extraction area. Therefore, compared with decoding the entire picture, the decoding pressure of the decoder will be reduced.

[0154] like Fig.17 As shown, step 1620 of decoding the compressed image data according to the region information to reconstruct the picture can be further decomposed into two steps 1710 and 1720 and an optional step 1730.

[0155] In step 1710, the processor may generate a first region of a picture based on the decoded image data. The first region is the extracted region represented by the decoded image data. As described above, decoded pixel values ​​from the decoded image data may be used to fill the first region.

[0156] In step 1720, the processor may generate a second region of the picture by filling pixels. Here, the second region is at least a portion of the region outside the first region. As an example, the processor may combine the first region and the second region to form the entire picture. As another example, the processor may ignore the second region, so the second region may be transparent when forming the picture. As another example, the processor may fill the second region according to a preset scheme. For example, the processor may fill the second region of the picture with "black" or "white" pixels, so the second region may exist as the background of the picture. The black pixel here is a pixel with an RGB value of (0, 0, 0), and the white pixel is (255, 255, 255). In another example, the processor may fill the second region with an existing background (e.g., a landscape). For example, the processor may select a landscape picture and then expand or shrink it to the size of the picture to be decoded. Then, the processor performs a cutout process on the landscape picture according to the region information. The remaining part of the landscape picture after the cutout may be used as the background to form the second region of the picture. Finally, the processor may merge the first region and the second region to form a complete picture.

[0157] As an optional step, in step 1730, the processor may restore the position of an instance in the picture based on the region information. In some examples, the decoded picture may be used for other processing, such as object recognition. The processor may also restore the position of the instance. As a premise, the first region (i.e., the extraction region) may cover one or more instances. By restoring the position of the first region, the positions of the one or more instances are known. Therefore, the focus may be placed on the restored position in subsequent processing, such as object recognition.

[0158] It should be noted that Fig.16 and Fig.17 The described region information is not necessary for decoding the compressed video information. Consistent with the disclosed embodiments, any decision scheme may be used to decode the video information compressed based on the above described method. For example, the compressed video information may be decoded by a conventional decoder without any special modification.

[0159] It should be understood that a certain embodiment of the present disclosure may be combined with another embodiment or multiple other embodiments.

[0160] In some embodiments, a non-transitory computer-readable storage medium is also provided, storing one or more bit streams processed according to the above-described method. For example, according to method 700-1200 ( Figure 7-12) generates one or more bitstreams. Specifically, according to some embodiments, the one or more bitstreams include a Figure 1 For example, the one or more bitstreams may be based on Figure 2A and 2B The encoding process described in the above is used to encode the bit stream 228. Before encoding, the input picture of the encoder can be processed by a preprocessor (e.g. Figure 1 The image / video preprocessor 122 in the embodiment performs pre-analysis and pre-processing according to the solution provided by the present disclosure, for example, Figure 7-12 In accordance with the disclosed embodiments, one or more bit streams may also be decoded by a decoder (e.g., Figure 1 For example, one or more bit streams may be decoded according to Figure 3A and 3B In addition, the decoded information can be used to generate a reconstructed picture according to the solution provided by the present disclosure, for example, Fig.16 and Fig.17 Describe the relevant processing procedures.

[0161] In some embodiments, a non-transient computer-readable storage medium including an instruction set is also provided, and the instruction set can be executed by a device for performing the above method (such as the disclosed encoder and decoder). Common forms of non-transient media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cassette and networked versions thereof. The device may include one or more processors (CPU), an input / output interface, a network interface and / or a memory.

[0162] In some embodiments, a computer program product is provided, including: a computer program instruction set, and the computer program instruction set enables a computer to execute the steps of the method described in any embodiment of the present disclosure.

[0163] In some embodiments, a computer program is provided, and the computer program enables a computer to execute the steps of the method described in the embodiments of the present disclosure.

[0164] The embodiments may be further described using the following terms: 1. A method for processing image data, comprising: receiving a video sequence; encoding one or more input pictures associated with the video sequence; and Generate a bit stream, The encoding includes: Perform instance segmentation on the input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representing the extracted area is compressed. 2. The method of clause 1, wherein the extraction region includes one or more pixels adjacent to the merge mask. 3. The method of clause 1, wherein extracting from the input picture a region including the merge mask comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 4. The method of clause 3, wherein a sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of the following: a quantization parameter (QP) used to compress the input picture, or The definition of the input image. 5. The method of clause 1, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the dilation mask are extracted from the input picture. 6. A method according to clause 5, wherein the merge mask is dilated by applying a convolution kernel to the merge mask, and the size of the convolution kernel is determined based on at least one of the following: a quantization parameter (QP) used to compress the input picture, or The definition of the input image. 7. The method of clause 1, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 8. The method of clause 1, wherein extracting from the input picture a region including the merge mask further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred. 9. The method of clause 1, wherein extracting from the input picture a region including the merge mask further comprises: Determining whether to blur at least a portion of the extraction area based on a quantization parameter QP used to compress the input picture; and In response to the QP being greater than a predetermined threshold, blurring the portion of the extraction region. 10. The method of clause 9, wherein the portion of the extracted region does not overlap with the merge mask. 11. A non-transitory computer-readable storage medium storing a bit stream, wherein a method for generating the bit stream comprises: Performing instance segmentation on an input image to generate one or more segmentation masks; combining one or more of the segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representing the extracted area is compressed to generate the bitstream. 12. The medium of clause 11, wherein the extraction region includes one or more pixels adjacent to the merge mask. 13. The medium of clause 11, wherein extracting from the input picture a region including the merge mask comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 14. The medium of clause 13, wherein a sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of: a quantization parameter QP used to compress the input picture, or The definition of the input image. 15. The medium of clause 11, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the merge mask are extracted from the input picture. 16. The medium of clause 15, wherein the merge mask is dilated by applying a convolution kernel to the merge mask, and a size of the convolution kernel is determined based on at least one of: a quantization parameter QP used to compress the input picture, or The definition of the input image. 17. The medium of clause 11, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 18. The medium of clause 11, wherein extracting from the input picture a region including the merge mask further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred. 19. The medium of clause 11, wherein extracting from the input picture a region including the merge mask further comprises: Determining whether to blur at least a portion of the extraction region based on a quantization parameter (QP) used to compress the input picture; and In response to the QP being greater than a predetermined threshold, blurring the portion of the extraction region. 20. The medium of clause 19, wherein the portion of the extracted region does not overlap with the merge mask. 21. An image data processing device, comprising: a memory storing an instruction set; and One or more processors are configured to execute the instruction set to cause the apparatus to perform operations including: Performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representative of the extracted region is compressed. 22. The apparatus of clause 21, wherein the extraction region comprises one or more pixels adjacent to the merge mask. 23. The apparatus of clause 21, wherein extracting from the input picture a region including the merge mask comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 24. The apparatus of clause 23, wherein a sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of: a quantization parameter (QP) used to compress the input picture, or The definition of the input image. 25. The apparatus of clause 21, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the dilation mask are extracted from the input picture. 26. The apparatus of clause 25, wherein the merge mask is dilated by applying a convolution kernel to the merge mask, and a size of the convolution kernel is determined based on at least one of: a quantization parameter (QP) used to compress the input picture, or The definition of the input image. 27. The apparatus of clause 21, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 28. The apparatus of clause 21, wherein extracting from the input picture a region including the merge mask further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred. 29. The apparatus of clause 21, wherein extracting from the input picture a region including the merge mask further comprises: Determining whether to blur at least a portion of the extraction area based on a quantization parameter QP used to compress the input picture; and In response to the QP being greater than a predetermined threshold, blurring the portion of the extraction region. 30. The apparatus of clause 29, wherein portions of the extracted region do not overlap with the merge mask. 31. A non-transitory computer-readable storage medium storing a set of instructions, which, when executed by a computer, causes the computer to perform a method comprising: Performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representing the extracted region is compressed to generate a bitstream. 32. The medium of clause 31, wherein the extraction region includes one or more pixels adjacent to the merge mask. 33. The medium of clause 31, wherein extracting from the input picture a region including the merge mask comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 34. The medium of clause 33, wherein a sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of: a quantization parameter QP used to compress the input picture, or The definition of the input image. 35. The medium of clause 31, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the merge mask are extracted from the input picture. 36. The medium of clause 35, wherein the merge mask is dilated by applying a convolution kernel to the merge mask, and a size of the convolution kernel is determined based on at least one of: a quantization parameter QP used to compress the input picture, or The definition of the input image. 37. The medium of clause 31, wherein extracting from the input picture a region including the merge mask comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture. 38. The medium of clause 31, wherein extracting from the input picture a region including the merge mask further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred. 39. The medium of clause 31, wherein extracting from the input picture a region including the merge mask further comprises: Determining whether to blur at least a portion of the extraction area based on a quantization parameter QP used to compress the input picture; and In response to the QP being greater than a predetermined threshold, blurring the portion of the extraction region. 40. The medium of clause 39, wherein portions of the extracted region do not overlap with the merge mask. 41. A method for processing image data, comprising: receiving a bit stream; and decoding one or more pictures using the encoding information of the bitstream, Wherein, the decoding includes: receiving a bitstream, the bitstream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in a picture; and The compressed image data is decoded according to the region information to reconstruct the picture. 42. A method according to clause 41, wherein decoding the compressed image data according to the region information to reconstruct the picture comprises: Generating a first region of the picture based on the decoded image data; and A second region of the picture is generated by filling pixels. 43. The method of clause 41, wherein decoding the compressed image data based on the region information to reconstruct the picture further comprises: A position of an instance in the picture is restored according to the region information. 44. A non-transitory computer-readable storage medium storing a bitstream, the bitstream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in a picture, and the bitstream is processed by a method comprising: The compressed image data is decoded according to the region information to reconstruct the picture. 45. The medium of clause 44, wherein decoding the compressed image data based on the region information to reconstruct the picture comprises: Generating a first region of the picture based on the decoded image data; and A second region of the picture is generated by filling pixels. 46. ​​The medium of clause 44, wherein decoding the compressed image data based on the region information to reconstruct the picture further comprises: A position of an instance in the picture is restored according to the region information. 47. An image data processing device, comprising: a memory storing an instruction set; and One or more processors are configured to execute the instruction set to cause the apparatus to perform operations including: receiving a bitstream, the bitstream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in a picture; and The compressed image data is decoded according to the region information to reconstruct the picture. 48. An apparatus according to clause 47, wherein decoding the compressed image data according to the region information to reconstruct the picture comprises: Generating a first region of the picture based on the decoded image data; and A second region of the picture is generated by filling pixels. 49. The apparatus of clause 47, wherein decoding the compressed image data based on the region information to reconstruct the picture further comprises: A position of an instance in the picture is restored according to the region information. 50. A non-transitory computer-readable storage medium storing a set of instructions, which, when executed by a computer, causes the computer to perform a method comprising: receiving a bitstream, the bitstream comprising compressed image data having region information, wherein the region information indicates a region represented by the compressed image data in a picture; and The compressed image data is decoded according to the region information to reconstruct the picture. 51. The medium of clause 50, wherein decoding the compressed image data based on the region information to reconstruct the picture comprises: Generating a first region of the picture based on the decoded image data; and A second region of the picture is generated by filling pixels. 52. The medium of clause 50, wherein decoding the compressed image data based on the region information to reconstruct the picture further comprises: A position of an instance in the picture is restored according to the region information.

[0165] It should be noted that relational terms such as "first" and "second" herein are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "comprise", "have", "contain", and "include" and other similar forms are intended to be synonymous and open-ended, in that one or more items following any of these words are not intended to be an exhaustive list of such one or more items, or to be limited to the one or more items listed.

[0166] As used herein, unless otherwise specifically stated, the term "or" encompasses all possible combinations unless not feasible. For example, if a database is specified to include either A or B, then unless otherwise specifically stated or not feasible, the database may include A, or B, or A and B. As a second example, if a database is specified to include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0167] It should be understood that the above embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer readable medium. When executed by a processor, the software can execute the disclosed method. The computer unit and other functional units described in the present disclosure can be implemented by hardware, or software, or a combination of hardware and software. It should also be understood by those of ordinary skill in the art that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0168] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain adaptations and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art by considering the description and practice of the present disclosure disclosed herein. The description and embodiments are intended to be considered as exemplary only, and the true scope and nature of the present disclosure are indicated by the appended claims. The order of steps shown in the figures is for illustrative purposes only and is not intended to be limited to any particular order of steps. Therefore, it will be appreciated by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0169] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are employed, they are used only in a general and descriptive sense and not for the purpose of limitation.

Claims

1. A method for processing image data, comprising: receiving a video sequence; encoding one or more input pictures associated with the video sequence; as well as Generate a bit stream, The encoding includes: Performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representative of the extracted region is compressed.

2. The method according to claim 1, wherein: The extraction area includes one or more pixels adjacent to the merge mask.

3. The method according to claim 1 or 2, wherein: Extracting a region including the merge mask from the input image comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture.

4. The method according to claim 3, wherein: A sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of the following: a quantization parameter QP used to compress the input picture, or The definition of the input image.

5. The method according to claim 1 or 2, wherein: Extracting a region including the merge mask from the input image comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the dilation mask are extracted from the input picture.

6. The method according to claim 5, wherein: The merge mask is expanded by applying a convolution kernel to the merge mask, wherein a size of the convolution kernel is determined based on at least one of the following: a quantization parameter QP used to compress the input picture, or The definition of the input image.

7. The method according to claim 1 or 2, wherein: Extracting a region including the merge mask from the input image comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture.

8. The method according to claim 1 or 2, wherein: Extracting a region including the merge mask from the input picture further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred.

9. The method according to claim 1 or 2, wherein: Extracting a region including the merge mask from the input picture further comprises: Determining whether to blur at least a portion of the extraction area based on a quantization parameter QP used to compress the input picture; and In response to the QP being greater than a predetermined threshold, blurring the portion of the extraction region.

10. The method according to claim 9, wherein: The portion of the extracted region does not overlap with the merge mask.

11. The method according to any one of claims 1 to 10, wherein: The instance segmentation is performed by a convolutional neural network (CNN).

12. A non-transitory computer-readable storage medium storing a bit stream, wherein a method of generating the bit stream comprises: Performing instance segmentation on an input image to generate one or more segmentation masks; combining one or more of the segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input image; as well as Image data representing the extracted region is compressed to generate the bitstream.

13. The medium according to claim 12, wherein: The extraction area includes one or more pixels adjacent to the merge mask.

14. The medium according to claim 12 or 13, wherein: Extracting a region including the merge mask from the input image comprises: In the input picture, identifying one or more blocks that do not overlap with the merge mask; and A region that does not include the one or more blocks is extracted from the input picture.

15. The medium according to claim 14, wherein A sliding window is used to identify the one or more blocks, and a size of the sliding window is determined based on at least one of the following: a quantization parameter (QP) used to compress the input picture, or The definition of the input image.

16. The medium according to claim 12 or 13, wherein: Extracting a region including the merge mask from the input image comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; and The first region and the merge mask are extracted from the input picture.

17. The medium according to claim 16, wherein The merge mask is expanded by applying a convolution kernel to the merge mask, and a size of the convolution kernel is determined based on at least one of the following: a quantization parameter (QP) used to compress the input picture, or The definition of the input image.

18. The medium according to claim 12 or 13, wherein: Extracting a region including the merge mask from the input image comprises: dilating the merge mask to determine a first region extending from a boundary of the merge mask; In the input picture, identifying one or more blocks that do not overlap with the first region or the merge mask; and A region that does not include the one or more blocks is extracted from the input picture.

19. The medium according to claim 12 or 13, wherein: Extracting a region including the merge mask from the input picture further comprises: In the extracted area, determining a second area that does not overlap with the merge mask; and The second area is blurred.

20. An image data processing device, comprising: A memory storing an instruction set; as well as One or more processors are configured to execute the instruction set to cause the apparatus to perform operations including: Performing instance segmentation on an input image to generate one or more segmentation masks; combining the one or more segmentation masks to generate a merge mask; Extracting a region including the merge mask from the input picture; and Image data representative of the extracted region is compressed.

21. A computer program product comprising: Computer program instructions, the computer program instructions causing a computer to execute the method according to any one of claims 1 to 11.

22. A computer program, wherein The computer program enables a computer to execute the method according to any one of claims 1 to 11.

Images