Image data coding method and system

Abstract

Methods and systems are provided for encoding and decoding image data for use in machine vision applications. According to certain embodiments, an exemplary encoding method may include determining one or more regions of interest (ROIs) in an image; modifying the image based on the one or more ROIs, where modifying the image includes at least one of: modifying the first ROI based on a portion of the image surrounding the first ROI; reducing the content of the second ROI; or reducing the size of the third ROI; and compressing the modified image.

Description

Technical Field

[0001] The present disclosure relates generally to image data processing, and more particularly, to methods and systems for encoding and decoding image data used by machine vision applications. Background Art

[0002] Limited communication bandwidth poses a challenge to transmitting images or videos (hereinafter collectively referred to as "image data") with ideal quality. Although various encoding technologies have been developed to compress image data before transmitting it, the growing demand for large amounts of image data has brought continuous pressure to image / video systems, especially mobile devices that often need to work in networks with limited bandwidth.

[0003] In addition, with the rise of machine learning technology and machine vision applications, the amount of videos and images consumed by machines is growing rapidly. Typical use cases include smart transportation, smart city technology, smart content management, etc., which combine machine vision tasks such as object detection, instance segmentation, and object tracking. Machine vision applications extract and analyze features from image data. Since the feature information is very different from traditional image or video data, the encoding technology and solutions used by machines may be different from traditional applications for human viewing to optimize performance. Therefore, it is necessary to develop encoding methods and systems suitable for machine use. Summary of the invention

[0004] Embodiments of the present disclosure relate to methods and systems for encoding and decoding image data used in machine vision applications. In some embodiments, a computer-implemented encoding method is provided. The method may include: determining one or more regions of interest (ROIs) in an image; modifying the image based on the one or more ROIs, wherein modifying the image includes at least one of the following: modifying the first ROI based on a portion of the image surrounding the first ROI; reducing the content of the second ROI; or reducing the size of the third ROI; reducing the content of the background remaining after selecting all ROIs, and compressing the modified image.

[0005] In some embodiments, a device for encoding image data is provided. The device includes a memory storing instructions; and one or more processors for executing the instructions to cause the device to perform operations including: determining one or more regions of interest (ROIs) in an image; modifying the image based on the one or more ROIs, wherein modifying the image includes at least one of the following: modifying a first ROI based on a portion of the image surrounding the first ROI; reducing the content of a second ROI; or reducing the size of a third ROI; reducing the content of a background remaining after selecting all ROIs; and compressing the modified image.

[0006] In some embodiments, a computer-implemented decoding method is provided. The method may include: decoding a bitstream including metadata describing: one or more regions of interest (ROIs) in an uncompressed image, a background in the uncompressed image, and modifications made to the uncompressed image; determining a reconstructed image; and restoring the uncompressed image by restoring the content or size of the ROI based on the metadata.

[0007] In some embodiments, a device for decoding image data is provided. The device includes a memory storing instructions; and one or more processors for executing the instructions to cause the device to perform operations including: decoding a bitstream including metadata describing: one or more regions of interest (ROIs) in an uncompressed image, and modifications made to the uncompressed image; determining a reconstructed image; and restoring the uncompressed image by restoring the content or size of the ROI based on the metadata.

[0008] Aspects of the disclosed embodiments may include a non-transitory tangible computer-readable medium storing software instructions that, when executed by one or more processors, are used and capable of performing one or more of methods, operations, etc. consistent with the disclosed embodiments. In addition, aspects of the disclosed embodiments may be performed by one or more processors configured as special-purpose processors based on software instructions programmed with logic and instructions that, when executed, perform one or more operations consistent with the disclosed embodiments.

[0009] Other purposes and advantages of the disclosed embodiments will be set forth in part in the following description, and in part will be apparent from the description, or may be learned through the practice of the embodiments. The purposes and advantages of the disclosed embodiments may be realized and obtained through the elements and combinations set forth in the claims.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating an exemplary architecture of a video coding machine (VCM) encoder consistent with embodiments of the present disclosure.

[0012] Figure 2 It is shown that the embodiment of the present disclosure is consistent with Figure 1 Schematic diagram of an exemplary image inpainting process performed by a VCM encoder.

[0013] Figure 3 It is shown that the embodiment of the present disclosure is consistent with Figure 1 Schematic diagram of an exemplary image simplification process performed by a VCM encoder.

[0014] Figure 4 It is shown that the embodiment of the present disclosure is consistent with Figure 1 Schematic diagram of an exemplary image retargeting process performed by a VCM encoder.

[0015] Figure 5 is a flow chart of an exemplary encoding method consistent with an embodiment of the present disclosure.

[0016] Figure 6 is a flow chart of an exemplary image restoration method consistent with an embodiment of the present disclosure.

[0017] Figure 7 is a flow chart of an exemplary image simplification method consistent with an embodiment of the present disclosure.

[0018] Figure 8 is a schematic diagram showing an object surrounded by adjacent bands consistent with an embodiment of the present disclosure.

[0019] Fig. 9 is a schematic diagram showing a plurality of objects surrounded by corresponding adjacent bands consistent with an embodiment of the present disclosure.

[0020] Fig.10 is a schematic diagram illustrating an exemplary process of performing redirection and de-redirection of an image consistent with an embodiment of the present disclosure.

[0021] Fig.11 is a flow chart of an exemplary image redirection method consistent with an embodiment of the present disclosure.

[0022] Fig.12 is a schematic diagram showing an input image divided into a plurality of blocks by a grid consistent with an embodiment of the present disclosure.

[0023] Fig.13A It is shown that the embodiment consistent with the present disclosure is Fig.12 Schematic diagram of the process of allocating initial horizontal scaling factors to multiple blocks divided in .

[0024] Fig. 13B It is shown that the embodiment consistent with the present disclosure is Fig.12 Schematic diagram of the process of allocating initial vertical scaling factors to multiple blocks divided in .

[0025] Fig.14A It is shown that the embodiment consistent with the present disclosure is Fig.12 Schematic representation of the process of determining the agreed horizontal downscaling factor for each column in a block.

[0026] Fig. 14B It is shown that the embodiment consistent with the present disclosure is Fig.12 Schematic diagram of the process of determining a commonly agreed vertical reduction factor for each row of blocks in FIG.

[0027] Fig.15 is a schematic diagram showing a process of performing reduction and enlargement of a grid consistent with an embodiment of the present disclosure.

[0028] Fig.16 is a schematic diagram illustrating a process of performing redirection and de-redirection of an image consistent with an embodiment of the present disclosure.

[0029] Fig.17 is a block diagram illustrating an exemplary architecture of a VCM decoder consistent with embodiments of the present disclosure.

[0030] Fig.18 is a flow chart of an exemplary decoding method consistent with an embodiment of the present disclosure.

[0031] Fig.19 is a schematic diagram illustrating an exemplary process of performing image inpainting and region of interest (ROI) synthesis consistent with an embodiment of the present disclosure.

[0032] Fig. 20 is a block diagram of an exemplary apparatus for encoding or decoding image data consistent with an embodiment of the present disclosure.

[0033] Fig.21A is a graph of the average precision of detection for example still images decoded from a Versatile Video Coding (VVC) bitstream.

[0034] Fig.21B is a plot of the average precision of instantaneous segmentation of example still images decoded from a VVC bitstream. DETAILED DESCRIPTION

[0035] Reference will now be made in detail to the exemplary embodiments, examples of which are shown in the accompanying drawings. The following description refers to the accompanying drawings, wherein, unless otherwise specified, the same numbers in different figures represent the same or similar elements. The embodiments set forth in the following description of the exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, these embodiments are merely examples of devices and methods consistent with the aspects related to the present invention described in the appended claims.

[0036] The present disclosure provides a coding and decoding architecture for compressing image data (including image data for machine vision applications). The disclosed architecture includes video coding for machines (VCM) technology, which is intended to compress the features of tasks performed by machines (such as video object detection, event analysis, object tracking, etc.). The input (original) and output (decoded) of the disclosed VCM architecture are static images or videos (i.e., image sequences). The architecture includes an encoder, any type of communication / storage system, and a decoder. The architecture is intended to represent the input image or video at the lowest possible bit rate, but ensure that the efficiency of the machine vision tasks performed on the output (decoded) image or video is as high as possible. The architecture is suitable for compressing image data used by various machine vision tasks. For example, the architecture can be used to improve the average accuracy of detection or segmentation performed on the output image or video. As another example, it can be used to improve tracking performance in the output video.

[0037] Figure 1 1 is a block diagram illustrating an exemplary architecture of a video coding machine (VCM) encoder 100 consistent with embodiments of the present disclosure. The encoder 100 is capable of compressing an image (also referred to as a "picture" or "frame"), multiple images, or a video. The image is a static picture. The multiple images may be related or unrelated in space or time. A video includes a set of images arranged in time order.

[0038] like Figure 1 As shown, the encoder 100 may include a region of interest (ROI) determination and classification module 102, an image restoration module 110, an image simplification module 120, an image redirection module 130, a ROI metadata formation module 140, and an encoding (i.e., compression) module 150. The input of the encoder 100 may include one or more input images 101, and the output of the encoder 100 may include a coded bitstream 160 and / or ROI metadata 162. Consistent with the disclosed embodiments, the encoder 100 may be scalable—depending on actual use cases, some of the above modules may not exist in a single implementation. For example, in a single use case, not all of the modules in the image restoration module 110, the image simplification module 120, and the image redirection module 130 are required.

[0039] Specifically, the determination and classification module 102 analyzes the input image 101 and determines one or more ROIs where the object of interest may be located. The ROI is an area in the input image that contains content that is critical to the performance or accuracy of machine vision applications (such as object detection and tracking). In other words, changing or removing the ROI will reduce the performance or accuracy of the machine vision application. Consistent with the disclosed embodiments, the ROI may also include one or more smaller ROIs, and / or part or all of the background of the input image. In some embodiments, detecting an object from an input image includes three stages: image segmentation (i.e., object isolation or recognition), feature extraction, and object classification. In the image segmentation stage, the encoder may execute an image segmentation algorithm to divide the input image into multiple segments, such as sets of pixels, each of which represents a portion of the input image. The image segmentation algorithm may assign a label (i.e., a category) to each set of pixels so that pixels with the same label have certain common features. The encoder may then group the pixels according to their labels and designate each group as a different object.

[0040] After segmenting the input image, i.e., after identifying at least one object, the encoder can extract features from the input image, i.e., features of the object. These features can be based on the size of the object or based on the shape of the object, such as the area of ​​the object, the length and width of the object, the distance around the perimeter of the object, the rectangularity of the object shape, the circularity of the object shape. The features associated with each object can be combined to form a feature vector.

[0041] In the classification stage, the encoder can classify each identified object based on the feature vector associated with the corresponding object. The feature value can be regarded as the coordinate of a point in N-dimensional space (one feature value represents a one-dimensional space, two features represent a two-dimensional space, and so on), and classification is the process of determining the subspace of the feature space to which the feature vector belongs. Since each subspace corresponds to a different object, classification completes the task of object recognition. For example, the encoder can classify the identified object into one of multiple predetermined categories such as people, faces, animals, cars, or vegetation. In some embodiments, the encoder can perform classification based on the model. The model can be trained using a pre-labeled training data set associated with a known object. In some embodiments, the encoder can perform classification using machine learning techniques such as convolutional neural networks (CNN) and deep neural networks (DNN).

[0042] The result 103 of ROI determination and classification is used to assist the operation of the image restoration module 110, the image simplification module 120, and / or the image redirection module 130. Specifically, based on the result 103, the image restoration module 110 removes certain less important objects from the input image 101 and replaces these objects with image elements with simple content so that no holes are left in the image. The removed less important objects can be small objects or objects that appear repeatedly in the image. As described below in relation to the decoder, the removed objects can be synthesized by the decoder and added back to the decoded image. In an exemplary embodiment, the image area originally occupied by the removed objects can be restored with simple content represented by a small number of bits. Figure 2 2 is a schematic diagram illustrating an image restoration process 200 according to some exemplary embodiments consistent with the present disclosure. Figure 2 As shown, the determination and classification module 102 determines objects 212a, 214a, 216a, and 218a from the original image 210a. The image restoration module 110 removes the objects 212a, 214a, 216a, and 218a and restores the affected image area using the content and color surrounding the removed objects to generate a restored image 210b.

[0043] In some embodiments, the image inpainting module 110 is informed of the removed content (e.g., object) in the bitstream so that the content can be restored at the decoder side. For example, the notification may include labels from a catalog of common feature objects (e.g., car, man, woman, dog, cat, car, etc.). Another example is to inform the location of similar objects in the same image or video by providing a displacement vector pointing to the similar object or providing a potential representation of the object replaced by the inpainted area. Other suitable methods may also be used to inform the removed content.

[0044] The image simplification module 120 is intended to produce content that can be represented with fewer bits after compression, without significantly reducing the efficiency of the machine vision tasks performed on the decoded (decompressed) images / video. Figure 3 is a schematic diagram illustrating an image simplification process 300 according to some exemplary embodiments consistent with the present disclosure. Figure 3As shown, the determination and classification module 102 determines objects 312 and 314 from the original image 310a. The image simplification module 120 also determines a buffer 322, a buffer 324, and a background area 330 from the original image 310a. The buffer 322 surrounds the object 312, the buffer 324 surrounds the object 314, and the background area 330 is the remaining portion of the original image 310a. The image simplification module 120 simplifies the content of the background area 330 by reducing the details in the background area 330 so that the simplified background area 330 can be represented with fewer bits. In contrast, the objects 312, 314 and the buffers 322, 324 remain unchanged or are only slightly simplified to maintain the accuracy of the machine vision task. For example, for larger objects, the interior portion of the object can be simplified without negatively affecting the performance of downstream machine vision tasks. As Figure 3 As shown, in the simplified image 310b, the background area 330 is significantly simplified, while the objects 312, 314 and the buffers 322, 324 are substantially unchanged. The goal of image simplification is to produce an image or video that can be represented with fewer bits (in a compressed representation), while the objects in the image or video can still be well analyzed by machine vision tasks (recognition, detection, tracking, etc.).

[0045] The image retargeting module 140 reduces the number of rows and columns of the input image or video frame according to the result 103 of the ROI determination and classification 102. Specifically, when one or more rows or columns of the input image do not contain samples of the object, the image retargeting module 140 may remove the one or more rows or columns. Figure 4 4 is a schematic diagram illustrating an image redirection process 400 according to some exemplary embodiments consistent with the present disclosure. Figure 4 As shown, the determination and classification module 102 determines ROIs 412 and 414 from the input image 410. The image redirection module 140 determines rows 422 and columns 424 in the image 410 that are not co-located or overlapped with the ROIs 412 and 414. The image redirection module 140 redirects the image 410 by removing the rows 422 and columns 424, so that the image 410 is reduced and the ROIs 412, 414 occupy a larger portion of the redirected image. In some embodiments, a local low-pass (i.e., anti-aliasing) filter is performed before removing the rows and columns. In some embodiments, the rows or columns containing the sample of the object can be reduced based on the analysis of the object without causing a reduction in the ability of the machine vision task to recognize the object.

[0046] Back to Figure 1After the input image 101 is processed by one or more of the image restoration module 110, the image simplification module 120, and the image redirection module 130, the encoding module 150 encodes (i.e., compresses) the output. The restoration, simplification, and redirection can be applied regardless of which image / video coding (compression) technology the encoding module 150 uses. For example, the encoding module 150 can operate according to any video coding standard such as advanced video coding (AVC), high efficiency video coding (HEVC), versatile video coding (VVC), AOMedia video 1 (AV1), joint photographic experts group (JPEG), moving picture experts group (MPEG), etc. The encoding module 150 outputs an encoded bitstream 160 including a compressed image / video.

[0047] The metadata forming module 140 formats the ROI metadata 162 for transmission as auxiliary information outside the bitstream 160 generated by the encoding module 150. The ROI metadata 162 may include information describing the input image processing performed by the repair module 110, the simplification module 120, and / or the redirection module 130. Various methods of transmitting the ROI metadata 162 may be used to ensure frame synchronization between the ROI metadata 162 and the encoded bitstream 160. For example, the ROI metadata 162 may be transmitted in the image parameter set of the encoded bitstream 160 by making necessary modifications to the existing image parameter set syntax used in the AVC, HEVC, and VVC standards of ISO, IEC, and ITU. Specifically, in order to avoid the ROI metadata 162 being transmitted out of phase with the encoded bitstream 160, the encoder and decoder may synchronize the ROI metadata 162 with the encoded bitstream 160 using a picture order count (POC). For other known standards, the transmission of the ROI metadata 162 may be organized in a similar manner without intervening in the high-level syntax of the encoded bitstream 160. Consistent with the disclosed embodiments, the format of the ROI metadata 162 is standardized so that it can be decoded unambiguously.

[0048] In accordance with the disclosed embodiments, encoder 100 may be used to compress still images or video. For video, the frame structure (e.g., a sequence of I, P, B frames) is configured in accordance with the corresponding temporal configurations of repair, simplification, and redirection. As is known in the art, a frame that is encoded without reference to another frame (i.e., it is its own reference frame) is called an "I frame." A frame that is encoded using a previous frame as a reference frame is called a "P frame." A frame that is encoded using a previous frame and a future frame as reference frames (i.e., the reference is "bidirectional") is called a "B frame."

[0049] Consistent with the disclosed embodiments, encoder 100 is scalable because not all of image restoration module 110, image simplification module 120, and image retargeting module 130 need to be present in every implementation. Omitting one or two of these modules can simplify the system and reduce complexity.

[0050] Next, the signal path in the encoder 100 is described. The input image 101 is an uncompressed representation of a still image or video, which can be monochrome or color, standard dynamic range (SDR), or high dynamic range (HDR), in any color space, etc. The input image 101 can be represented by samples including values ​​of color component samples.

[0051] Signal 115 is uncompressed (i.e., unencoded) image data. Signal 115 is generated by inpainting regions of input image 101 corresponding to objects of less interest (e.g., less important or typical objects) described in ROI metadata 162. For example, objects removed by the inpainting process may be described by an object index to a directory (database) that includes a list of predetermined objects. As another example, the removed object may be described by a pointer (in time or space) to the same or similar object in a previous frame or at another location in the current frame. As another example, the removed object may be described by a potential representation generated by a neural network for the object. Other object description methods may also be used.

[0052] Signal 125 is uncompressed (ie, unencoded) image data. Signal 125 is generated by simplifying the content in certain areas of the image, such as background areas that do not contain objects.

[0053] Signal 135 is uncompressed (ie, unencoded) image data. Signal 135 is generated by redirecting an image. Because redirection removes certain rows and / or columns from the original image, the redirected image is smaller than the original image. Therefore, the redirected image contains fewer samples than the original image.

[0054] Redirection parameters 132 are output by the image redirection module 130. The redirection parameters 132 describe the number and positions of rows and columns that are removed during the redirection process.

[0055] The coded bitstream 160 is generated by the encoding module 150. It can be a traditional bitstream generated based on any existing standard such as AVC, HEVC, VVC, AV1, JPEG, MPEG, etc. Depending on whether the encoder 100 includes an image restoration module 110, an image simplification module 120, or an image redirection module 130, the coded bitstream 160 can be generated by the coded signal 115, 125, or 135.

[0056] The ROI metadata 162 is transmitted as auxiliary information outside the coded bitstream 160 generated by the encoding module 150. Any transmission mechanism can be used to transmit the ROI metadata 162 as long as it can ensure frame synchronization between the ROI metadata 162 and the coded bitstream 160. For example, the ROI metadata 162 can be signaled in the image parameter set used in the AVC, HEVC, and VVC standards of ISO, IEC, and ITU, respectively. However, this implementation requires modifying the high-level syntax of these standards. In some embodiments, in order to avoid the ROI metadata 162 being transmitted out of phase with the coded bitstream 160, the transmission of the ROI metadata 162 and the coded bitstream 160 can be synchronized using the picture order count (POC) in the encoder and decoder. For other known standards, the transmission of the ROI metadata 162 can be organized in a similar manner without intervening the high-level syntax of the coded bitstream 160. Consistent with the disclosed embodiments, the format of the ROI metadata 162 can be standardized so that it can be decoded unambiguously.

[0057] The ROI metadata 162 may include three types of metadata, as described below.

[0058] Metadata Category 1 - Metadata defining ROIs that correspond to potential objects that are important for the machine vision task at the decoder side. ROIs can be defined in various ways depending on the specific implementation. For example, the ROI can be defined as a bounding box around the object. Other known methods can also be used to segment the ROI from the rest of the image. The shape of the ROI can be square, rectangular, polygonal, etc.

[0059] Metadata category 2—Metadata describing the redirection, which can be used by a decoder to implement reverse redirection.

[0060] Metadata Category 3 - Metadata defining object representations of objects removed and replaced by image inpainting. Removed objects may be defined in any reasonable manner, such as an index indicating the object type and / or basic features of the object (e.g., size and / or orientation of the removed object). Removed objects may also be described by other potential representations.

[0061] Figure 5 is a flowchart of an exemplary encoding method 500 for compressing image data consistent with an embodiment of the present disclosure. For example, the method 500 may be performed by the encoder 100 ( Figure 1 ) to execute. Figure 5 As shown, the method 500 includes the following steps 502 to 506.

[0062] At step 502, the encoder determines one or more regions of interest (ROIs) in the image. In some embodiments, the encoder may detect one or more objects from the image, and then determine one or more ROIs based on the objects, each ROI containing one or more of the detected objects. In some embodiments, each ROI may also include a buffer around the objects contained in the ROI. In some embodiments, one or more ROIs may be bounding boxes in the shape of a triangle, square, rectangle, polygon, circle, etc.

[0063] At step 504, the encoder modifies the image by performing at least one of restoration, simplification, or re-orientation of the image based on the one or more ROIs. Figure 6 , Figure 7 ,and Fig.10 Exemplary image inpainting, simplification, and retargeting methods are described. Image inpainting is used to remove ROIs that are less important to the performance of downstream machine vision tasks. Image simplification is used to simplify the content of image regions that do not include ROIs. Image retargeting is used to increase the proportion or weight of ROIs in the entire image.

[0064] At step 506, the encoder compresses the modified image. The encoder may perform compression (i.e., encoding) according to any image data encoding standard, such as AVC, HEVC, VVC, AV1, JPEG, MPEG, etc., and generate an encoded bitstream. In some embodiments, the encoder may also include metadata about the ROI (e.g., Figure 1 The ROI metadata 162 in the bitstream is encoded into the bitstream for decoding. For example, the ROI metadata can be sent by signal in the header part of the bitstream (e.g., the picture parameter set).

[0065] Figure 6 600 is a flowchart of an exemplary image restoration method 600 consistent with an embodiment of the present disclosure. For example, the method 600 may be performed by the encoder 100 ( Figure 1 ) to execute. Figure 6 As shown, the method 600 includes the following steps 602 to 608.

[0066] At step 602, the encoder determines a first ROI to be removed from an input image.

[0067] At step 604, the encoder generates a repair image element. The repair image element is a portion of the image used to replace the first ROI. The content of the repair image element is simpler than the first ROI. Therefore, the number of bits used to represent the repair image element is less than the number of bits required to represent the first ROI. In some embodiments, the repair image element can be generated based on one or more samples surrounding the first ROI. For example, Figure 2 As shown, objects 212a to 212d in the original image 210a are replaced by inpainted image elements that are similar to surrounding areas of the corresponding objects.

[0068] Back to Figure 6 In step 606, the encoder replaces the first ROI with the repaired image element to generate a repaired image. Figure 2 As shown, in the inpainted image 210b, the regions originally occluded by the objects 212a to 212d are inpainted by samples similar to the samples surrounding these objects.

[0069] Back to Figure 6 , the encoder generates metadata describing the removed first ROI at step 608. For example, the metadata may include an index associated with the category of the removed ROI, a feature vector describing the characteristics of the removed ROI, and / or a shift vector pointing to a non-removed ROI in the image that is similar to the removed first ROI.

[0070] Figure 7 700 is a flowchart of an exemplary image simplification method 700 consistent with an embodiment of the present disclosure. For example, the method 700 may be performed by the encoder 100 ( Figure 1 ) to execute. Figure 7 As shown, method 700 includes the following steps 702 to 704.

[0071] At step 702, the encoder determines an object in an image, a buffer area surrounding the object, and a background area surrounding the buffer area. Figure 8 As shown, the encoder detects an object 810 in the image 800 and defines a buffer region 820 around the object 810. In the following description, "buffer region" is also referred to as "band", and the two are used interchangeably. Consistent with the disclosed embodiments, the background region is an image region that does not include any object or band. In some embodiments, after determining the object and band from the image, the encoder determines the rest of the image as the background region. For example, in Figure 8 , after determining the object 810 and the buffer 820 , the encoder determines the rest of the image 800 as a background area 830 surrounding the buffer 820 .

[0072] although Figure 8Only one object is shown, but the encoder can detect multiple objects from the image and define multiple bands surrounding the multiple objects respectively. Fig. 9 As shown, the encoder first identifies object 910 and object 920 in image 900. The encoder then defines a buffer 912 around object 910 and defines a buffer 922 around object 920. After determining these objects and buffers, the encoder determines the rest of image 900 as background region 930, which surrounds buffers 912 and 922. Figure 8 and Fig. 9 The bands (i.e., buffers) are shown to completely surround their corresponding objects, but in the disclosed embodiments, the bands may also partially surround the associated objects. Therefore, the term "surrounding" as used in this disclosure means completely or partially surrounding an image area (e.g., an image area occupied by an object). In addition, some objects may not have corresponding bands (i.e., buffers). That is, the size of the bands associated with these objects is zero.

[0073] In some embodiments, in order to detect or identify an object in an image, the encoder may perform an image segmentation algorithm to divide the input image into multiple segments, i.e., sets of pixels, each of which represents a portion of the image. The image segmentation algorithm may assign a label (i.e., a category) to each set of pixels so that pixels with the same label have certain common features. The encoder may then determine whether each image segment corresponds to an object, such as a human body, an animal, a car, or vegetation. According to some embodiments, image segmentation classifies each pixel in an image into one of multiple categories. The encoder may perform classification based on a model. A pre-labeled training data set associated with a known object may be used to train the model. In some embodiments, the encoder may perform classification using machine learning techniques such as convolutional neural networks (CNNs) and deep neural networks (DNNs).

[0074] In the disclosed embodiments, the width of the band can be defined in a variety of ways. For example, the band around the object can have a uniform width or a variable width. For another example, when multiple objects and bands are determined in the image, the widths of the multiple bands can be the same or different. The present disclosure does not limit the manner in which the width of the band is defined.

[0075] In some embodiments, the encoder determines the width of the band based on the feature size of the object surrounded by the band. For example, the encoder can define the width of the band as a percentage of the feature size according to the following formula 1:

[0076] BW=kw·L, (1)

[0077] In Formula 1, BW represents the width of the band. L represents the characteristic size of the object surrounded by the band. kw represents the width coefficient, which can be a constant (e.g., 0.15) or a variable. The band width BW and / or the characteristic size L can be defined and measured using the Euclidean metric, the Manhattan metric, or any other suitable distance metric. As is known in the art, the Euclidean metric defines the distance between two points as being equal to the length of a straight line between the two points, while the Manhattan metric defines the distance between two points as being equal to the sum of the absolute differences of the Cartesian coordinates of the points.

[0078] In addition, the above-mentioned band width BW and / or feature size L can be defined based on the actual size of the object. Unlike the apparent size of the object displayed in the image, the actual size of the object is its pre-adjusted size based on the image. In some embodiments, in order to determine the actual size of the object, the encoder can determine the apparent size of the object in the image and the distance between the object and the image sensor (e.g., a camera) that captures the image. Various methods can be used to measure the distance between the object and the camera. In one example, the distance can be measured by a distance sensor. Exemplary distance sensors include, but are not limited to, infrared sensors or laser sensors designed to measure the distance based on the time it takes for a light signal to propagate between the object and the camera. As another example, the distance can be determined using the lens formula shown in Equation 2:

[0079]

[0080] In Equation 2, OD is the distance between the object and the center of the camera lens, CL is the distance from the center of the camera lens to the image, and f is the focal length of the camera lens. Equation 2 is based on assumptions and therefore applies to situations where OD is much larger than f. After determining the apparent size of the object in the image and the distance between the object and the camera, the encoder can determine the actual size of the object based on Equation 3:

[0081]

[0082] In Formula 3, AS represents the actual size of the object, IS represents the apparent size of the object in the image, OD represents the distance between the object and the center of the camera lens, and f represents the focal length of the camera lens. The above example of determining the actual size of the object is for illustration only and is not intended to limit the disclosed embodiments.

[0083] In some embodiments, the feature size of an object can be defined as the maximum size of the object. For example, the feature size L in Equation 1 can be defined as the maximum size of an object. For another example, when the encoder runs a machine vision algorithm that uses a bounding box to track an object, the maximum object size L can be defined as the maximum size of the bounding box associated with the object. Figure 8 In the example shown, the encoder may determine the maximum size of the object 810 as a rectangular bounding box ( Figure 8 ), which corresponds to the distance from a person's head to his feet. Fig. 9 In the example shown, the encoder can use the bounding box ( Fig. 9 910 and 920) to find the corresponding maximum size of the objects 910 and 920. The above-mentioned way of defining the feature size L is only an example and is not intended to limit the disclosed embodiments.

[0084] In some embodiments, the band width BW may depend on the object class associated with the object. As described above, image segmentation techniques may be used to identify objects in an image. In doing so, the image segmentation techniques may classify image segments into predetermined classes. Some predetermined classes are used to describe the class of objects. The object class may be defined based on the object type (e.g., car, building, person, face, toy, etc.). In some embodiments, the parameter kw in Equation 1 may be determined based on the object class, thereby assigning wider or narrower bands to various types of objects (e.g., face, building, etc.). For example, Figure 8 and Fig. 9 As shown, the encoder may classify object 810 as “human” and objects 910 and 920 as “pets.” Based on the classification, the encoder may assign a wider band to object 810 than to objects 910 or 920 .

[0085] In some embodiments, the band width BW may be a variable that depends on the local properties of the object. For example, the parameter kw in equation 1 may be a variable that depends on the edge position of the object. Figure 8 In the example, the encoder can increase the value of parameter kw in areas with rich texture (such as animal fur, human face) or large curvature (such as human joints).

[0086] The width coefficient kw in formula 1 can be a variable that depends on the position of the edge of the object or on the local texture or curvature of the edge of the object. For example, the value of the width coefficient kw can be set to be proportional to the value of the local curvature. Alternatively or additionally, the width coefficient kw can be based on the color of the object and / or the color of the background area. For example, the value of the width coefficient kw can be set to be proportional to the difference between the hue of the object and the hue of the background area.

[0087] Back to Figure 7 In step 704, the encoder simplifies at least the content of the background region. Modifying the image data reduces the number of bits representing the compressed image. Specifically, the encoder may modify the image data by simplifying the content of the background region to reduce the number of bits representing the background region in the compressed image.

[0088] The encoder can simplify the content of the background area in various ways. In some embodiments, the encoder can perform low-pass filtering on the image data representing at least a portion of the background area. The low-pass filtering can use a constant cutoff frequency. Alternatively, the low-pass filtering can use a variable cutoff frequency that is inversely proportional to the distance from the nearest band.

[0089] In some embodiments, the encoder may modify the image content in the background area, such as by attenuating the image content (i.e., reducing the brightness and color contrast of the image content). For example, if the distance d between a pixel in the background area and the nearest object is less than a predefined value B, the image data representing the pixel in the background area may be modified according to Equations 4 to 6:

[0090] Y′=0.5+(Y-0.5)(1-(d-BW) / B), (4)

[0091] CR′=(CR)(1-(d-BW) / B), (5)

[0092] CB′=(CB)(1-(d-BW) / B) (6)

[0093] In equations 4 to 6, Y is the normalized brightness component of the pixel value before modification, and CB and CR are the normalized chrominance components of the pixel value before modification. The values ​​of Y, CB, and CR satisfy the following ranges: 0≤Y≤1, -0.5≤CB, CR≤0.5. In addition, Y', CB', and CR' represent the corresponding brightness and chrominance values ​​after modification. BW represents the width of the band. On the contrary, if the distance d to the nearest object is greater than BW+B, the image data representing the background area can be modified according to equations 7 to 9:

[0094] Y′=0.5, (7)

[0095] CR'=0, (8)

[0096] CB'=0 (9)

[0097] Furthermore, if the distance d to the nearest object is between B and BW+B (ie, B≤d≤BW+B), the image data is unchanged.

[0098] In some embodiments, the pre-processing may set the values ​​of pixels in the entire background area according to Equations 7 to 9.

[0099] In some embodiments, the encoder may also modify the values ​​of pixels in the object to reduce the number of bits used to represent the object in the compressed image. For example, the encoder may reduce the size of the object according to a scaling factor sc. The value of the scaling factor is less than 1. The encoder may set the value of the scaling factor based on the object class associated with the object. The scaling factor may be included in the encoded bitstream sent to the decoder so that the decoder can use the scaling factor to restore the object to its original size. Alternatively, the value of the scaling factor may be predetermined; therefore, the scaling factor does not need to be sent to the decoder. As another example, the encoder may perform low-pass filtering on image data representing a non-edge portion of an object. A non-edge portion is a portion that does not include the edge of an object, also referred to as an internal portion of an object. Low-pass filtering removes high-frequency components of pixel values, thereby reducing the number of bits used to represent the object. Since non-edge portions are less important for identifying and tracking objects, low-pass filtering non-edge portions may not reduce the efficiency of machine vision tasks.

[0100] In some embodiments, the encoder may also modify the values ​​of the pixels in the band to reduce the number of bits used to represent the band in the compressed image. The encoder may perform the modification in various ways. In some embodiments, if the encoder reduces the size of the object surrounded by the band, the encoder may also reduce the size of the band using the same scaling factor. In some embodiments, the encoder may perform low pass filtering on the image data representing at least a portion of the band. For example, low pass filtering may be performed on the portion of the band that does not touch the edge of the object.

[0101] Fig.10 1 is a schematic diagram illustrating a process 1000 for performing redirection and de-redirection of an image according to some embodiments consistent with the present disclosure. Fig.10As shown, the original image 1010 includes three ROIs: ROI 1012 containing object 1011 (i.e., house), ROI 1014 containing object 1013 (i.e., cloud), and background ROI 1016 covering the entire area of ​​the original image. Background ROI 1016 includes ROI 1012, ROI 1014, and the background of the original image 1010. For example, the scaling factors associated with ROIs 1012, 1014, and 1016 can be set to 1.0, 1.0, and 2.0, respectively. This means that ROIs 1012 and 1014 cannot be reduced, while the width and / or height of ROI 1016 can be reduced by 2.0 times. Therefore, redirecting image 1010 can produce a redirected image 1030, in which ROIs 1032 and 1034 are the same size as ROIs 1012 and 1014, respectively, and ROI 1036 is reduced by 2.0 times from ROI 1016. Image 1030 may also be inversely reoriented to restore the original image size. That is, ROIs 1052, 1054, and 1056 in image 1050 obtained after inverse reorientation are the same size as ROIs 1012, 1014, and 1016 in original image 1010, respectively.

[0102] like Fig.10 As shown in the example of , the reorientation is performed by shrinking one or more ROIs, and the inverse reorientation is performed by stretching (ie, enlarging) one or more ROIs. The shrinking and enlarging are performed based on the position of the one or more ROIs and the rescaling factor.

[0103] Next, a more detailed description of an exemplary image redirection method is provided.

[0104] Fig.11 1 is a flowchart of an exemplary image redirection method 1100 consistent with an embodiment of the present disclosure. For example, the method 1100 may be performed by the encoder 100 ( Figure 1 ) to execute. Fig.11 As shown, method 1100 includes the following steps 1102 to 1110.

[0105] Still refer to Fig.11, in step 1102, the encoder determines a reduction factor associated with one or more ROIs in the input image. Each ROI may be a rectangle or a square or any other shape. Thereafter, the encoder also determines the position information and the reduction factor associated with each ROI. The encoder may determine the position information based on the coordinates of each ROI in the input image and / or the size (e.g., width and height) of each ROI. In addition, the encoder may determine the reduction factor associated with each ROI based on the category of each ROI. Specifically, the encoder may classify one or more ROIs by classifying the objects associated with the ROI. For example, the encoder may determine that each ROI has the same category as its associated object. Therefore, the category is also referred to as an "object / ROI category". The reduction factor of the ROI may be set to be related to its category. This is because the object / ROI size and category may affect the accuracy of the machine vision task. For example, when a neural network is trained to recognize a specific category of objects / ROIs (e.g., faces) of size 224×224 pixels, reasoning about objects / ROIs of the training size (i.e., 224×224 pixels) typically produces the highest accuracy. Therefore, each object / ROI category has a corresponding target object size / ROI that can maximize the accuracy of the machine vision task when analyzing objects / ROIs belonging to that particular category. For example, the following Tables 1 to 3 show the bitrate gains that can be achieved by shrinking various object / ROI categories. The bitrate gain in Table 1 is measured by the detection quality associated with the scoring parameters of the neural network used for the machine vision task. The bitrate gain in Table 2 is measured by the detection quality associated with the best intersection over union (IoU). The bitrate gain in Table 3 is measured by the detection quality associated with the matching classification with a threshold of IoU=0.5. In Tables 1 to 3, the values ​​from 0 to 1 represent the ratio of the reduced object / ROI size to its original size. As shown in Tables 1 to 3, each object / ROI category has a specific size that can maximize the bitrate gain.

[0106] Table 1: Detection quality vs. object / ROI size, relative to the scoring parameters of the neural network

[0107] Table 2: Detection quality vs. object / ROI size, relative to best IoU

[0108] Table 3: Detection quality vs. object / ROI size, relative to matching classification with threshold IoU=0.5

[0109] Based on the relationship between the object / ROI category and its corresponding target object / ROI size, the encoder can determine the minimum allowed size of each ROI in one or more ROIs detected from the input image. The encoder can also determine a reduction factor for each ROI based on the original size of each ROI in the input image and its minimum allowed size. For example, the reduction factor can be set equal to the ratio of the original size to the minimum allowed size.

[0110] At step 1104, the encoder divides the input image into a plurality of blocks using a grid consisting of horizontal and vertical lines. The encoder may create the grid based on the determined one or more ROIs. For example, each horizontal grid line is aligned with at least one horizontal edge of the one or more ROIs, and each vertical grid line is aligned with at least one vertical edge of the one or more ROIs.

[0111] Fig.12 1 is a schematic diagram showing an input image 1210 divided into a plurality of blocks by a grid according to some exemplary embodiments. Fig.12 As shown, the input image 1210 includes three ROIs: ROI 1212, ROI 1214, and background ROI 1216. The reduction factor associated with each ROI may have a horizontal component and a vertical component. The horizontal component is used to reduce the width of the ROI, while the vertical component is used to reduce the height of the ROI. For example, ROI 1212 has S_horizontal=S_vertical=1; ROI 1214 has S_horizontal=2 and S_vertical=3; and ROI 1216 has S_horizontal=5 and S_vertical=6. In addition, the input image 1210 is divided into a plurality of blocks 1250 using a plurality of horizontal grid lines 1220 and a plurality of vertical grid lines 1240. The horizontal grid lines 1220 and the vertical grid lines 1240 may be created based on the edges of the ROIs. Specifically, each horizontal grid line 1220 is aligned with at least one horizontal edge of ROIs 1212 - 1216 , and each vertical grid line 1240 is aligned with at least one vertical edge of ROIs 1212 - 1216 , such that each edge of ROIs 1212 - 1216 is aligned with either a horizontal grid line 1220 or a vertical grid line 1240 .

[0112] Back to Fig.11At step 1106, the encoder determines an allowed downscaling factor for each of the plurality of blocks, respectively. The allowed downscaling factor is a target downscaling factor for downscaling the corresponding block and is less than or equal to a downscaling factor associated with one or more ROIs, such that downscaling the block based on the allowed downscaling factor will satisfy the allowed downscaling levels of all ROIs.

[0113] To illustrate how to determine the allowed reduction factors for each of the plurality of blocks, an exemplary method is described below. Specifically, the method begins with the encoder assigning an initial reduction factor to each of the plurality of blocks. The initial reduction factor of the block can be set equal to the minimum reduction factor associated with the ROI that overlaps the corresponding block. If the ROI uses different reduction factors for its horizontal and vertical edges, the horizontal axis ( Fig.13A ) and the vertical axis ( Fig. 13B ) perform this step independently. Fig.13A is a diagram showing a method for Fig.12 Schematic diagram of the process of allocating initial horizontal scaling factors to multiple blocks divided in . Fig.13A As shown, the block overlapping both ROI 1212 and ROI 1216 is assigned S_horizontal=min(1;5)=1; the block overlapping both ROI 1214 and ROI 1216 is assigned S_horizontal=min(2;5)=2; and the remaining blocks overlap only ROI 1216 and are therefore assigned S_horizontal=5. Fig. 13B is a diagram showing a method for Fig.12 Schematic diagram of the process of allocating initial horizontal scaling factors to multiple blocks divided in . Fig. 13B As shown, the block overlapping with both ROI 1212 and ROI 1216 is assigned S_vertical=min(1;6)=1; the block overlapping with ROI 1214 and ROI 1216 is assigned S_vertical=min(3;6)=3; and the remaining blocks overlap only with ROI 1216 and are therefore assigned S_horizontal=6.

[0114] After assigning the initial downscaling factor, the encoder determines a commonly agreed horizontal downscaling factor for each column of the block ( Fig.14A ), and determine a mutually agreed vertical reduction factor for each row of the block ( Fig. 14B ). Fig.14A is a diagram showing a method for Fig.12 Schematic diagram of the process of determining the agreed horizontal reduction factor for each column of the block in . Fig.14AAs shown, the commonly agreed horizontal reduction factor for a given column is calculated as the minimum of all horizontal reduction factors in that column. For example, from left to right, the commonly agreed horizontal reduction factors for the columns are 5, 1, 5, 2, and 5. This means that the first column from the left can be reduced horizontally by a factor of 5; the second column from the left cannot be reduced horizontally and must remain the same horizontal size; the third column from the left can be reduced horizontally by a factor of 5; the fourth column from the left can be reduced horizontally by a factor of 2; and the fifth column from the left can be reduced horizontally by a factor of 5. On the other hand, Fig. 14B is a diagram showing a method for Fig.12 Schematic diagram of the process of determining the agreed horizontal reduction factor for each column of the block in . Fig. 14B As shown, the commonly agreed vertical reduction factor for a given row is calculated as the minimum of all vertical reduction factors in the row. For example, from top to bottom, the commonly agreed vertical reduction factors of the rows are 6, 3, 1, 1, and 6. This means that the first row from the top can be reduced horizontally by 6 times; the second row from the top can be reduced horizontally by 3 times; the third and fourth rows from the top cannot be reduced horizontally and must remain the same horizontal size; the fifth row from the top can be reduced horizontally by 6 times.

[0115] To summarize the above process of determining the allowed reduction factors for multiple blocks, the encoder can determine one or more ROIs horizontally collocated with the block; then determine the allowed vertical reduction factor of the block to be equal to the minimum vertical reduction factor associated with the one or more ROIs horizontally collocated with the block. In addition, independently, the encoder can determine one or more ROIs vertically collocated with the block; then determine the allowed horizontal reduction factor of the block to be equal to the minimum horizontal reduction factor associated with the one or more ROIs vertically collocated with the block.

[0116] Equivalently, since multiple blocks are arranged in rows and columns, the above process of determining the allowed reduction factor can also be performed according to the rows and columns of the blocks. Specifically, the encoder can determine one or more ROIs co-located with a row of blocks; then determine the allowed vertical reduction factor of the row of blocks to be equal to the minimum vertical reduction factor associated with the one or more ROIs co-located with the row of blocks. In addition, independently, the encoder can determine one or more ROIs co-located with a column of blocks; then determine the allowed horizontal reduction factor of the column of blocks to be equal to the minimum horizontal reduction factor associated with the one or more ROIs co-located with the column of blocks.

[0117] Back to Fig.11 In step 1108, the encoder performs at least one of horizontal downscaling or vertical downscaling on the plurality of blocks based on the corresponding allowed downscaling factors. Specifically, the encoder may vertically downscale each block based on the corresponding allowed vertical downscaling factor. Alternatively or additionally, the encoder may horizontally downscale each block based on the corresponding allowed horizontal downscaling factor.

[0118] For example, Fig.15As shown, in order to zoom out, you can first zoom out Fig.15 Each block formed by the reduced grid has a width and height according to Equations 10 and 11:

[0119] rescaled_width = original_width / min_in_column (S_horizontal) (10)

[0120] rescaled_height = original_height / min_in_row (S_vertical) (11)

[0121] In equations 10 and 11, the rescaled width and height, i.e., rescaled_width and rescaled_height, can be calculated using any rounding method, such as rounding to zero, the nearest integer, the nearest power of 2, etc. The rescaled width and height can also be calculated using a maximum clipping method, i.e., limiting the maximum size of each block. The rescaled width and height can also be calculated using a minimum clipping method, i.e., limiting the minimum size of each block.

[0122] After the grid is reduced, the content in the original input image can be redirected by reducing the image content in the original grid to fit the corresponding blocks of the reduced grid, as Fig.16 shown.

[0123] like Fig.15 As shown in , the reduced grid can also be enlarged (i.e., reversely reduced) to restore the original grid. Fig.16 As shown, the redirected image can also be un-redirected to restore the original image.

[0124] In some embodiments, instead of reducing each block individually, a column of blocks or a row of blocks may be reduced simultaneously. Specifically, the encoder may vertically reduce a row of blocks based on a corresponding allowed vertical reduction factor. Alternatively or additionally, the encoder may horizontally reduce a column of blocks based on a corresponding allowed vertical reduction factor.

[0125] Consistent with the disclosed embodiments, the downscaling process may be performed using any suitable technique known in the art, such as nearest neighbor downsampling, linear downsampling, bilinear downsampling, cubic downsampling, bicubic downsampling, spline downsampling, polynomial downsampling, affine transformation, graphics card (GPU)-based texture downsampling, or the like.

[0126] Back to Fig.11 At step 1110, the encoder generates parameters used in redirection. For example, the redirection parameters may include a reduction factor used in the redirection process.

[0127] Fig.17 1 is a block diagram illustrating an exemplary architecture of a VCM decoder 1700 consistent with an embodiment of the present disclosure. Fig.17 As shown, the input of the decoder 1700 includes a coded bitstream 1701 and ROI metadata 1702. For example, the coded bitstream 1701 and the ROI metadata 1702, such as Figure 1 The encoded bitstream 160 and ROI metadata 162 in the format may be generated by an encoder and sent from the encoder. Fig.17 As shown, the decoder 1700 includes a decoding module 1710 , a reverse redirection module 1720 , and a ROI synthesis module 1730 .

[0128] As described above, the disclosed encoding and decoding methods are independent of image / video compression technology. Therefore, the decoding module 1710 can decode according to any existing standards such as AVC, HEVC, VVC, AV1, JPEG, MPEG, etc.

[0129] The inverse redirection module 1720 restores the original size (e.g., the original number of rows and columns) of the image or video frame. The inverse redirection module 1720 can use metadata category 2, which includes redirection parameters generated on the encoder side during the redirection process. In addition, metadata category 1 can be used to provide information of ROI. For a video sequence, the frame size after inverse redirection can be set to a constant. For example, according to the requirements of some video coding standards, the frame size of the entire sequence starting from the instantaneous decoder refresh (IDR) frame needs to remain unchanged.

[0130] The ROI synthesis module 1730 restores the objects removed by the repair using metadata category 1 and / or category 3. Metadata category 1 defines the ROI corresponding to the objects (including the objects removed by the repair). Metadata category 3 defines the object representation of the objects removed and replaced by the repair. The metadata can be defined in any reasonable way, such as as an index indicating the object type and / or the basic characteristics of the object (such as size and orientation). The metadata can also include a potential representation of the object. In some embodiments, the restored object can be similar to the original object. For example, the restored object can retain the main features of the original object to allow downstream machine vision tasks to perform correct object detection, recognition, classification, or tracking.

[0131] Next, the signal path in the decoder 1700 is described.

[0132] The encoded bit stream 1702 is generated by an encoder (e.g., encoder 100 ( Figure 1)) is generated and received from an encoder. The encoded bitstream 1702 may be a traditional bitstream generated based on any existing standard such as AVC, HEVC, VVC, AV1, JPEG, MPEG, etc.

[0133] ROI metadata 1702 may include one or more metadata categories 1 to 3, which are described above in conjunction with encoder 100 ( Figure 1 ) were described.

[0134] Signal 1715 includes decoded (ie, decompressed) image data. Some objects (eg, objects removed by restoration) are still missing from signal 1715. In addition, the image / frame size represented by signal 1715 may be reduced relative to the original image / frame size.

[0135] Signal 1725 includes an image or video frame of original size (eg, original number of rows and columns) that is restored by inverse re-orientation 1720 .

[0136] The decoded image / video 1740 includes an image or video frame of the original size and having the inpainted object restored by the ROI synthesis 1730 .

[0137] As described above, the disclosed encoding and decoding architecture can be used to compress still images and videos. For video processing, the disclosed encoder and decoder can use the following time settings. For example, the disclosed encoder and decoder conform to the configuration of the standard video scheme in terms of basic video encoding parameters (such as frame size). For I frames (and particularly IDR frames), the above process can be used for encoding and decoding. For other frame types (such as P frames or B frames), the following rules can be followed.

[0138] In some embodiments, the set of objects removed by the inpainting is fixed for all frames of the sequence (e.g., the complete intervals between IDR images). The inpainted content may remain constant for all frames of the sequence (e.g., the complete intervals between IDR images). For example, on the encoder side, the same inpainted image element may be used for a sequence of frames. On the decoder side, the same ROI is synthesized for a sequence of frames.

[0139] The simplified background may remain constant for all frames of a sequence (eg, the entire interval between IDR pictures). For example, in the image reduction performed at the encoder side, the same simplified background region may be used for a sequence of frames.

[0140] For retargeting and detargeting, the object-related ROI may be defined by the sum of pixels at all positions of the object in the sequence. Retargeting and detargeting are performed in the same way for all frames of a video sequence. For example, on the encoder side, image retargeting is performed using the same set of ROIs and associated downscaling factors in each frame of the frame sequence. On the decoder side, detargeting is performed using the same set of ROIs and associated downscaling factors in each frame of the frame sequence.

[0141] Fig.18 1800 is a flowchart of an exemplary decoding method 1800 consistent with an embodiment of the present disclosure. For example, the method 1800 may be performed by a VCM decoder (e.g., decoder 1700 ( Fig.17 ))Execute. Fig.18 As shown, the decoding method 1800 includes the following steps.

[0142] At step 1802, the decoder receives a coded bitstream. The coded bitstream includes compressed image data and metadata, which describes: one or more regions of interest (ROIs) in an uncompressed image (i.e., the original image), and modifications made to the uncompressed image. For example, the metadata may indicate the location, shape, or category of the ROI. In addition, the modifications made to the uncompressed image may include at least one of image restoration, simplification, or redirection. For example, the metadata may include a reduction factor for reducing the ROI and redirecting the uncompressed image. As another example, the metadata may include information describing one or more ROIs removed from the uncompressed image by restoration, such as an index associated with the category of the removed ROI, a feature vector describing the removed ROI, or a shift vector pointing to the unremoved ROI in the uncompressed image.

[0143] At step 1804, the decoder decodes the bitstream to generate a reconstructed image. The decoder may decode the bitstream based on any existing image or video coding standard, such as AVC, HEVC, VVC, AV1, JPEG, MPEG, etc.

[0144] At step 1806, the decoder restores the uncompressed image by at least one of performing inverse re-targeting or ROI synthesis on the reconstructed image based on the metadata.

[0145] To perform the inverse reorientation, the decoder first locates one or more ROIs in the reconstructed image based on the metadata. The decoder then divides the reconstructed image into a plurality of blocks using a grid consisting of horizontal and vertical lines. Each horizontal line of the grid is aligned with at least one horizontal edge of the one or more ROIs, and each vertical line of the grid is aligned with at least one vertical edge of the one or more ROIs. The decoder also determines an allowed reduction factor for each of the plurality of blocks, respectively. The allowed reduction factor is less than or equal to the one or more reduction factors associated with the one or more ROIs. Finally, the decoder horizontally or vertically enlarges each of the plurality of blocks based on the corresponding allowed reduction factor. The inverse reorientation process is as follows: Fig.10 As shown, Fig.10 The original image 1010 is shown being re-oriented at the encoder side and then un-re-oriented at the decoder side to restore the original size.

[0146] To perform ROI synthesis, the decoder determines the inpainted image elements in the reconstructed image based on the metadata. Then, the decoder synthesizes the first ROI based on the metadata. Finally, the decoder replaces the inpainted image elements with the first ROI. Fig.19 1 is a schematic diagram illustrating an exemplary process 1900 for performing image restoration and ROI synthesis consistent with an embodiment of the present disclosure. Fig.19 As shown, the original image 1910 includes objects (i.e., ROIs) 1912, 1914, 1916, and 1918, etc. On the encoder side, image restoration is performed on the original image 1910, wherein the objects 1912, 1914, 1916, and 1918 are replaced respectively with restoration image elements 1932, 1934, 1936, and 1938 having synthesized contents. Then, the encoder compresses the resulting restoration image 1930. On the decoder side, ROI synthesis is performed on the restoration image 1930, wherein the restoration image elements 1932, 1934, 1936, and 1938 are replaced respectively by ROIs 1952, 1954, 1956, and 1958. The ROIs 1952, 1954, 1956, and 1958 include objects belonging to the same or similar categories as the objects 1912, 1914, 1916, and 1918.

[0147] Next, a hardware environment for implementing the disclosed encoder and decoder is described. Fig. 20 2 is a block diagram of an exemplary apparatus 2000 for encoding or decoding image data according to some embodiments of the present disclosure. Fig. 20As shown, the device 2000 may include a processor 2002. When the processor 2002 executes the instructions described herein, the device 2000 may become a dedicated machine for processing image data. The processor 2002 may be any type of circuit capable of manipulating or processing information. For example, the processor 2002 may include any number of central processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLA), programmable array logic (PAL), generic array logic (GAL), complex programmable logic devices (CPLD), field-programmable gate arrays (FPGA), systems on chip (SoC), application-specific integrated circuits (ASIC), and the like. In some embodiments, processor 2002 may also be a group of processors grouped into a single logical component. Fig. 20 As shown, processor 2002 may include a plurality of processors, including processor 2002a, processor 2002b, ..., and processor 2002n.

[0148] The apparatus 2000 may also include a memory 2004 for storing data (e.g., a set of instructions, computer code, intermediate data, etc.). Fig. 20As shown, the stored data may include program instructions (e.g., program instructions for implementing image data processing) and image data to be processed. The processor 2002 may access the program instructions and data for processing (e.g., via the bus 2010), and execute the program instructions to operate or manipulate the data. The memory 2004 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, the memory 2004 may include any number of random-access memories (RAM), read-only memories (ROM), optical disks, magnetic disks, hard disks, solid-state drives, flash drives, security digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Any combination. The memory 2004 may also be a group of memories ( Fig. 20 not shown).

[0149] The bus 2010 may be a communication device for transmitting data between components within the apparatus 2000 , such as an internal bus (eg, a CPU memory bus), an external bus (eg, a universal serial bus port, a peripheral component interconnect express port), and the like.

[0150] For ease of explanation and without ambiguity, the processor 2002 and other data processing circuits (such as analog / digital converters, dedicated circuits, digital signal processors, interface circuits, Fig. 20 The data processing circuit (not shown) is collectively referred to as "data processing circuit" in the present disclosure. The data processing circuit can be fully implemented as hardware, or implemented as a combination of software, hardware or firmware. In addition, the data processing circuit can be a single independent module, or it can be fully or partially combined into any other component of the device 2000.

[0151] The device 2000 may also include a network interface 2006 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 2006 may include any number of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication ("NFC") adapters, cellular network chips, etc.

[0152] In some embodiments, the apparatus 2000 may optionally further include a peripheral interface 2008 to provide a connection with one or more peripheral devices. Fig. 20As shown, peripheral devices may include but are not limited to a cursor control device (such as a mouse, touchpad or touch screen), a keyboard, a display (such as a cathode ray tube display, a liquid crystal display or a light emitting diode display), an image / video input device (such as a camera or an input interface coupled to an image / video file), etc.

[0153] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and these instructions can be executed by a device (e.g., a disclosed encoder and decoder) to perform the above method. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a tape or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cassette memory and networked versions thereof. The device may include one or more processors (CPU), input / output interfaces, network interfaces, and / or memories.

[0154] The disclosed system and method for VCM encoding and decoding can be implemented by general image / video encoding technology. The solution is independent of the image / video encoding technology applied. That is, any current or future image / video encoding technology can be applied.

[0155] The disclosed system and method are independent of and compatible with general image / video compression technology. In addition, the disclosed system and method are effective for low bit rates and very low bit rates of images and videos for which anchoring technology (i.e., image compression technology) cannot provide satisfactory efficiency suitable for machine vision tasks. For example, the image compression technology can be the most advanced video codec VVC (a versatile video codec standardized by ISO / IEC as 23090-3 and ITU as Rec.H.266, where ISO represents the International Organization for Standardization, IEC represents the International Electrotechnical Commission, and ITU represents the International Telecommunication Union). The use of VVC provides good results for relatively light compression with the efficiency of machine vision tasks performed on decoded images / videos being almost the same as the efficiency of these tasks performed on original (uncompressed) images / videos. However, for low bit rates and very low bit rates, when machine vision tasks are performed on images / videos decoded by VVC, the performance of the machine vision tasks is unsatisfactory, resulting in low accuracy of object detection, instant segmentation, etc. Fig.21A and Fig.21B Figure 2 shows a comparison of the average accuracy of detection and on-the-fly segmentation of example still images decoded from a VVC bitstream (the dashed graph is labeled “Anchor”) with the improvement obtained by the disclosed method (the solid graph is labeled “Proposal”). Fig.21A and Fig.21BAs shown, for an example still image, the VVC bitstream is associated with a lower average accuracy when the bit rate is below 0.1 bit / pixel. This situation can be improved by the disclosed method.

[0156] The disclosed solution is also extensible. One or two of the main building blocks (repair, simplify, redirect) can be omitted, thus reducing the complexity of VCM encoding.

[0157] Furthermore, the disclosed solution can control the balance between the quality of the output (decoded) images / video perceived by humans and the efficiency of the machine vision tasks performed on the output (decoded) images / video.

[0158] Furthermore, the disclosed encoder and decoder are compatible with the ISO / IEC standard for immersive video coding (MIV - MPEG-I Part 12 of ISO / IEC). The disclosed architecture can be used to compress a single 2D video in immersive video coding (MIV).

[0159] It should be noted that relational terms such as "first", "second", etc. used herein are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include", and other similar forms are intended to have the same meaning and are open-ended, in that the items following any of these words are not intended to be an exhaustive list of the items, nor are they intended to be limited to the listed items.

[0160] As used herein, unless expressly stated otherwise, the term "or" encompasses all possible combinations unless not feasible. For example, if it is stated that a database may include A or B, unless expressly stated otherwise or not feasible, the database may include A, or B, or A and B. As a second example, if it is stated that a database may include A, B, or C, unless expressly stated otherwise or not feasible, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0161] It should be understood that the present invention is not limited to the specific structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention should only be limited by the appended claims.

Claims

1. A coding method, comprising: determining one or more regions of interest (ROIs) in the image; Modifying the image based on the one or more ROIs, wherein modifying the image comprises at least one of the following: modifying a first ROI based on a portion of the image surrounding the first ROI; Reduce the content of the second ROI; or reducing the size of the third ROI; and The image is modified by compression.

2. The method according to claim 1, wherein: An ROI of the one or more ROIs includes a background region of the image.

3. The method according to claims 1 to 2, wherein: The second ROI and the third ROI are the same.

4. The method of claim 1 , further comprising generating metadata describing: the one or more ROIs, and Modifications to said image.

5. The method according to claim 4, wherein: The metadata includes: At least one of the position, shape, or category of the ROI.

6. The method according to any one of claims 4 or 5, wherein: The metadata includes: A reduction factor associated with reducing the size of the third ROI.

7. The method according to any one of claims 4 to 6, wherein: The metadata includes: Describe the information of the first ROI.

8. The method according to claim 7, wherein: The metadata describing the first ROI includes at least one of the following: the index associated with the category of the first ROI, A feature vector describing the first ROI, or A shift vector pointing to a fourth ROI in the image.

9. The method according to any one of claims 4 to 8, further comprising: Outputting a bitstream comprising the compressed image and the metadata, wherein the metadata is signaled in a header portion associated with the compressed image.

10. The method according to any one of claims 1 to 9, wherein: Modifying the first ROI includes: The first ROI is replaced with an inpainted image element, wherein the first ROI and the inpainted image element are represented by a first number of bits and a second number of bits, respectively, the second number being smaller than the first number.

11. The method according to claim 10, further comprising: The inpainted image element is generated based on one or more samples surrounding the first ROI.

12. The method according to any one of claims 1 to 11, wherein: The second ROI includes a background area, and reducing the content of the second ROI includes: determining the background region in the image based on the one or more ROIs; and The content of the background region is simplified to reduce the number of bits representing the background region.

13. The method according to claim 12, wherein: Simplifying the content of the background area includes: A low pass filter is performed on at least a portion of the background area.

14. The method according to claim 13, wherein: The low pass filtering uses a variable cutoff frequency that is inversely proportional to a distance between a sample in the background region and an object closest to the sample in the background region.

15. The method according to any one of claims 12 to 14, wherein: The second ROI includes: one or more objects; and One or more buffers respectively surrounding the one or more objects.

16. The method according to claim 15, wherein: The one or more buffers include a first buffer, and reducing the content of the second ROI includes: Low pass filtering is performed on at least a portion of the first buffer.

17. The method according to any one of claims 15 or 16, wherein: The one or more objects include a first object, and reducing the content of the second ROI includes: A low-pass filter is performed on the non-edge portion of the first object.

18. The method according to claim 16, wherein: The one or more objects include a first object surrounded by the first buffer, and reducing the content of the second ROI includes: The first object and the first buffer are reduced in size.

19. The method according to any one of claims 1 to 18, wherein: Reducing the size of the third ROI includes: determining at least one of a row or a column of the image, wherein at least one of the row or the column is not aligned with the one or more ROIs; and At least one of the rows or columns is removed from the image.

20. The method according to any one of claims 1 to 18, wherein: Reducing the size of the third ROI includes: defining at least one of the one or more ROIs to have a rectangular shape; and dividing the image into a plurality of blocks using a grid consisting of horizontal and vertical lines, wherein at least one of the horizontal lines of the grid is aligned with at least one horizontal edge of the one or more ROIs, and at least one of the vertical lines of the grid is aligned with at least one vertical edge of the one or more ROIs; and At least one of the plurality of blocks is scaled down horizontally or vertically.

21. The method according to claim 20, wherein: Horizontally or vertically reducing at least one of the plurality of blocks comprises: determining a target downscaling factor for at least one block of the plurality of blocks, respectively, wherein the target downscaling factor is less than or equal to one or more downscaling factors associated with the one or more ROIs; and The at least one block among the plurality of blocks is scaled down horizontally or vertically based on a corresponding target scale-down factor.

22. The method according to any one of claims 1 to 18, wherein: The image is an I frame in a frame sequence, and the method further comprises at least one of the following: applying the same inpainted image element to the sequence of frames; applying the same simplified background region to the frame sequence; or The one or more ROIs are determined from at least one frame in the frame sequence, and the frame sequence is redirected based on the one or more ROIs and one or more reduction factors respectively associated with the one or more ROIs, wherein the one or more ROIs and the one or more reduction factors remain the same throughout the frame sequence.

23. A device comprising: a memory for storing instructions; as well as One or more processors, configured to execute the instructions to cause the device to perform operations including: determining one or more regions of interest (ROIs) in the image; Modifying the image based on the one or more ROIs, wherein modifying the image comprises at least one of the following: modifying a first ROI based on a portion of the image surrounding the first ROI; Reduce the content of the second ROI; or reducing the size of the third ROI; and The image is modified by compression.

24. A non-transitory computer readable medium storing a set of instructions executable by one or more processors of a device to cause the device to perform a method comprising: determining one or more regions of interest (ROIs) in the image; Modifying the image based on the one or more ROIs, wherein modifying the image comprises at least one of the following: modifying a first ROI based on a portion of the image surrounding the first ROI; Reduce the content of the second ROI; or reducing the size of the third ROI; and The image is modified by compression.

25. A decoding method, comprising: Decode a bitstream that includes metadata describing: one or more regions of interest (ROIs) in the uncompressed image, and modifications made to the uncompressed image; determining a reconstructed image; as well as The uncompressed image is restored by restoring the content or size of the ROI based on the metadata.

26. The method according to claim 25, wherein: The metadata includes: At least one of the position, shape, or category of the ROI.

27. The method according to any one of claims 25 or 26, wherein: The metadata includes: A reduction factor associated with the one or more ROIs.

28. A method according to any one of claims 25 to 27, wherein: The metadata includes: Information describing the ROI removed from the uncompressed image.

29. The method according to claim 28, wherein: The metadata includes at least one of the following: the index associated with the category of the ROI to be removed, a feature vector describing the ROI that was removed, or A shift vector pointing to the unremoved ROI in the uncompressed image.

30. The method according to any one of claims 25 to 29, wherein: The size of the restored ROI includes: locating the ROI in the reconstructed image based on the metadata; and dividing the reconstructed image into a plurality of blocks using a grid consisting of horizontal lines and vertical lines, wherein at least one of the horizontal lines of the grid is aligned with at least one horizontal edge of the ROI, and at least one of the vertical lines of the grid is aligned with at least one vertical edge of the ROI; and At least one block among the plurality of blocks is enlarged horizontally or vertically.

31. The method according to claim 30, wherein: Horizontally or vertically enlarging at least one of the plurality of blocks comprises: determining a target downscaling factor for at least one block of the plurality of blocks, respectively, wherein the target downscaling factor is less than or equal to one or more downscaling factors associated with the ROI; and The at least one block among the plurality of blocks is horizontally or vertically enlarged based on a corresponding target downscaling factor.

32. The method according to any one of claims 25 to 29, wherein: Restoring ROI includes: determining, based on the metadata, an inpainted image element in the reconstructed image; synthesizing the ROI based on the metadata; The inpainted image element is replaced with the ROI.

33. A method according to any one of claims 25 to 32, wherein: The image is an I frame in a frame sequence, and the method further comprises at least one of the following: synthesizing the same ROI for the frame sequence; or The one or more ROIs are determined from at least one frame in the frame sequence, and the frame sequence is inversely reoriented based on the one or more ROIs and one or more reduction factors respectively associated with the one or more ROIs, wherein the one or more ROIs and the one or more reduction factors remain the same throughout the frame sequence.

34. An apparatus comprising: a memory for storing instructions; as well as One or more processors, configured to execute the instructions to cause the device to perform operations including: Decode a bitstream that includes metadata describing: one or more regions of interest (ROIs) in the uncompressed image, and modifications made to the uncompressed image; determining a reconstructed image; as well as The uncompressed image is restored by restoring the content or size of the ROI based on the metadata.

35. A non-transitory computer readable medium storing a set of instructions executable by one or more processors of a device to cause the device to perform a method comprising: Decode a bitstream that includes metadata describing: one or more regions of interest (ROIs) in the uncompressed image, and modifications made to the uncompressed image; determining a reconstructed image; as well as The uncompressed image is restored by restoring the content or size of the ROI based on the metadata.

Images