Method and system for image data processing

Abstract

Methods and systems for preprocessing image data before encoding the image data are disclosed. According to some embodiments, an exemplary method may include determining a plurality of regions in an image. The plurality of regions include: one or more objects, one or more buffer regions surrounding the one or more objects, respectively, and a background region surrounding the one or more buffer regions. The exemplary preprocessing method may also include: assigning samples of the image to the plurality of regions; modifying values ​​of the one or more samples based on the assignments; and compressing the image based on the modified values ​​of the one or more samples.

Description

Technical Field

[0001] The present disclosure relates generally to image data processing and, more particularly, to methods and systems for pre-processing image data before encoding the image data. Background Art

[0002] Limited communication bandwidth poses a challenge to transmitting images or videos (hereinafter collectively referred to as "image data") with ideal quality. Although various encoding technologies have been developed to compress image data before transmission, the growing demand for massive image data has brought continuous pressure to image / video systems (especially mobile devices that often need to work in limited bandwidth networks). Summary of the invention

[0003] Embodiments of the present disclosure relate to methods for preprocessing image data before encoding the image data. In some embodiments, a computer-implemented processing method is provided. The method may include determining multiple regions in an image. The multiple regions include: one or more objects, one or more buffer regions surrounding the one or more objects, and a background region surrounding the one or more buffer regions. The exemplary preprocessing method may also include: assigning a sample of the image to one of the multiple regions; modifying the value of the one or more samples based on the assignment; and compressing the image based on the modified value of the one or more samples.

[0004] In some embodiments, a computer-implemented preprocessing method is provided. The method is performed before compressing an image, and the method may include: determining an object, a buffer area surrounding the object, and a background area surrounding the buffer area in the image; modifying image data representing the background area; wherein the modification simplifies the content of the background area and thereby reduces the number of bits used to represent the compressed image.

[0005] In some embodiments, a device for processing image data is provided. The device includes: a memory for storing instructions; and one or more processors configured to execute the instructions to cause the device to perform the following operations: determine multiple regions in the image. The multiple regions include: one or more objects, one or more buffer regions surrounding the one or more objects, and a background region surrounding the one or more buffer regions. The operation also includes: assigning samples of the image to the multiple regions; modifying the value of one or more samples based on the assignment; and compressing the image based on the modified value of the one or more samples.

[0006] In some embodiments, a device for preprocessing image data is provided to preprocess the image before compressing the image. The device includes: a memory for storing instructions; and one or more processors configured to execute the instructions to cause the device to perform the following operations: determine an object, a buffer area surrounding the object, and a background area surrounding the buffer area in the image; modify the image data representing the background area; wherein the modification reduces the number of bits used to represent the compressed image.

[0007] Aspects of the disclosed embodiments may include a non-transitory tangible computer-readable medium storing software instructions that, when executed by one or more processors, are configured for and capable of performing and implementing one or more of the methods, operations, etc., in accordance with the disclosed embodiments. In addition, aspects of the disclosed embodiments may be performed by one or more processors configured as one (or more) special-purpose processors based on software instructions that are programmed with logic and instructions that, when executed, perform one or more operations in accordance with the disclosed embodiments.

[0008] Other purposes and advantages of the disclosed embodiments will be set forth in the following description, and in part will be apparent from the description, or may be learned through the practice of the embodiments. The purposes and advantages of the disclosed embodiments may be realized and obtained through the elements and combinations in the claims.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram showing an exemplary system for processing and encoding image data according to an embodiment of the present disclosure.

[0011] Figure 2 is a block diagram of an exemplary apparatus for processing image data according to an embodiment of the present disclosure.

[0012] Figure 3 is a flowchart illustrating an exemplary method for processing image data according to an embodiment of the present disclosure.

[0013] Figure 4 is a schematic diagram showing an object surrounded by adjacent bands according to an embodiment of the present disclosure.

[0014] Figure 5 is a schematic diagram showing that a plurality of objects are surrounded by corresponding adjacent bands according to an embodiment of the present disclosure.

[0015] Figure 6 is a flowchart illustrating an exemplary method for processing image data according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present invention will now be described in detail, examples of which are shown in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements, unless otherwise specified. The implementations set forth in the following description of exemplary embodiments do not represent all implementations according to the present invention. Instead, they are merely examples of devices and methods according to aspects related to the invention described in the appended claims.

[0017] The present disclosure relates to image data processing before encoding (i.e., compression) of image data. Specifically, an input image or video is processed to remove unimportant content or details therein. According to some exemplary embodiments, the disclosed image data processing method divides the input image into one or more objects, one or more buffer areas (also referred to herein as "bands") surrounding the one or more objects, respectively, and a background area. Since the content of the background area is not critical for most machine vision tasks (e.g., object detection, object recognition, object tracking, etc.), the disclosed image data processing method further simplifies the content of the background area, so that the amount of bits used to represent the encoded / compressed image data can be reduced. In some embodiments, the content of the object and / or buffer area can also be simplified. For example, removing some small details from a large object further reduces the amount of bits used to represent the encoded / compressed image data without reducing the accuracy of object detection, recognition, or tracking.

[0018] The processed image data is then encoded / compressed and sent to a decoder, which decodes the encoded image data and provides the decoded image data to a terminal device, which can use the decoded image data for various tasks, such as machine vision applications. Examples of machine vision applications include, but are not limited to, segmenting / recognizing / tracking objects from images or videos, detecting object categories associated with objects in a scene, etc.

[0019] The disclosed image data processing method can be used in low bit rate situations, which means that the amount of image data that can be transmitted per time unit is limited. According to the present disclosure, one way to process image data before encoding / compression is to identify regions of interest (ROI) in the image before encoding / compressing the image. ROI is a portion of the image (i.e., a pixel block) where a potential object exists, and can be identified by a preliminary image analysis performed before encoding / compressing the image. More specifically, before encoding / compression, a preliminary image analysis can be performed to detect objects that may exist in the image. Then the ROI containing the objects that may exist is defined. Afterwards, the image data is input to an image / video encoder, and the data about the ROI is used by the encoder so that the ROI is encoded with better quality than other image parts (e.g., background). The quality can be controlled by the quantization step used by the encoder, where a shorter quantization step is used for higher quality. The quantization step is measured by a quantization parameter (QP). In this way, the codec can allocate more bandwidth to the ROI (i.e., information that is considered to be more important), thereby improving the image data encoding / transmission efficiency.

[0020] The above-described ROI-based method is performed before encoding and is therefore a "preprocessing" technique. It can be used with codecs that explicitly or implicitly enable ROI-based encoding. For example, macroblocks used in the Advanced Video Coding (AVC) codec or coding units used in the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) codecs can be directly considered as ROIs. In addition, although some codecs (e.g., the Joint Photographic Experts Group 2000 (JPEG 2000) codec using wavelet transforms) do not directly encode pixels in blocks, these codecs can still be adapted to handle ROI-based preprocessing.

[0021] Although ROI-based preprocessing techniques have their advantages, ROI-based preprocessing techniques generate additional decoder control information (e.g., the value of a quantization parameter) that may need to be sent with the encoded image / video bitstream. This additional information adds additional cost, i.e., additional bits to be transmitted. In order to solve this problem, the present disclosure provides preprocessing techniques that do not generate additional decoder control information or only generate a small amount of decoder control information. As described in detail below, these preprocessing techniques can be used to remove or reduce details in image parts that are not important or less important for tasks such as machine vision (e.g., object recognition, object segmentation, object tracking, etc.). More specifically, by simplifying the image content (i.e., details) in certain areas of the image, the disclosed preprocessing techniques can reduce the number of bits used to represent the image while maintaining the efficiency of the machine vision task (measured by its average accuracy). After simplification, the entire image can be encoded using the same quantization step or using a quantization step selected by the encoder according to a predetermined bit allocation process. Therefore, there is no need to send the value of the quantization step for a single object or image area.

[0022] Figure 1 is a block diagram illustrating a system 100 for preprocessing and encoding image data according to some disclosed embodiments. The image data may include an image (also referred to as a picture or frame), multiple images, or a video. An image is a static picture (e.g., a frame). Multiple images may be spatially or temporally related or unrelated. A video includes a collection of images arranged in chronological order.

[0023] like Figure 1 As shown, system 100 includes a source device 120 that provides encoded video data that is decoded by a destination device 140. According to the disclosed embodiments, each of source device 120 and destination device 140 may include any of a variety of devices, including: a desktop computer, a notebook (e.g., laptop) computer, a server, a tablet computer, a set-top box, a mobile phone, a vehicle, a camera, an image sensor, a robot, a television, a camera, a wearable device (e.g., a smart watch or a wearable camera), a display device, a digital media player, a video game console, a video streaming device, etc. Source device 120 and destination device 140 may be equipped for wireless or wired communication.

[0024] refer to Figure 1, the source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. The destination device 140 may include an input interface 142, an image / video decoder 144, and one or more image / video applications 146. The image / video preprocessor 122 preprocesses image data (i.e., one (or more) images or one (or more) videos) and generates an input bitstream for the image / video encoder 124. The image / video encoder 124 encodes the input bitstream and outputs an encoded bitstream 162 through the output interface 126. The encoded bitstream 162 is transmitted through the communication medium 160 and received through the input interface 142. Then, the image / video decoder 144 decodes the encoded bitstream 162 to generate decoded data, which can be used by the image / video application 146.

[0025] In some embodiments, source device 120 may also include various devices (not shown) for providing source image data processed by image / video preprocessor 122. The devices for providing raw image data may include an image / video capture device, such as a camera, an image / video archive or storage device containing previously captured images / videos, or an image / video feed interface for receiving images / videos from an image / video content provider.

[0026] Both the image / video encoder 124 and the image / video decoder 144 may be implemented as any of a variety of suitable encoder or decoder circuits (e.g., one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof). When encoding or decoding is implemented partially or entirely in software, the image / video encoder 124 or the video / image decoder 144 may store instructions for the software in a suitable, non-transitory computer-readable medium, and use one or more processors to execute the instructions in hardware to perform techniques according to the present disclosure. Each of the image / video encoder 124 or the image / video decoder 144 may be included in one or more encoders or decoders, any of which may be integrated into a corresponding device as part of a combined encoder / decoder (CODEC).

[0027] The image / video encoder 124 and the image / video decoder 144 may operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photographic Experts Group (JPEG-1 or JPEG2000), etc. Alternatively, the image / video encoder 124 and the image / video decoder 144 may be custom devices that do not conform to existing standards. Figure 1 Not shown, but in some embodiments, the image / video encoder 124 and the image / video decoder 144 may be integrated with an audio encoder and decoder, respectively, and may include appropriate multiplexing-demultiplexing (MUX-DEMUX) units or other hardware and software to handle the encoding of audio and video in a common data stream or separate data streams.

[0028] The output interface 126 may include any type of medium or device capable of transmitting the encoded bitstream 162 from the source device 120 to the destination device 140. For example, the output interface 126 may include a transmitter or a transceiver configured to transmit the encoded bitstream 162 directly from the source device 120 to the destination device 140 in real time. The encoded bitstream 162 may be modulated according to a communication standard (such as a wireless communication protocol) and transmitted to the destination device 140.

[0029] The communication medium 160 may include a transient medium such as a wireless broadcast or a wired network transmission. For example, the communication medium 160 may include a radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). The communication medium 160 may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). In some embodiments, the communication medium 160 may include a router, a switch, a base station, or any other device that can be used to facilitate communication from the source device 120 to the destination device 140. For example, a network server (not shown) may receive the encoded bit stream 162 from the source device 120 and provide the encoded bit stream 162 to the destination device 140, for example, via a network transmission.

[0030] Communication medium 160 may also be in the form of a storage medium (e.g., a non-transitory storage medium), such as a hard disk, an SSD disk, a flash drive, a compact disk, a digital video disk, a Blu-ray disk, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, a computing device of a media production facility (e.g., a disk stamping facility) may receive the encoded image data from source device 120 and generate a disk containing the encoded image data.

[0031] Input interface 142 may include any type of medium or device capable of receiving information from communication medium 160. The received information includes encoded bitstream 162. For example, input interface 142 may include a receiver or transceiver configured to receive encoded bitstream 162 in real-time.

[0032] The image / video application 146 includes various hardware and / or software for utilizing the decoded image data generated by the image / video decoder 144. For example, the image / video application 146 may include a display device for displaying the decoded image data to a user, and may include any display device, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. As another example, the image / video application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications, such as object recognition and tracking, facial recognition, image matching, image / video search, augmented reality, robotic vision and navigation, autonomous driving, three-dimensional structure construction, stereo matching, motion tracking, etc.

[0033] Next, an exemplary structure of the disclosed image data preprocessor will be described in detail. Figure 2 2 is a block diagram of an exemplary apparatus 200 for processing image data according to an embodiment of the present disclosure. For example, the apparatus 200 may be implemented as the image / video preprocessor 122 ( Figure 1 ).

[0034] like Figure 2 As shown, the device 200 may include a processor 202. When the processor 202 executes the instructions described herein, the device 200 may become a special-purpose machine for pre-processing image data. The processor 202 may be any type of circuit capable of manipulating or processing information. For example, the processor 202 may include any number of central processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general array logic (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), and the like. In some embodiments, the processor 202 may also be a group of processors grouped into a single logical component. For example, as Figure 2As shown, processor 202 may include a plurality of processors, including processor 202a, processor 202b, and processor 202n.

[0035] The apparatus 200 may also include a memory 204 configured to store data (eg, instruction sets, computer codes, intermediate data, etc.). Figure 2 As shown, the stored data may include program instructions (e.g., program instructions for implementing preprocessing of image data) and image data to be processed. The processor 202 may access the program instructions and data for processing (e.g., via the bus 210), and execute the program instructions to perform operations or manipulations on the data. The memory 204 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, the memory 204 may include any number of random access memories (RAMs), read-only memories (ROMs), optical disks, magnetic disks, hard disk drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Any combination. The memory 204 may also be a group of memories (e.g., a memory stick) grouped into a single logical component. Figure 2 not shown).

[0036] The bus 210 may be a communication device that transmits data between components within the apparatus 200 , such as an internal bus (eg, a CPU memory bus), an external bus (eg, a universal serial bus port, a peripheral component interconnect express port), and the like.

[0037] For ease of explanation and without ambiguity, the processor 202 and other data processing circuits (e.g., analog / digital converters, dedicated circuits, digital signal processors, interface circuits, Figure 2 The data processing circuits (not shown) are collectively referred to as "data processing circuits" in this disclosure. The data processing circuits may be implemented entirely as hardware, or as a combination of software, hardware, or firmware. In addition, the data processing circuits may be a single independent module, or may be fully or partially combined into any other component of the device 200.

[0038] The device 200 may also include a network interface 206 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 206 may include any number of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0039] In some embodiments, the apparatus 200 may optionally further include a peripheral interface 208 to provide a connection to one or more peripheral devices. Figure 2 As shown, peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), an image / video input device (e.g., a camera or input interface coupled to an image / video archive), and the like.

[0040] Figure 3 3 is a flow chart of an exemplary method 300 for processing image data according to some embodiments of the present disclosure. The method 300 may be used to pre-process an image before it is compressed (i.e., encoded) so that the amount of bits used to represent the compressed image may be reduced. For example, the method 300 may be performed by a feature encoder (such as the image / video preprocessor 122 ( Figure 1 ))Execute. Figure 3 As shown, the method 300 includes the following steps 302-306.

[0041] In step 302, the image data preprocessor determines an object in the image, a buffer area surrounding the object, and a background area surrounding the buffer area. Figure 4 As shown schematically, the image data preprocessor detects an object 410 in an image 400, and defines a buffer region 420 around the object 410. In the following description, the "buffer region" is also referred to and used interchangeably as a "band". According to the disclosed embodiments, the background region is an image region that does not include any object or band. In some embodiments, after determining the object and band from the image, the image data preprocessor determines the rest of the image as the background region. For example, in Figure 4 , after determining the object 410 and the buffer area 420 , the image data preprocessor determines that the remaining portion of the image 400 is a background area 430 surrounding the buffer area 420 .

[0042] Although Figure 4 Only one object is shown, but the image data preprocessor can detect multiple objects from the image and define multiple bands around the multiple objects respectively. Figure 5 As shown schematically, the image data preprocessor first identifies an object 510 and an object 520 in the image 500. The image data preprocessor then defines a buffer region 520 around the image 510, and a buffer region 522 around the object 520. After determining these objects and buffer regions, the image data preprocessor determines that the remainder of the image 500 is a background region 530 around the buffer regions 512 and 522. Although Figure 4 and Figure 5The bands (buffer regions) are shown to completely surround their corresponding objects, but in the disclosed embodiments, the bands may also partially surround the associated objects. Therefore, the term "surrounding" as used in this disclosure means completely or partially surrounding an image area (e.g., an image area occupied by an object). In addition, some objects may not have a corresponding band (i.e., a buffer region). That is, the size of the bands associated with these objects is zero.

[0043] In some embodiments, in order to detect or identify one (or more) objects in an image, an image data preprocessor may perform an image segmentation algorithm to divide an input image into multiple segments, i.e., pixel sets, each pixel set representing a portion of an image. The image segmentation algorithm may assign a label (i.e., a category) to each pixel set so that pixels with the same label share certain common characteristics. The image data preprocessor may then determine whether each image segment corresponds to an object (e.g., a human body, an animal, a car, or vegetation). According to some embodiments, image segmentation classifies each pixel in an image into one of a plurality of categories. The image data preprocessor may perform classification based on a model. A pre-labeled training data set associated with a known object may be used to train the model. In some embodiments, the image data preprocessor may perform classification using machine learning techniques (e.g., convolutional neural networks (CNNs), deep neural networks (DNNs), etc.).

[0044] In the disclosed embodiments, the width of the band can be defined in various ways. For example, the width of the band around the object can be uniform or variable. As another example, when multiple objects and bands are determined in the image, the widths of the multiple bands can be the same or different. The present disclosure does not limit the manner in which the width of the band is defined.

[0045] In some embodiments, the image data preprocessor determines the width of the band based on the characteristic size of the object surrounded by the band. For example, the image data preprocessor may define the width of the band as a percentage of the characteristic size according to the following equation 1:

[0046] BW=kw·L (1)

[0047] In Equation 1, BW represents the width of the band. L represents the characteristic size of the object surrounded by the band. kw represents the width factor, which can be a constant (e.g., 0.15) or a variable. The bandwidth BW and / or the characteristic size L can be defined and measured using the Euclidean metric, the Manhattan metric, or any other suitable distance metric. As is known in the art, the Euclidean metric defines the distance between two points as equal to the length of the straight line between the two points, while the Manhattan metric defines the distance between two points as equal to the sum of the absolute differences of the Cartesian coordinates of the points.

[0048] In addition, the above-mentioned bandwidth BW and / or characteristic size L can be defined based on the actual size of the object. Unlike the apparent size of the object displayed in the image, the actual size of the object is that it is pre-sized based on the image. In some embodiments, in order to determine the actual size of the object, the image data preprocessor can determine the apparent size of the object in the image, and the distance between the object and the image sensor (e.g., a camera) that captures the image. Various methods can be used to measure the distance between the object and the camera. In one example, the distance can be measured by a distance sensor. Exemplary distance sensors include, but are not limited to, infrared sensors or laser sensors, which are designed to measure the distance based on the time it takes for a light signal to propagate between the object and the camera. As another example, the distance can be determined using the lens formula shown in Equation 2:

[0049]

[0050] In Equation 2, OD represents the distance between the object and the center of the camera lens, CL represents the distance between the center of the camera lens and the image, and f represents the focal length of the camera lens. Equation 2 is based on an assumption and therefore applies to situations where OD is much longer than f. After determining the apparent size of the object in the image and the distance between the object and the camera, the image data preprocessor can determine the actual size of the object based on Equation 3:

[0051]

[0052] In Equation 3, AS represents the actual size of the object, IS represents the apparent size of the object in the image, OD represents the distance between the object and the center of the camera's lens, and f represents the focal length of the camera's lens. The above examples for determining the actual size of an object are for illustrative purposes only and are not meant to limit the disclosed embodiments.

[0053] In some embodiments, the characteristic size of an object may be limited to the maximum size of the object. For example, the characteristic size L in Equation 1 may be limited to the maximum size of the object. Also, for example, when the image data preprocessor runs a machine vision algorithm that uses a bounding box to track an object, the maximum object size L may be limited to the maximum size of the bounding box associated with the object. Thus, in Figure 4 In the example shown, the image data preprocessor may determine the maximum size of the object 410 as a rectangular bounding box ( Figure 4 ), which corresponds to the distance from a person's head to her feet. Figure 5 In the example shown, the image data preprocessor can use a bounding box ( Figure 5(not shown) to find the corresponding maximum size of objects 510 and 520. The above-mentioned way of defining the characteristic size L is only an example and is not meant to limit the disclosed embodiments.

[0054] In some embodiments, the bandwidth BW may depend on the object category associated with the object. As described above, image segmentation techniques may be used to identify one (or more) objects in an image. During this process, the image segmentation technique may classify segments of the image into predetermined categories. Some predetermined categories are used to describe the category of the object. The object category may be defined based on the object type (e.g., vehicle, building, person, face, toy, etc.). In some embodiments, the parameters in Equation 1 may be determined based on the object category, so that wider or narrower bands may be assigned to various types of objects (e.g., face, building, etc.). For example, Figure 4 and Figure 5 As shown, the image data preprocessor may classify object 410 as a “person” and objects 510 and 520 as “pets.” Based on the classification, the image data preprocessor may assign a wider band to object 410 than to objects 510 or 520 .

[0055] In some embodiments, the bandwidth BW may be a variable that depends on local properties of the object. For example, the parameter kw in Equation 1 may be a variable that depends on the position on the edge of the object. Figure 4 In the example, the image data preprocessor may increase the value of the parameter kw in regions with rich textures (eg, animal fur, human faces) or large curvatures (eg, joints of the human body).

[0056] In some embodiments, the width coefficient kw in equation 1 can be a variable that depends on the position on the edge of the object or on the local texture or curvature of the edge of the object. For example, the value of the width coefficient kw can be set proportionally to the value of the local curvature. Alternatively or additionally, the width coefficient kw can be based on the color of the object and / or the color of the background area. For example, the value of the width coefficient kw can be set proportionally to the difference between the hue of the object and the hue of the background area.

[0057] Back to Figure 3 In step 304, the image data preprocessor modifies at least the image data representing the background region. The modification of the image data reduces the number of bits used to represent the compressed image. Specifically, the image data preprocessor may modify the image data by simplifying the content of the background region to reduce the amount of bits used to represent the background region in the compressed image.

[0058] The image data preprocessor can simplify the content of the background area in various ways. In some embodiments, the image data preprocessor can perform low-pass filtering on the image data representing at least a portion of the background area. The low-pass filtering can use a constant cutoff frequency. Alternatively, the low-pass filtering can use a variable cutoff frequency that is negatively correlated with the distance from the nearest band.

[0059] In some embodiments, the image data preprocessor may modify the image content in the background region, for example, attenuating the image content (i.e., reducing the brightness and color contrast of the image content). For example, if the distance d between a pixel in the background region and the nearest object is less than a predetermined value B, the image data representing the pixel in the background region may be modified according to Equations 4-6:

[0060] Y′=0.5+(Y-0.5)(1-(d-BW) / B) (4)

[0061] CR′=(CR)(1-(d-BW) / B) (5)

[0062] CB′=(CB)(1-(d-BW) / B) (6)

[0063] In equations 4-6, Y is the normalized luminance component of the pixel value before modification, and CB and CR are the normalized chrominance components of the pixel before modification. The values ​​of Y, CB, and CR satisfy the following ranges: 0≤Y≤1, -0.5≤CB, CR≤0.5. In addition, Y', CB', and CR' represent the corresponding luminance and chrominance values ​​after modification. BW represents the width of the band. Conversely, if the distance d from the nearest object is greater than BW+B, the image data representing the background area can be modified according to equations 7-9:

[0064] Y′=0.5 (7)

[0065] CR'=0 (8)

[0066] CB'=0 (9)

[0067] Furthermore, if the distance d to the nearest object is between B and BW+B (ie, B≤d≤BW+B), the image data is unchanged.

[0068] In some embodiments, pre-processing may set the values ​​of pixels in the entire background area according to Equations 7-9.

[0069] In some embodiments, the image data preprocessor may also modify the values ​​of pixels located in the object to reduce the number of bits used to represent the object in the compressed image. For example, the image data preprocessor may reduce the size of the object according to a scaling factor sc. The value of the scaling factor is less than 1. The image data preprocessor may set the value of the scaling factor based on the object class associated with the object. The scaling factor may be included in the encoded bit stream sent to the decoder so that the decoder can use the scaling factor to return the object to its original size. Optionally, the value of the scaling factor may be predetermined; therefore, the scaling factor does not need to be sent to the decoder. As another example, the image data preprocessor may perform low-pass filtering on image data representing a non-edge portion of an object. The non-edge portion is a portion that does not include the edge of the object, also referred to as the internal portion of the object. Low-pass filtering removes high-frequency components of the pixel value, thereby reducing the number of bits used to represent the object. Since the non-edge portion is less important in identifying and tracking objects, performing low-pass filtering on the non-edge portion does not reduce the efficiency of the machine vision task.

[0070] In some embodiments, the image data preprocessor may also modify the values ​​of pixels located in the band to reduce the number of bits used to represent the band in the compressed image. The image data preprocessor may perform the modification in various ways. In some embodiments, if the image data preprocessor reduces the size of an object surrounded by the band, the image data preprocessor may also reduce the size of the band using the same scaling factor. In some embodiments, the image data preprocessor may perform a low pass filter on the image data representing at least a portion of the band. For example, a low pass filter may be performed on a portion of the band that does not touch an edge of an object.

[0071] In step 306, the image data preprocessor provides the modified image data to the encoder. Specifically, the image data preprocessor can output the modified image in a bitstream, which is then input to the encoder. In some embodiments, the modified image data can be encoded using one or more predetermined quantization parameters without sending the quantization parameters to the decoder. For example, the entire preprocessed image can be encoded using a predetermined single quantization parameter. In this way, in addition to the simplified image data, the preprocessing method 300 does not generate additional metadata or control information that needs to be encoded or decoded. Therefore, the disclosed preprocessing technique can be transparent to the decoder. For the same reason, the disclosed preprocessing technique can be used with standard image / video encoders and decoders.

[0072] In the disclosed embodiments, Figure 5 As shown, multiple objects and bands can be identified from the image. When some objects and / or bands partially overlap each other, the overlapping area can be assigned to one of the overlapping objects and / or bands. Figure 66 is a flow chart of an exemplary method 600 for processing image data according to some embodiments of the present disclosure. The method 600 may be used to process an image before the image is compressed (ie, encoded). For example, the method 600 may be performed by an image data preprocessor (eg, Figure 1 The image / video preprocessor 122 in FIG. Figure 6 As shown, method 600 includes the following steps 602-608.

[0073] In step 602, the image data preprocessor determines a plurality of regions in the image. The plurality of regions include: one or more objects, one or more buffer regions surrounding the one or more objects, and a background region surrounding the one or more buffer regions. For example, the image data preprocessor may segment the image into a plurality of blobs and determine whether the blobs correspond to the one or more objects. After identifying the one or more objects, the image data preprocessor may determine a buffer region (i.e., a band) surrounding each of the one or more objects. The image data preprocessor may then determine that the remainder of the image is a background region. In this way, the determined background region does not contain any objects or bands.

[0074] The image data preprocessor can use the above combined with step 302 ( Figure 3 ) or any other suitable method to set the width of one or more bands. According to some embodiments, the image data preprocessor may set the width of the band based on a characteristic size of the object surrounded by the band. The characteristic size may be determined by the maximum size of the object surrounded by the band, and / or based on the actual size of the object. According to some embodiments, the image data preprocessor may set the width of the band based on an object class associated with the object surrounded by the band.

[0075] Back to Figure 6 In step 604, the image data preprocessor assigns samples of the image to the plurality of regions. The samples may include luminance values ​​and / or chrominance values ​​associated with a pixel or a group of pixels. According to some embodiments, in response to a sample being located in an overlapping region of two or more objects, the image data preprocessor assigns the sample to one of the two or more objects. For example, the image data preprocessor may assign the sample to an object that is not occluded by other objects at the overlapping region.

[0076] According to some embodiments, in response to a sample being located in an overlapping region between the buffer region and the object, the image data pre-processor assigns the sample to the object.

[0077] According to some embodiments, in response to a sample being located in an overlapping area of ​​two or more buffer areas, the image data preprocessor performs the following operations: selecting an object having a shortest distance to a pixel from objects respectively surrounded by the two or more buffer areas; and assigning the sample to the buffer area surrounding the selected object. The distance may be measured using a Euclidean metric, a Manhattan metric, or any other suitable metric.

[0078] The image data preprocessor modifies the values ​​of the samples based on the allocation in step 606. The modification reduces the number of bits used to represent the image.

[0079] The image data preprocessor may modify the sample values ​​differently based on whether the associated sample is located in an object, band, or background region. Specifically, the image data preprocessor may use the above method in conjunction with step 304 ( Figure 3 ) described in the method to modify the value. According to some embodiments, the image data preprocessor can reduce the size of the object by modifying the value of the sample assigned to the object. For example, the image data preprocessor can reduce the size of the object by downsampling the sample assigned to the object. As another example, the image data preprocessor can reduce the size of the object according to a scaling factor, which is determined based on the object category associated with the object. According to some embodiments, the image data preprocessor can low-pass filter the non-edge portion of the object by modifying the value of the sample assigned to the object. Specifically, the low-pass filter can be applied to the value of the sample representing the non-edge portion to remove the high-frequency content in the non-edge portion. According to some embodiments, the image data preprocessor can reduce the size of the band by modifying the value of the sample assigned to the band. Similar to adjusting the size of the object, the image data preprocessor can reduce the size of the band by downsampling the samples assigned to the band and / or using a scaling factor determined based on the object category associated with the object surrounded by the band. According to some embodiments, the image data preprocessor can low-pass filter a portion of the band by modifying the value of the sample assigned to the band. Specifically, the low-pass filter can be applied to the value of the sample representing at least a portion of the band to remove the high-frequency component therein. According to some embodiments, the image data preprocessor may perform low pass filtering on at least a portion of the background area by modifying the values ​​of the samples assigned to the background area. Specifically, a low pass filter may be applied to the values ​​of the samples assigned to the background area to remove high frequency components therein. The low pass filter may use a variable cutoff frequency that is negatively correlated with the distance from the nearest band. According to some embodiments, the image data preprocessor may modify the image content in the background area by modifying the values ​​of the samples assigned to the background area. For example, the image data preprocessor may modify the image content based on the method described above in conjunction with equations 2-7.

[0080] Back to Figure 6In step 608, the image data preprocessor provides the modified image data to the encoder. Specifically, the image data preprocessor may output the modified image in a bitstream, which is then input to the encoder. In some embodiments, the modified image data may be encoded using one or more predetermined quantization parameters without sending the quantization parameters to the decoder.

[0081] The image data preprocessing techniques described in the present disclosure can be used in applications where the bit rate of the transmitted image / video is low. In these applications, the strong compression caused by the low bit rate significantly reduces the quality of the decoded image. In such severely damaged images / videos, small details often disappear and other features become severely distorted, making them almost unrecognizable. Strong compression also causes the color of the object and / or the color of the background to change significantly. In addition, strange false patterns may be generated and edges may become blurred. And false edges may appear due to blocking or other compression artifacts. In the video, the movement of small objects may also be distorted. The intensity and characteristics of the above-mentioned compression artifacts depend on the specific compression technology used and the control parameters (e.g., bit rate) applied to the encoder. If the damage is obvious, the deterioration of the image or video quality significantly reduces the efficiency of machine vision tasks performed using classical methods or neural networks. Specifically, neural networks are usually trained using original or slightly distorted images and are therefore generally sensitive to compression artifacts.

[0082] The disclosed preprocessing techniques can be used to improve the performance of machine vision tasks under low bitrate conditions. The disclosed preprocessing techniques are not limited to any particular machine vision application. For example, the disclosed preprocessing techniques can be used to increase the average accuracy of detection, segmentation, or tracking of visual objects extracted from decoded image data. The disclosed preprocessing techniques can also be used with techniques for detecting various classes of objects, segmenting objects from video frames, etc. In addition, the disclosed preprocessing techniques are applicable to both still images and videos.

[0083] Image data preprocessed using the disclosed preprocessing techniques can be encoded / compressed by an encoder at a low bit rate while still maintaining the efficiency of the machine vision task. The performance of the disclosed preprocessing techniques can be measured based on the relationship between the encoding bit rate and the efficiency of the machine vision operation. The efficiency of the machine vision operation can be defined as the average accuracy of the machine vision operation performed using classical techniques or using artificial neural networks. The average accuracy can be defined according to the following equation 10:

[0084]

[0085] However, the application of the disclosed preprocessing techniques is not limited to scenarios where optimality is defined using accuracy. For example, the disclosed preprocessing techniques can be used in applications where performance criteria are defined in other suitable ways.

[0086] In some embodiments, a non-transitory computer-readable storage medium is also provided, which includes instructions, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) for performing the above method. Common forms of non-transitory media include, for example, floppy disks, disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a hole pattern, RAM, PROM, EPROM, FLASH-EPROM or any other flash memory, NVRAM, cache, registers, any other memory chip or cartridge memory, and networked versions thereof. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memories.

[0087] It may be noted that relational terms such as "first" and "second" herein are used only to distinguish an entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. In addition, the words "include", "have", "contain", and "comprise" and other similar forms are intended to represent equivalent meanings and are open-ended, and one or more items following any one of the words do not mean an exhaustive list of these items, nor are they limited to the listed one or more items.

[0088] The term "or" as used herein encompasses all possible combinations unless otherwise specifically stated or not feasible. For example, if it is stated that a database may include A or B, then unless otherwise specifically stated or not feasible, the database may include A, or B, or A and B. As a second example, if it is stated that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0089] It is to be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope of the present invention. The scope of the present invention shall be limited only by the appended claims.

Claims

1. A method for image processing, comprising: A plurality of regions in an image are determined, the plurality of regions comprising: One or more objects, one or more buffer regions surrounding the one or more objects, respectively, and a background area surrounding the one or more buffer areas; assigning samples of the image to the plurality of regions; modifying the value of one or more samples based on the assignment; and The image is compressed based on the modified values ​​of the one or more samples.

2. The method according to claim 1, wherein: Modification of the values ​​of the one or more samples reduces the number of bits used to represent the compressed image.

3. The method according to claim 1 or 2, wherein: Compressing the image includes encoding the modified values ​​of the one or more samples using one or more predetermined quantization parameters.

4. The method according to any one of claims 1 to 3, wherein: Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region of two or more objects, the sample is assigned to one of the two or more objects.

5. The method according to any one of claims 1 to 4, wherein: Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region between a buffer region and an object, the sample is assigned to the object.

6. The method according to any one of claims 1 to 5, wherein: Assigning samples of the image to the plurality of regions comprises: In response to the sample being located in an overlapping region of two or more buffer regions, performing the following operations: selecting an object having the shortest distance to the sample from among the objects respectively surrounded by the two or more buffer areas; and The samples are distributed to a buffer region surrounding the selected object.

7. The method according to any one of claims 1 to 6, further comprising: A width of each of the one or more buffer regions is determined based on a characteristic size of an object surrounded by the buffer region.

8. The method according to claim 7, wherein: The characteristic size is the maximum size of an object surrounded by the buffer area.

9. The method according to any one of claims 1 to 8, further comprising: The width of each of the one or more buffer areas is determined based on an actual size of an object surrounded by the buffer area.

10. The method according to any one of claims 1 to 9, further comprising: A width of each of the one or more buffer regions is determined based on an object class associated with an object surrounded by the buffer region.

11. The method according to any one of claims 1 to 10, wherein: The one or more objects include a first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first object is reduced by modifying values ​​assigned to one or more samples of the first object.

12. The method according to claim 11, wherein: The size of the first object is reduced according to a scaling factor determined based on an object class associated with the first object.

13. The method according to claim 11, wherein: The one or more buffer regions include a first buffer region surrounding the first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first buffer area is reduced by modifying the value of one or more samples allocated to the first buffer area.

14. The method according to any one of claims 1 to 13, wherein: The one or more objects include a second object, and modifying the value of the one or more samples based on the assignment includes: Low pass filtering is performed on non-edge portions of the second object by modifying values ​​assigned to one or more samples of the second object.

15. The method according to any one of claims 1 to 14, wherein: The one or more buffer regions include a second buffer region, and modifying the value of the one or more samples based on the allocation includes: Low pass filtering is performed on at least a portion of the second buffer region by modifying values ​​of one or more samples assigned to the second buffer region.

16. The method according to any one of claims 1 to 15, wherein: Modifying the value of the one or more samples based on the allocation includes: Simplify the content of the background area.

17. The method according to claim 16, wherein: Simplifying the content of the background region includes performing low pass filtering on at least a portion of the background region using a variable cutoff frequency that is negatively correlated to a distance between a sample in the background region and an object closest to the sample in the background region.

18. The method according to any one of claims 1 to 17, wherein: Modifying the value of the one or more samples based on the allocation includes: The value of the sample assigned to the background region is modified based on the distance between the sample and the buffer region closest to the sample.

19. A method for image processing, comprising: Preprocessing the image before compressing the image, wherein the preprocessing of the image includes: determining in the image an object, a buffer region surrounding the object, and a background region surrounding the buffer region; modifying image data representing the background area; Wherein the modification reduces the number of bits used to represent the compressed image.

20. The method according to claim 19, wherein: The modified image data is compressed using one or more predetermined quantization parameters.

21. The method according to claim 19 or 20, wherein: The width of the buffer area is determined based on a characteristic size of the object.

22. The method according to claim 21, wherein: The characteristic size is the maximum size of the object.

23. The method according to any one of claims 19 to 22, wherein: The width of the buffer area is determined based on the actual size of the object.

24. The method according to any one of claims 19 to 23, wherein: The width of the buffer region is determined based on an object class associated with the object.

25. The method according to any one of claims 19 to 24, wherein: The preprocessing of the image further comprises: Reduce the size of the object.

26. The method according to claim 25, wherein: The size of the object is reduced according to a scaling factor determined based on an object class associated with the object.

27. The method according to any one of claims 25 to 25, wherein: The preprocessing of the image further comprises: Reduce the size of the buffer area.

28. The method according to any one of claims 19 to 27, wherein: The preprocessing of the image further comprises: Image data representing non-edge portions of the object are low pass filtered.

29. The method according to any one of claims 19 to 28, wherein: The preprocessing of the image further comprises: Image data representing at least a portion of the buffer region is low pass filtered.

30. The method according to any one of claims 19 to 29, wherein: Modifying the image data representing the background area comprises: Image data representing at least a portion of the background region is low pass filtered.

31. The method according to claim 30, wherein: The low pass filtering of image data representing at least a portion of the background region uses a variable cutoff frequency that is inversely related to a distance from the buffer region.

32. The method according to any one of claims 19 to 31, wherein: The modifying of the image data representing the background area comprises: Content in the background area is attenuated.

33. An apparatus for image processing, comprising: a memory for storing instructions; as well as One or more processors configured to execute the instructions so that the apparatus performs the following operations: Determine a plurality of regions in the image, the plurality of regions comprising: One or more objects, one or more buffer regions surrounding the one or more objects, respectively, and a background area surrounding the one or more buffer areas; assigning samples of the image to the plurality of regions; modifying the value of one or more samples based on the assignment; and The image is compressed based on the modified values ​​of the one or more samples.

34. The device according to claim 33, wherein: Modification of the values ​​of the one or more samples reduces the number of bits used to represent the compressed image.

35. The device according to claim 33 or 34, wherein: Compressing the image includes encoding the modified values ​​of the one or more samples using one or more predetermined quantization parameters.

36. The device according to any one of claims 33 to 35, wherein: Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region of two or more objects, the sample is assigned to one of the two or more objects.

37. The device according to any one of claims 33 to 36, wherein: Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region between a buffer region and an object, the sample is assigned to the object.

38. The device according to any one of claims 33 to 37, wherein: Assigning samples of the image to the plurality of regions comprises: In response to the sample being located in an overlapping region of two or more buffer regions, performing the following operations: selecting an object having the shortest distance to the sample from among the objects respectively surrounded by the two or more buffer areas; and The samples are distributed to a buffer region surrounding the selected object.

39. The device according to any one of claims 33 to 38, wherein: The operations also include determining a width of each of the one or more buffer regions based on a characteristic size of an object surrounded by the buffer region.

40. The device according to claim 39, wherein The characteristic size is the maximum size of an object surrounded by the buffer area.

41. The device according to any one of claims 33 to 40, wherein: The operations also include determining a width of each of the one or more buffer regions based on an actual size of an object surrounded by the buffer region.

42. The device according to any one of claims 33 to 41, wherein: The operations also include determining a width of each of the one or more buffer regions based on an object class associated with an object surrounded by the buffer region.

43. The device according to any one of claims 33 to 42, wherein: The one or more objects include a first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first object is reduced by modifying values ​​assigned to one or more samples of the first object.

44. The device according to claim 43, wherein The size of the first object is reduced according to a scaling factor determined based on an object class associated with the first object.

45. The apparatus of claim 43, wherein: The one or more buffer regions include a first buffer region surrounding the first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first buffer area is reduced by modifying the value of one or more samples allocated to the first buffer area.

46. ​​The device according to any one of claims 33 to 45, wherein: The one or more objects include a second object, and modifying the value of the one or more samples based on the assignment includes: Low pass filtering is performed on non-edge portions of the second object by modifying values ​​assigned to one or more samples of the second object.

47. The device according to any one of claims 33 to 46, wherein: The one or more buffer regions include a second buffer region, and modifying the value of the one or more samples based on the allocation includes: Low pass filtering is performed on at least a portion of the second buffer region by modifying values ​​of one or more samples assigned to the second buffer region.

48. The device according to any one of claims 33 to 47, wherein: Modifying the value of the one or more samples based on the allocation includes: Simplify the content of the background area.

49. The apparatus of claim 48, wherein: Simplifying the content of the background region includes performing low pass filtering on at least a portion of the background region using a variable cutoff frequency that is negatively correlated to a distance between a sample in the background region and an object closest to the sample in the background region.

50. The device according to any one of claims 33 to 49, wherein: Modifying the value of the one or more samples based on the allocation includes: The value of the sample assigned to the background region is modified based on the distance between the sample and the buffer region closest to the sample.

51. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executed by one or more processors of a device to cause the device to perform a method, the method comprising: A plurality of regions in an image are determined, the plurality of regions comprising: One or more objects, one or more buffer regions surrounding the one or more objects, respectively, and a background area surrounding the one or more buffer areas; assigning samples of the image to the plurality of regions; modifying the value of one or more samples based on the assignment; and The image is compressed based on the modified values ​​of the one or more samples.

52. The medium of claim 51, wherein: Modification of the values ​​of the one or more samples reduces the number of bits used to represent the compressed image.

53. A medium according to claim 51 or 52, wherein: Compressing the image includes encoding the modified values ​​of the one or more samples using one or more predetermined quantization parameters.

54. A medium according to any one of claims 51 to 53, wherein Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region of two or more objects, the sample is assigned to one of the two or more objects.

55. A medium according to any one of claims 51 to 54, wherein Assigning samples of the image to the plurality of regions comprises: In response to a sample being located in an overlapping region between a buffer region and an object, the sample is assigned to the object.

56. A medium according to any one of claims 51 to 55, wherein Assigning samples of the image to the plurality of regions comprises: In response to the sample being located in an overlapping region of two or more buffer regions, performing the following operations: selecting an object having the shortest distance to the sample from among the objects respectively surrounded by the two or more buffer areas; and The samples are distributed to a buffer region surrounding the selected object.

57. A medium according to any one of claims 51 to 56, wherein The method further includes determining a width of each of the one or more buffer regions based on a characteristic size of an object surrounded by the buffer region.

58. The medium of claim 57, wherein: The characteristic size is the maximum size of an object surrounded by the buffer area.

59. A medium according to any one of claims 51 to 58, wherein The method further includes determining a width of each of the one or more buffer areas based on an actual size of an object surrounded by the buffer area.

60. A medium according to any one of claims 51 to 59, wherein The method also includes determining a width of each of the one or more buffer regions based on an object class associated with an object surrounded by the buffer region.

61. A medium according to any one of claims 51 to 60, wherein The one or more objects include a first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first object is reduced by modifying values ​​assigned to one or more samples of the first object.

62. The medium of claim 61, wherein: The size of the first object is reduced according to a scaling factor determined based on an object class associated with the first object.

63. The medium of claim 61, wherein: The one or more buffer regions include a first buffer region surrounding the first object, and modifying the value of the one or more samples based on the allocation includes: The size of the first buffer area is reduced by modifying the value of one or more samples allocated to the first buffer area.

64. A medium according to any one of claims 51 to 63, wherein The one or more objects include a second object, and modifying the value of the one or more samples based on the assignment includes: Low pass filtering is performed on non-edge portions of the second object by modifying values ​​assigned to one or more samples of the second object.

65. A medium according to any one of claims 51 to 64, wherein The one or more buffer regions include a second buffer region, and modifying the value of the one or more samples based on the allocation includes: Low pass filtering is performed on at least a portion of the second buffer region by modifying values ​​of one or more samples assigned to the second buffer region.

66. A medium according to any one of claims 51 to 65, wherein Modifying the value of the one or more samples based on the allocation includes: Simplify the content of the background area.

67. The medium of claim 66, wherein: Simplifying the content of the background region includes performing low pass filtering on at least a portion of the background region using a variable cutoff frequency that is negatively correlated to a distance between a sample in the background region and an object closest to the sample in the background region.

68. A medium according to any one of claims 51 to 67, wherein Modifying the value of the one or more samples based on the allocation includes: The value of the sample assigned to the background region is modified based on the distance between the sample and the buffer region closest to the sample.

Images