Image encoding device, image decoding device, image encoding method, and image decoding method

Abstract

The present invention is provided with: a circuit; and a memory connected to the circuit. In operation, the circuit decodes a first bitstream to acquire a first image, decodes a second bitstream to acquire specification information specifying a specific region in the first image and a second image including image data of the specific region, and generates a third image on the basis of the first image, the specification information, and the second image.

Description

Image encoding device, image decoding device, image encoding method, and image decoding method

[0001] The present disclosure relates to an image encoding device, an image decoding device, an image encoding method, and an image decoding method.

[0002] US Pat. No. 6,299,499 discloses a video encoding and decoding method using an adaptive combined pre-filter and post-filter.

[0003] US Patent No. 6,299,649 discloses a method for encoding image data for loading into an artificial intelligence (AI) integrated circuit.

[0004] U.S. Patent No. 9,883,207 U.S. Patent No. 1,045,2955

[0005] The present disclosure aims to achieve both the protection of personal privacy information and the execution of machine tasks or human vision using the privacy information on the image decoding device side when transmitting images from an image encoding device to an image decoding device.

[0006] An image decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, obtains a first image by decoding a first bitstream, obtains designation information that designates a specific region within the first image and a second image that includes image data of the specific region by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

[0007] 1 is a diagram illustrating a simplified configuration of an image processing system according to an embodiment of the present disclosure. FIG. 1 is a flowchart illustrating a flow of processing performed by an image encoding device. FIG. 2 is a diagram illustrating an example of an input image. FIG. 3 is a diagram illustrating a bounding box as an example of a specific region. FIG. 4 is a diagram illustrating an example of designation information. FIG. 5 is a diagram illustrating an example of designation information. FIG. 6 is a diagram illustrating an example of designation information. FIG. 7 is a diagram illustrating an example of designation information. FIG. 8 is a diagram illustrating an example of designation information. FIG. 9 is a diagram illustrating an example of a first image. FIG. 10 is a diagram illustrating an example of a second image. FIG. 11 is a diagram illustrating an example of a second image. FIG. 12 is a diagram illustrating an example of a second image. FIG. 13 is a diagram illustrating an example of a second image.

[0008] (Findings that Form the Basis of the Present Disclosure) Conventional encoding methods aim to provide optimal video for human vision under bit rate constraints.

[0009] Advances in machine learning or neural network-based applications, along with a wealth of sensors, have enabled many intelligent platforms that handle large amounts of data, including connected cars, video surveillance, smart cities, etc. The constant generation of large amounts of data makes traditional methods of including humans in the pipeline inefficient and impractical in terms of latency and scale.

[0010] Furthermore, there is a concern for more compact data representation and low latency solutions in transmission and archiving systems, which has led to the introduction of Video Coding for Machines (VCM).

[0011] In some cases, machines can communicate with each other and perform tasks without human intervention, while in other cases, certain decompressed streams require additional human processing, such as a human "supervisor" searching for specific people or scenes in a surveillance camera video stream.

[0012] In other cases, the corresponding bitstream may be used by both humans and machines: in the case of connected cars, the features can be used for image enhancement functions for humans and object detection and segmentation for machines.

[0013] A typical system architecture includes an image encoder and decoder pair. The input of the system can be video, still images, or features. Examples of machine tasks include object detection, object segmentation, object tracking, action recognition, pose estimation, or any combination thereof. Human vision is a potential use case that can be used with machine tasks.

[0014] According to conventional technologies, when a captured image contains privacy information such as an individual's face or a vehicle license plate number, the image containing the privacy information is transmitted from an image encoding device to an image decoding device even if the information is not necessary for a machine task, which may result in the leakage of the privacy information. On the other hand, it is also possible to transmit an image in which the privacy information has been subjected to a concealment process such as blurring from an image encoding device to an image decoding device, but this poses a problem in that the privacy information cannot be used when it is necessary for a machine task or human vision.

[0015] In order to solve this problem, the present inventor discovered that the above problem can be solved by separately transmitting a bit stream of an image containing privacy information and a bit stream of an image not containing privacy information from an image encoding device to an image decoding device, and then combining them in the image decoding device when privacy information is required, which led to the present disclosure.

[0016] Next, each aspect of the present disclosure will be described.

[0017] An image decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, obtains a first image by decoding a first bitstream, obtains designation information that designates a specific region within the first image and a second image that includes image data of the specific region by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

[0018] According to the first aspect, by not including the privacy information in the first bitstream, it is possible to protect the privacy information of an individual. Also, by transmitting the second image including the privacy information as the second bitstream and generating the third image based on the second image, it is possible to execute a machine task or human vision using the privacy information on the image decoding device side.

[0019] In the image decoding device according to the second aspect of the present disclosure, in the first aspect, the specific area in the first image may include an image in which a concealment process has been performed on a privacy area containing privacy information.

[0020] According to the second aspect, the private information included in the specific area in the first image can be appropriately protected by the concealment process.

[0021] In the image decoding device according to the third aspect of the present disclosure, in the second aspect, the second image and the third image may include an image in which the concealment processing has not been applied to the privacy area included in the specific area.

[0022] According to the third aspect, since the second image and the third image contain images in which no concealment processing has been applied to the privacy area contained in the specific area, machine tasks or human vision using privacy information can be appropriately executed on the image decoding device side.

[0023] In the image decoding device according to the fourth aspect of the present disclosure, in the first aspect, the specific area in the first image includes an image in which a concealment process has been applied to a privacy area containing privacy information, and the second image includes an image in which the concealment process has not been applied to the privacy area included in the specific area, and in generating the third image, in applications in which an image in which privacy information has been concealed is used, the third image is generated using the first image, and in applications in which an image in which privacy information has not been concealed is used, the third image is generated using the first image and the second image.

[0024] According to the fourth aspect, the image decoding device can switch between a machine task or human vision using an image in which privacy information is concealed and a machine task or human vision using an image in which privacy information is not concealed.

[0025] In the image decoding device according to the fifth aspect of the present disclosure, in any one of the first to fourth aspects, the designation information may include information indicating the position and size of the specific area within the first image.

[0026] According to the fifth aspect, the position and size of the specific area in the first image can be appropriately specified by the designation information.

[0027] An image decoding device according to a sixth aspect of the present disclosure is the fifth aspect, wherein the designation information further includes information indicating a correspondence between the specific region in the first image and the second image.

[0028] According to the sixth aspect, the correspondence between the specific area in the first image and the second image can be appropriately specified using the designation information.

[0029] In the image decoding device according to the seventh aspect of the present disclosure, in any one of the first to sixth aspects, the circuit may obtain the designation information by decoding a header region of the second bitstream.

[0030] According to the seventh aspect, the designation information is stored in the header area of ​​the second bitstream, so that the designation information can be easily decoded from the second bitstream.

[0031] In the image decoding device of the eighth aspect of the present disclosure, in any one of the first to seventh aspects, the information indicating the position and size of the specific area within the first image among the designation information is stored in a header area of ​​the first bitstream, and the circuit may obtain the information by decoding the header area of ​​the first bitstream.

[0032] According to the eighth aspect, the information originally included in the header area of ​​the first bitstream can be used as the designation information.

[0033] In the image decoding device of the ninth aspect of the present disclosure, in any one of the first to eighth aspects, the second image may include, as the image data, differential image data between the specific area in the first image and the specific area in the third image.

[0034] According to the ninth aspect, the amount of code transmitted from the image encoding device to the image decoding device can be reduced compared to the case where image data of a specific region that has not been subjected to concealment processing is transmitted.

[0035] In the image decoding device according to a tenth aspect of the present disclosure, in any one of the first to eighth aspects, the second image may include, as the image data, image data of the specific region within the third image.

[0036] According to the tenth aspect, the circuit only needs to replace the image data of the specific region in the first image with the image data of the second image, thereby reducing the processing load on the circuit.

[0037] In an image decoding device according to an eleventh aspect of the present disclosure, in any one of the first to tenth aspects, the specific area may include a plurality of specific areas, and the picture of the second bitstream may include the image data of one of the plurality of specific areas.

[0038] According to the eleventh aspect, since the picture of the second bitstream contains image data of only one specific region, it becomes easy to associate each specific region in the first image with each second image.

[0039] In an image decoding device according to a twelfth aspect of the present disclosure, in any one of the first to tenth aspects, the specific area may include a plurality of specific areas, and the picture of the second bitstream may include the image data of the plurality of specific areas.

[0040] According to the twelfth aspect, since each picture of the second bitstream includes image data of a plurality of specific regions, it is possible to reduce the number of pictures of the second bitstream to be transmitted from the image encoding device to the image decoding device.

[0041] In the image decoding device according to the thirteenth aspect of the present disclosure, in any one of the first to twelfth aspects, the specific area may be included within one or more coding unit blocks that constitute a picture of the second bitstream.

[0042] According to the thirteenth aspect, the specific region is included in one or more coding unit blocks, and therefore the specific region can be appropriately decoded in the circuit.

[0043] In the image decoding device according to a fourteenth aspect of the present disclosure, in any one of the first to twelfth aspects, the specific region may be included in a sub-picture or tile that constitutes a picture of the second bitstream.

[0044] According to the fourteenth aspect, the specific region is included in a sub-picture or a tile, so that the specific region can be appropriately decoded in a circuit.

[0045] In an image decoding device according to a fifteenth aspect of the present disclosure, in any one of the first to fourteenth aspects, the specific area may include a plurality of specific areas, and the second image may include the image data corresponding to the plurality of specific areas.

[0046] According to the 15th aspect, even when multiple objects, such as people, move closer to or further away from each other in a captured image, multiple specific regions corresponding to the multiple objects can be included in a single second image.

[0047] In the image decoding device according to a sixteenth aspect of the present disclosure, in any one of the first to fifteenth aspects, it is preferable that the size of the pictures in the first bitstream and the size of the pictures in the second bitstream are the same.

[0048] According to the sixteenth aspect, the size of the pictures in the first bitstream and the size of the pictures in the second bitstream are the same, which makes it easy to associate a specific area in the first image with the second image.

[0049] In the image decoding device according to a seventeenth aspect of the present disclosure, in any one of the first to sixteenth aspects, the pictures of the second bitstream may be intra pictures for which intra prediction is performed.

[0050] According to the seventeenth aspect, since the pictures in the second bitstream are intra-pictures, there is no need to take into account the movement of objects over time, and therefore there is no need to store the previous frame or calculate the difference with the previous frame.

[0051] In an image decoding device according to an 18th aspect of the present disclosure, in any one of the 1st to 16th aspects, the picture of the second bitstream may be an interpicture that performs inter-frame prediction using a reference picture obtained by decoding the second bitstream or the first bitstream.

[0052] According to the eighteenth aspect, the pictures of the second bitstream are interpictures, so that the amount of code can be reduced.

[0053] In an image decoding device according to a 19th aspect of the present disclosure, in any one of the first to 18th aspects, the first bitstream may be transmitted from the image encoding device to the image decoding device using a base layer of a multi-layer system, and the second bitstream may be transmitted from the image encoding device to the image decoding device using an enhancement layer of a multi-layer system.

[0054] According to the nineteenth aspect, the first bit stream and the second bit stream can be easily transmitted using the base layer and enhancement layer of a multi-layer method standardized by VVC or the like.

[0055] In the image decoding device according to the 20th aspect of the present disclosure, in any one of the first to 19th aspects, it is preferable that the first bitstream and the second bitstream are transmitted from the image encoding device to the image decoding device using different transmission paths.

[0056] According to the twentieth aspect, by transmitting the first bit stream using a transmission path such as a public network and transmitting the second bit stream using a transmission path such as a private network, it is possible to achieve both reduction in transmission costs and protection of privacy information.

[0057] An image encoding device according to a twenty-first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, specifies a specific area in an input image, generates designation information for the specific area, generates a first image in which the specific area of ​​the input image is processed, generates a second image including image data corresponding to the specific area, encodes the first image to generate and output a first bitstream, and encodes the designation information and the second image to generate and output a second bitstream.

[0058] According to the twenty-first aspect, by not including the privacy information in the first bitstream, it is possible to protect the privacy information of an individual. Also, by transmitting the second image including the privacy information as the second bitstream from the image encoding device to the image decoding device, and by the image decoding device generating an image including the privacy information based on the first image and the second image, it is possible to execute a machine task or human vision using the privacy information on the image decoding device side.

[0059] An image encoding device according to a 22nd aspect of the present disclosure is the 21st aspect, wherein the specific area in the first image may include an image in which a concealment process has been applied to a privacy area containing privacy information.

[0060] According to the twenty-second aspect, the private information included in the specific area in the first image can be appropriately protected by the concealment process.

[0061] An image encoding device according to a 23rd aspect of the present disclosure is, in the 22nd aspect, preferably, wherein the second image includes an image in which the concealment processing has not been applied to the privacy region included in the specific region.

[0062] According to the 23rd aspect, since the second image and the third image contain images in which no concealment processing has been applied to the privacy area contained in the specific area, machine tasks or human vision using privacy information can be appropriately executed on the image decoding device side.

[0063] In the image encoding device according to the 24th aspect of the present disclosure, in any one of the 21st to 23rd aspects, the designation information may include information indicating the position and size of the specific area within the first image.

[0064] According to the twenty-fourth aspect, the position and size of the specific area in the first image can be appropriately specified by the designation information.

[0065] An image encoding device according to a 25th aspect of the present disclosure is the 24th aspect, wherein the designation information further includes information indicating a correspondence between the specific region in the first image and the second image.

[0066] According to the twenty-fifth aspect, the correspondence between the specific area in the first image and the second image can be appropriately specified using the designation information.

[0067] In the image encoding device according to a 26th aspect of the present disclosure, in any one of the 21st to 25th aspects, the circuit may store the designation information in a header area of ​​the second bitstream.

[0068] According to the 26th aspect, the designation information is stored in the header area of ​​the second bitstream, so that the image decoding apparatus can easily decode the designation information from the second bitstream.

[0069] In the image encoding device according to the 27th aspect of the present disclosure, in any one of the 21st to 26th aspects, the circuit may store, in the header area of ​​the first bitstream, information indicating the position and size of the specific area within the first image, among the designation information.

[0070] According to the twenty-seventh aspect, the information originally included in the header area of ​​the first bitstream can be used as the designation information.

[0071] In the image encoding device according to the 28th aspect of the present disclosure, in any one of the 21st to 27th aspects, the second image may include, as the image data, differential image data between the specific area in the input image and the specific area in the first image.

[0072] According to the 28th aspect, the amount of code transmitted from the image encoding device to the image decoding device can be reduced compared to the case where image data of a specific region that has not been subjected to concealment processing is transmitted.

[0073] In the image encoding device according to the 29th aspect of the present disclosure, in any one of the 21st to 27th aspects, the second image may include, as the image data, image data of the specific region within the input image.

[0074] According to the 29th aspect, the image decoding device only needs to replace the image data of the specific region in the first image with the image data of the second image, thereby reducing the processing load on the image decoding device.

[0075] In an image encoding device relating to the 30th aspect of the present disclosure, in any one of the 21st to 29th aspects, it is preferable that the specific area includes multiple specific areas, and the picture of the second bitstream includes the image data of one of the multiple specific areas.

[0076] According to the 30th aspect, since the picture of the second bitstream contains image data of only one specific region, it becomes easy to associate each specific region in the first image with each second image.

[0077] In an image encoding device according to the 31st aspect of the present disclosure, in any one of the 21st to 29th aspects, the specific area may include a plurality of specific areas, and the picture of the second bitstream may include the image data of the plurality of specific areas.

[0078] According to the thirty-first aspect, since each picture of the second bitstream includes image data of a plurality of specific regions, it is possible to reduce the number of pictures of the second bitstream to be transmitted from the image encoding device to the image decoding device.

[0079] In the image coding device according to the 32nd aspect of the present disclosure, in any one of the 21st to 31st aspects, it is preferable that the specific area is included within one or more coding unit blocks that constitute a picture of the second bitstream.

[0080] According to the thirty-second aspect, the specific region is included in one or more coding unit blocks, and therefore the specific region can be appropriately decoded in the image decoding device.

[0081] In the image encoding device according to the 33rd aspect of the present disclosure, in any one of the 21st to 31st aspects, the specific region may be included in a sub-picture or tile that constitutes a picture of the second bitstream.

[0082] According to the thirty-third aspect, the specific region is included in a sub-picture or a tile, and therefore the specific region can be appropriately decoded in the image decoding device.

[0083] In an image encoding device according to a 34th aspect of the present disclosure, in any one of the 21st to 33rd aspects, the specific area may include a plurality of specific areas, and the second image may include the image data corresponding to the plurality of specific areas.

[0084] According to the 34th aspect, even when multiple objects, such as people, move closer to or further away from each other in a captured image, multiple specific areas corresponding to the multiple objects can be included in a single second image.

[0085] In the image encoding device according to the 35th aspect of the present disclosure, in any one of the 21st to 34th aspects, it is preferable that the size of the pictures in the first bitstream and the size of the pictures in the second bitstream are the same.

[0086] According to the thirty-fifth aspect, the size of the pictures in the first bit stream and the size of the pictures in the second bit stream are the same, which makes it easy to associate a specific area in the first image with the second image.

[0087] In the image encoding device according to a 36th aspect of the present disclosure, in any one of the 21st to 35th aspects, the pictures of the second bitstream may be intra pictures for which intra prediction is performed.

[0088] According to the 36th aspect, since the pictures in the second bitstream are intra-pictures, there is no need to take into account the movement of objects over time, and therefore there is no need to store the previous frame or calculate the difference with the previous frame.

[0089] In the image encoding device according to the 37th aspect of the present disclosure, in any one of the 21st to 35th aspects, the picture of the second bitstream may be an interpicture that performs inter-frame prediction using a reference picture obtained by decoding the second bitstream or the first bitstream.

[0090] According to the thirty-seventh aspect, the pictures of the second bitstream are interpictures, so that the amount of code can be reduced.

[0091] In the image encoding device according to the 38th aspect of the present disclosure, in any one of the 21st to 37th aspects, the circuit may transmit the first bitstream to the image decoding device using a base layer of a multi-layer system, and transmit the second bitstream to the image decoding device using an enhancement layer of a multi-layer system.

[0092] According to the thirty-eighth aspect, the first bit stream and the second bit stream can be easily transmitted using the base layer and enhancement layer of a multi-layer method standardized by VVC or the like.

[0093] In the image encoding device according to the 39th aspect of the present disclosure, in any one of the 21st to 38th aspects, the circuit may transmit the first bitstream and the second bitstream to the image decoding device using different transmission paths.

[0094] According to the 39th aspect, by transmitting the first bit stream using a transmission path such as a public network and transmitting the second bit stream using a transmission path such as a private network, it is possible to achieve both reduction in transmission costs and protection of privacy information.

[0095] An image decoding method according to the 40th aspect of the present disclosure obtains a first image by decoding a first bitstream, obtains designation information that designates a specific area within the first image and a second image that includes image data of the specific area by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

[0096] According to the fortieth aspect, by not including the privacy information in the first bitstream, it is possible to protect the privacy information of an individual. Furthermore, by transmitting a second image including the privacy information as the second bitstream and generating a third image based on the second image, it is possible to execute a machine task or human vision using the privacy information on the image decoding device side.

[0097] An image encoding method according to a forty-first aspect of the present disclosure specifies a specific area in an input image, generates designation information for the specific area, generates a first image by processing the specific area of ​​the input image, generates a second image including image data corresponding to the specific area, encodes the first image to generate and output a first bitstream, and encodes the designation information and the second image to generate and output a second bitstream.

[0098] According to the forty-first aspect, by not including the privacy information in the first bitstream, it is possible to protect the privacy information of an individual. Also, by transmitting the second image including the privacy information as the second bitstream from the image encoding device to the image decoding device, and by the image decoding device generating an image including the privacy information based on the first image and the second image, it is possible to execute a machine task or human vision using the privacy information on the image decoding device side.

[0099] (Embodiments of the Present Disclosure) Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0100] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0101] 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an image encoding device 1, transmission paths NW1 and NW2, and an image decoding device 2.

[0102] The image encoding device 1 includes an area designation unit 11 , a first image generation unit 12 , a second image generation unit 13 , a first encoding unit 14 , and a second encoding unit 15 .

[0103] The region designation unit 11 designates a specific region in the input image D1 and generates designation information D2 for the specific region. The first image generation unit 12 generates a first image D3 by processing the specific region of the input image D1. The second image generation unit 13 generates a second image D4 including image data corresponding to the specific region. The first encoding unit 14 generates and outputs a first bit stream D5 by encoding the first image D3. The first bit stream D5 is transmitted to the image decoding device 2 via a transmission path NW1. The second encoding unit 15 generates and outputs a second bit stream D6 by encoding the designation information D2 and the second image D4. The second bit stream D6 is transmitted to the image decoding device 2 via a transmission path NW2 different from the transmission path NW1.

[0104] The transmission paths NW1 and NW2 may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. Furthermore, the transmission paths NW1 and NW2 are not necessarily limited to bidirectional communication networks, but may also be unidirectional communication networks transmitting broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, the transmission paths NW1 and NW2 may also be recording media such as DVDs (Digital Versatile Discs) or BDs (Blue-Ray Discs) on which the first bit stream D5 or the second bit stream D6 is recorded. The transmission path NW1 is a public network, etc., and the transmission path NW2 is a private network, etc., in which secure communication is ensured by access restrictions. Note that if different access restrictions can be set for the first bit stream D5 and the second bit stream D6, the transmission paths NW1 and NW2 may physically be the same communication network or recording medium.

[0105] The image decoding device 2 includes a first decoding unit 21, a second decoding unit 22, and an image generating unit 23.

[0106] The first decoding unit 21 receives, via the transmission path NW1, the first bit stream D5 transmitted by the first encoding unit 14. The first decoding unit 21 decodes the received first bit stream D5 to obtain a first image D7 corresponding to the first image D3.

[0107] The second decoding unit 22 receives, via the transmission path NW2, the second bit stream D6 transmitted by the second encoding unit 15. The second decoding unit 22 decodes the received second bit stream D6 to obtain designation information D8 corresponding to the designation information D2 and a second image D9 corresponding to the second image D4.

[0108] The image generation unit 23 generates a third image D10 corresponding to the input image D1 based on the first image D7, the designation information D8, and the second image D9. Specifically, for example, in a use case using an image that does not include privacy information (i.e., an application using an image in which privacy information is concealed), the image generation unit 23 generates the third image D10 by using the first image D7 as is. Use cases using an image that does not include privacy information include, for example, viewing by a user without special access privileges, or use for machine tasks that do not require detailed information such as faces or vehicle license plates. Furthermore, for example, in a use case using an image that includes privacy information (i.e., an application using an image in which privacy information is not concealed), the image generation unit 23 generates the third image D10 by using the first image D7 and the second image D9 for image data within a specific region based on the designation information D8, and generates the third image D10 by using the first image D7 for image data outside the specific region. Use cases in which an image including privacy information is used include, for example, viewing by a user with special access privileges, or for a machine task that requires detailed information such as a face or a vehicle license plate. In this way, by switching the generation of the third image D10 depending on the use of the image, the image decoding device 2 can switch between a machine task or human vision using an image in which privacy information is concealed and a machine task or human vision using an image in which privacy information is not concealed. Machine tasks include object detection, object segmentation, object tracking, action recognition, pose estimation, etc.

[0109] FIG. 12 is a simplified diagram illustrating an example implementation of the image encoding device 1. The image encoding device 1 includes a processor 101 and a memory 102 connected to the processor 101. However, the memory 102 may be included within the processor 101. The processor 101 is a circuit that performs information processing. The processor 101 includes a CPU or a GPU, etc. The memory 102 includes a semiconductor memory such as a ROM or a RAM, a magnetic disk, an optical disk, etc. The memory 102 stores information necessary for the processor 101 to execute encoding processing, etc. For example, the memory 102 stores an input image D1, designation information D2, a first image D3, and a second image D4. The memory 102 also stores a program. By the processor 101 executing the program read from the memory 102, the processor 101 functions as a region designation unit 11, a first image generation unit 12, a second image generation unit 13, a first encoding unit 14, and a second encoding unit 15.

[0110] FIG. 13 is a simplified diagram illustrating an example implementation of the image decoding device 2. The image decoding device 2 includes a processor 201 and a memory 202 connected to the processor 201. However, the memory 202 may be included within the processor 201. The processor 201 is a circuit that performs information processing. The processor 201 includes a CPU, a GPU, or the like. The memory 202 includes a semiconductor memory such as a ROM or a RAM, a magnetic disk, an optical disk, or the like. The memory 202 stores information necessary for the processor 201 to execute a decoding process, etc. For example, the memory 202 stores a first bitstream D5, a second bitstream D6, a first image D7, designation information D8, and a second image D9. The memory 202 also stores a program. The processor 201 executes the program read from the memory 202, thereby functioning as a first decoding unit 21, a second decoding unit 22, and an image generation unit 23.

[0111] FIG. 2 is a flowchart showing the flow of processing executed by the image encoding device 1.

[0112] First, in step SP11, the area designation unit 11 designates a specific area in the input image D1 and generates designation information D2 for the specific area.

[0113] Next, in step SP12, the first image generating unit 12 generates a first image D3 by processing a specific region of the input image D1.

[0114] Next, in step SP13, the second image generating unit 13 generates a second image D4 including image data corresponding to the specific region.

[0115] Next, in step SP14, the first encoding unit 14 generates and outputs a first bitstream D5 by encoding the first image D3.

[0116] Next, in step SP15, the second encoding unit 15 generates and outputs a second bit stream D6 by encoding the specification information D2 and the second image D4.

[0117] 3 is a diagram showing an example of an input image D1. The input image D1 includes an image 31 of a person's face, an image 32 of the whole body including the person's face, and an image 33 of a background. The person's face is an example of personal privacy information. In addition to the person's face, the privacy information also includes a vehicle license plate, a house nameplate, etc.

[0118] 4 is a diagram showing bounding boxes 41 and 42 as examples of specific regions. The region designation unit 11 sets the bounding box 41, which is a rectangular box that surrounds the periphery of a privacy region including image 31, as one of the specific regions. The region designation unit 11 also sets the bounding box 42, which is a rectangular box that surrounds the periphery of a privacy region that includes image 32, as one of the specific regions. Since image 33 does not contain privacy information, no bounding box is set for it.

[0119] The designation information D2 includes information indicating the positions and sizes of the bounding boxes 41 and 42 within the first image D3.

[0120] 5A to 5F are diagrams illustrating examples of designation information D2. As shown in FIG. 5A, designation information D2 includes the position coordinates (X1, Y1) of a specific vertex of the bounding box, the vertical size H of the bounding box, and the horizontal size W of the bounding box. As shown in FIG. 5B, designation information D2 includes the position coordinates (X1, Y1) of a specific vertex of the bounding box and the position coordinates (X4, Y4) of the vertex diagonally opposite the specific vertex. As shown in FIG. 5C, designation information D2 includes the position coordinates (X0, Y0) of the center of the bounding box, the vertical size H of the bounding box, and the horizontal size W of the bounding box. As shown in FIG. 5D, designation information D2 includes the position coordinates (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) of the four vertices of the bounding box. In this case, as shown in FIG. 5E, the position and size of a tilted bounding box can also be specified. As shown in FIG. 5F, the designation information D2 includes mask information in which pixel positions within the specific region are represented by "1" and pixel positions outside the specific region are represented by "0".

[0121] Furthermore, the designation information D2 includes information indicating the correspondence between the bounding boxes 41, 42 in the first image D3 and the second image D4. Note that if the second image D4 includes multiple bounding boxes 41, 42, the designation information D2 includes information indicating the correspondence between the bounding boxes 41, 42 in the first image D3 and the bounding boxes 41, 42 in the second image D4.

[0122] FIG. 6 is a diagram showing an example of a first image D3. The area within the bounding boxes 41 and 42 included in the first image D3 includes an image in which a concealment process has been applied to a privacy area containing privacy information. The first image generation unit 12 applies a concealment process to the privacy area in the input image D1 based on the designation information D2 input from the area designation unit 11. The concealment process includes blurring, mosaic processing, or silhouette processing. FIG. 6 shows an example in which silhouette processing has been applied.

[0123] 7A to 7H are diagrams showing an example of the second image D4.

[0124] As shown in FIG. 7A , the second image D4A has the same picture size as the input image D1. The rectangular areas in the second image D4A defined by dashed lines correspond to coding unit blocks. The coding unit blocks are coding units (CUs) or coding tree units (CTUs), and have a size of, for example, 8×8 pixels. The image data in the areas outside the bounding boxes 41 and 42 in the second image D4A may be the same as that in the input image D1, or may be padded with meaningless data (e.g., "0"). The positions and sizes of the bounding boxes 41 and 42 in the second image D4A are the same as those in the input image D1. Because the picture sizes of the input image D1 and the second image D4A are the same, the first encoding unit 14 encodes the image in accordance with H.264 encoding. The first bitstream D5 can be transmitted to the image decoding device 2 using a base layer of a multi-layer system standardized in H.266 / VVC (Versatile Video Coding) or the like. The second encoding unit 15 can transmit a second bitstream D6 to the image decoding device 2 using an enhancement layer of the multi-layer system. When the multi-layer system is used, the enhancement layer can be encoded as differential information from the base layer. Therefore, differential information of the entire picture between the first image D3 in which privacy information is concealed and the input image D1 in which privacy information is not concealed may be encoded as the second bitstream D6 without setting a bounding box. The image decoding device 2 allows any user to access the base layer, thereby enabling machine tasks or human vision to be performed using images in which privacy information is concealed. Furthermore, by allowing only users with special privileges to access the enhancement layer, machine tasks or human vision to be performed using images in which privacy information is not concealed.

[0125] The second image D4 includes an image in which the privacy region included in the bounding boxes 41 and 42 has not been concealed. The second image includes image data corresponding to the specific region as an image in which the concealment process has not been performed. The image data corresponding to the specific region may be differential image data between the specific region in the input image D1 and the specific region in the first image D3, or may be the image data of the specific region in the input image D1 itself. When the specific region of the second image D4 is differential image data, the first image D3 and the second image D4 are added together to obtain an image before the concealment process has been performed on the privacy region. When the specific region of the second image D4 is the image data of the input image D1, the first image D3 is multiplied by a weight value α and the second image D4 is multiplied by a weight value β to obtain an image before the concealment process has been performed on the privacy region. The weight values ​​α and β are both between 0 and 1, and satisfy the relationship α + β = 1. When α=0 and β=1, the specific region of the second image D4 becomes the image data of the input image D1 itself, and in this case, the image before the concealment process is performed on the privacy region is obtained without using any image data of the specific region of the first image D3. Also, β may be defined as 1-α, and only one weight value α may be used.

[0126] The second image D4 may be an intra-picture coded by intra-prediction. That is, the second coding unit 15 may code the second image D4 using intra-prediction. Because the picture of the second bitstream D6 is an intra-picture, there is no need to consider the movement of an object over time, and therefore there is no need to store the previous frame or calculate the difference with the previous frame.

[0127] On the other hand, the second image D4 may be an interpicture coded by inter prediction. That is, the second coding unit 15 may code the second image D4 using inter prediction. As a reference picture used for inter prediction, a picture of the second image D4 obtained by decoding (local decoding) the second bitstream D6 may be used, or a picture of the first image D3 obtained by decoding (local decoding) the first bitstream D5 may be used. By using an interpicture as the picture of the second bitstream D6, the amount of coding can be reduced.

[0128] Although the second image D4A shown in FIG. 7A includes a plurality of bounding boxes 41 and 42, the bounding box 41 and the bounding box 42 may be included in different second images D4B and D4C.

[0129] As shown in Figure 7B, the second image D4B includes only a bounding box 41. The second image D4B has the same picture size as the input image D1. The image data of the area outside the bounding box 41 in the second image D4B may be the same as that in the input image D1, or may be padded with meaningless data. The position and size of the bounding box 41 in the second image D4B are the same as those in the input image D1.

[0130] As shown in Figure 7C, the second image D4C includes only the bounding box 42. The second image D4C has the same picture size as the input image D1. The image data of the area outside the bounding box 42 in the second image D4C may be the same as that in the input image D1, or may be padded with meaningless data. The position and size of the bounding box 42 in the second image D4C are the same as those of the bounding box 42 in the input image D1.

[0131] 7D, the second image D4D includes only the bounding box 41. Considering that the size of the bounding box 41 changes for each frame, the second image D4D is configured with the minimum number of coding unit blocks (four in this example) that encompass the bounding box 41, with the bounding box 41 packed in the upper left corner. Image data in the area of ​​the second image D4D outside the bounding box 41 is padded with meaningless data.

[0132] 7E, the second image D4E includes only the bounding box 42. Considering that the size of the bounding box 42 changes for each frame, the second image D4E is configured with the minimum number of coding unit blocks (15 in this example) that encompass the bounding box 42, with the bounding box 42 packed in the upper left corner. Image data in the area of ​​the second image D4E outside the bounding box 42 is padded with meaningless data.

[0133] As shown in Figure 7F, the second image D4F includes bounding boxes 41 and 42. The 15 coding unit blocks that include the bounding box 42 are arranged to the right of the four coding unit blocks that include the bounding box 41. Furthermore, to define the second image D4F as a rectangular area, six coding unit blocks are arranged below the four coding unit blocks that include the bounding box 41. Image data in the area of ​​the second image D4F outside the bounding boxes 41 and 42 is padded with meaningless data.

[0134] As shown in Figure 7G, the second image D4G includes bounding boxes 41 and 42. The picture of the second image D4G is divided into multiple sub-pictures standardized by VVC or the like. The bounding box 41 is included in the sub-picture in the upper left corner, and the bounding box 42 is included in the sub-picture adjacent to the right. Image data in the area of ​​the second image D4G outside the bounding boxes 41 and 42 is padded with meaningless data.

[0135] As shown in Figure 7H, the second image D4H includes bounding boxes 41 and 42. The picture of the second image D4H is divided into multiple tiles standardized by VVC or the like. The bounding box 41 is included in the tile in the upper left corner, and the bounding box 42 is included in the tiles adjacent to the tile to the right and below. Image data in the areas of the second image D4H outside the bounding boxes 41 and 42 is padded with meaningless data.

[0136] FIG. 8 is a diagram showing a first example of a second image D4I in which multiple bounding boxes 41 and 42 partially overlap. As shown in FIG. 8, the second image D4I includes bounding boxes 41 and 42. Due to the two people moving closer to each other, the bounding boxes 41 and 42 partially overlap. The second image D4I is composed of the minimum number of coding unit blocks (24 in this example) that encompass the bounding boxes 41 and 42, with the bounding box 41 moved toward the upper left corner. Image data in the areas of the second image D4I outside the bounding boxes 41 and 42 is padded with meaningless data. When the bounding boxes 41 and 42 no longer overlap due to the two people moving farther apart, the second image D4I is divided into the second image D4D shown in FIG. 7D and the second image D4E shown in FIG. 7E. Even if the bounding boxes 41, 42 partially overlap, the image may be divided into the second image D4D shown in Fig. 7D and the second image D4E shown in Fig. 7E while including the overlapping portions in accordance with the specifications of the bounding boxes 41, 42. The second image D4I may also be defined by subpictures or tiles instead of by multiple coding unit blocks.

[0137] 9A and 9B are diagrams showing second examples of second images D4J and D4K in which multiple bounding boxes 41 and 42 partially overlap. As shown in FIG. 9A, the second image D4J includes bounding boxes 41 and 42. The bounding boxes 41 and 42 partially overlap due to two people approaching each other. The second image D4I is composed of a subpicture that includes the bounding boxes 41 and 42, with the bounding box 41 moved to the upper left corner. The number of coding unit blocks included in the subpicture is set to a number (35 in this example) greater than the minimum number of coding unit blocks (24 in this example) required to include the bounding boxes 41 and 42. Image data in the areas of the second image D4J outside the bounding boxes 41 and 42 is padded with meaningless data.

[0138] When the bounding boxes 41 and 42 no longer overlap due to the two people moving farther apart, the overlap between the bounding boxes 41 and 42 is resolved in the second image D4K as shown in Figure 9B. The second image D4K includes bounding boxes 41 and 42 that do not overlap. When the bounding box 42 moves out of the range of the second image D4K due to the two people moving further apart, the second image D4K is divided into different sub-pictures, such as the second image D4G shown in Figure 7G. Note that the second images D4J and D4K may be defined by tiles or multiple coding unit blocks instead of sub-pictures.

[0139] 10A is a diagram showing a first example of the data structure of bitstream D6. Bitstream D6 has a header area H in which management information and the like are stored, and a payload area P in which image data is stored. The second encoding unit 15 stores the encoded data of the second image D4 in the payload area P. The second encoding unit 15 also stores the encoded data 70 of the designation information D2 in a predetermined location in the payload area P.

[0140] FIG. 10B illustrates a second example of the data structure of the bitstream D6. Similar to FIG. 10A , the bitstream D6 has a header area H and a payload area P. The second encoding unit 15 stores the encoded data of the second image D4 in the payload area P. The second encoding unit 15 also stores the encoded data 70 of the designation information D2 in a predetermined location in the header area H. The predetermined location is, for example, a supplemental enhancement information (SEI) area for storing additional information. The predetermined location may be a VPS, SPS, PPS, PH, SH, APS, or tile header. By storing the designation information D2 in the header area H of the bitstream D6, the second decoding unit 22 can easily obtain the designation information D8 by decoding. Furthermore, by storing the designation information D2 in the SEI area, the designation information D2 can be easily handled as additional information.

[0141] The first encoding unit 14 may store information from the designation information D2 indicating the positions and sizes of the bounding boxes 41, 42 within the first image D3 in a predetermined location in the header region H of the first bitstream D5. The predetermined location may be, for example, an annotated region SEI (ARSEI) region for storing bounding box information. This allows the bounding box information originally included in the header region H of the first bitstream D5 to be used as part of the designation information D2.

[0142] FIG. 11 is a flowchart showing the flow of processing executed by the image decoding device 2.

[0143] First, in step SP21, the first decoding unit 21 decodes the first bitstream D5 received from the image encoding device 1 to obtain a first image D7 corresponding to the first image D3.

[0144] Next, in step SP22, the second decoding unit 22 decodes the second bit stream D6 received from the image encoding device 1 to obtain designation information D8 corresponding to designation information D2 and a second image D9 corresponding to second image D4.

[0145] Next, in step SP23, the image generating unit 23 generates a third image D10 corresponding to the input image D1 based on the first image D7, the designation information D8, and the second image D9.

[0146] Similar to the first image D3 in the image encoding device 1, the area within the bounding boxes 41 and 42 included in the first image D7 in the image decoding device 2 includes an image in which a concealment process has been applied to a privacy area containing privacy information. The concealment process includes blurring, mosaic processing, or silhouette processing.

[0147] Similar to the second image D4 and input image D1 in the image encoding device 1, the second image D9 and third image D10 in the image decoding device 2 include images in which no concealment processing has been applied to the privacy regions included in the bounding boxes 41 and 42. The second image D9 may include differential image data between the bounding boxes 41 and 42 in the first image D7 and the bounding boxes 41 and 42 in the third image D10, or may include image data of the bounding boxes 41 and 42 in the third image D10.

[0148] The second decoding unit 22 obtains the designation information D8 by decoding the payload area P or the header area H of the second bitstream D6. Note that, of the designation information D8, information indicating the positions and sizes of the bounding boxes 41, 42 within the first image D7 may be stored in the header area H of the first bitstream D5. In this case, the first decoding unit 21 obtains the information by decoding the header area H of the first bitstream D5.

[0149] Similar to the designation information D2 in the image encoding device 1, the designation information D8 in the image decoding device 2 includes information indicating the positions and sizes of the bounding boxes 41 and 42 in the first image D7. The designation information D8 may further include information indicating the correspondence between the bounding boxes 41 and 42 in the first image D7 and the second image D9.

[0150] Similar to the second image D4 in the image encoding device 1, the second image D9 in the image decoding device 2 may include image data of one of the bounding boxes 41, 42, or may include image data of multiple bounding boxes 41, 42. Furthermore, in the second image D9, the bounding boxes 41, 42 may be included in one or more coding unit blocks constituting a picture of the second bitstream D6, or may be included in a subpicture or tile constituting a picture of the second bitstream D6. Furthermore, the second image D9 may include image data corresponding to the multiple bounding boxes 41, 42.

[0151] Furthermore, the size of the pictures in the first bitstream D5 and the size of the pictures in the second bitstream D6 may be the same. The first bitstream D5 may be transmitted from the image encoding device 1 to the image decoding device 2 using a base layer in a multi-layer system, and the second bitstream D6 may be transmitted from the image encoding device 1 to the image decoding device 2 using an enhancement layer in a multi-layer system. The first bitstream D5 and the second bitstream D6 may be transmitted from the image encoding device 1 to the image decoding device 2 using different transmission paths NW1 and NW2. The pictures in the second bitstream D6 may be intra-pictures for intra-picture prediction or inter-pictures for inter-picture prediction. As a reference picture used for inter-picture prediction, a picture of the second image D9 obtained by decoding the second bitstream D6 may be used, or a picture of the first image D7 obtained by decoding the first bitstream D5 may be used.

[0152] (Effects, etc.) According to the image encoding device 1 and the image decoding device 2 of this embodiment, it is possible to protect an individual's privacy information by not including the privacy information in the first bitstream D5. Furthermore, by transmitting the second image D4 including the privacy information as the second bitstream D6 and having the image decoding device 2 generate the third image D10 based on the second image D9, it is possible for the image decoding device 2 to execute a machine task or human vision using the privacy information.

[0153] The present disclosure is particularly useful when applied to an image processing system that includes an image encoding device that transmits images and an image decoding device that receives images.

Claims

1. A circuit, a memory connected to the circuit; Equipped with The circuit, in operation, Obtaining the image and the designated information from the bitstream; The designation information indicates that a specific area including privacy information in the image has been processed. Image decoding device.

2. The processing includes at least one of blurring, mosaic processing, or silhouette processing. The image decoding device according to claim 1 .

3. The privacy information includes at least one of information indicating a person's face, a vehicle license plate, or an address, The image decoding device according to claim 1 .

4. The circuit generates a generated image based on the image and the specified information. The image decoding device according to claim 1 .

5. The designation information includes information indicating a position and a size of the specific area within the image. The image decoding device according to claim 1 .

6. The circuit obtains the designation information from a header area of ​​the bitstream. The image decoding device according to claim 1 .

7. The header region includes an SEI. The image decoding device according to claim 6.

8. The specific region includes a plurality of specific regions, The circuit generates a packed image in which the plurality of specific regions are packed. The image decoding device according to claim 1 .

9. The specific area in the packed image is included in one or more coding unit blocks constituting a picture of the bitstream. The image decoding device according to claim 8 .

10. The specific region in the packed image is included in a subpicture or a tile constituting a picture of the bitstream. The image decoding device according to claim 8 .

11. The designation information includes information indicating a correspondence relationship between a specific area in the image and a specific area in the packed image. The image decoding device according to claim 8 .

12. The packed image is such that areas other than the plurality of specific areas are padded. The image decoding device according to claim 8 .

13. Some of the plurality of specific regions have regions that overlap each other. The image decoding device according to claim 8 .

14. A circuit, comprising: a memory connected to the circuit; Equipped with The circuit, in operation, Obtaining a first image from a first bitstream; acquiring, from a second bit stream, designation information designating a specific region in the first image and a second image including image data of the specific region; generating a third image based on the first image, the designation information, and the second image; Image decoding device.

15. The first image is transmitted from an image encoding device to the image decoding device using a base layer of a multi-layer scheme; The second image is transmitted from the image encoding device to the image decoding device using an enhancement layer in a multi-layer scheme. The image decoding device according to claim 14.

16. A circuit, comprising: a memory connected to the circuit; Equipped with The circuit, in operation, Identifying a specific region including privacy information from an input image; generating an image in which the specific region has been processed; generating designation information indicating that processing has been performed on the specific region in the image; encoding the image and the designation information; Image encoding device.

17. The processing includes at least one of blurring, mosaic processing, or silhouette processing. The image encoding device according to claim 16.

18. The privacy information includes at least one of information indicating a person's face, a vehicle license plate, or an address. The image encoding device according to claim 16.

19. The designation information includes information indicating a position and a size of the specific area within the image. The image encoding device according to claim 16.

20. The circuit writes the designation information into a header area of ​​the bit stream. The image encoding device according to claim 16.

21. The header region includes an SEI. The image encoding device according to claim 20.

22. The specific region includes a plurality of specific regions, The circuit generates a packed image in which the plurality of specific regions are packed. The image encoding device according to claim 16.

23. The specific area in the packed image is included in one or more coding unit blocks constituting a picture of the bit stream.

23. An image encoding device according to claim 22.

24. The specific region in the packed image is included in a subpicture or a tile constituting a picture of the bitstream.

23. An image encoding device according to claim 22.

25. The designation information includes information indicating a correspondence relationship between a specific area in the image and a specific area in the packed image.

23. An image encoding device according to claim 22.

26. The packed image is such that areas other than the plurality of specific areas are padded.

23. An image encoding device according to claim 22.

27. ​​Some of the plurality of specific regions have regions that overlap each other.

23. An image encoding device according to claim 22.

28. A circuit, comprising: a memory connected to the circuit; Equipped with The circuit, in operation, Identifying a specific area in an input image and generating designation information for the specific area; generating a first image by processing the specific region of the input image; generating a second image including image data corresponding to the identified region; generating a first bitstream by encoding the first image; generating a second bitstream by encoding the designation information and the second image; Image encoding device.

29. The first image is transmitted from the image encoding device to the image decoding device using a base layer of a multi-layer scheme; The second image is transmitted from the image encoding device to the image decoding device using an enhancement layer in a multi-layer scheme.

29. An image encoding device according to claim 28.

30. Obtaining an image and specific information from a bitstream, The designation information indicates that a specific area including privacy information in the image has been processed. Image decoding method.

31. A method for identifying a specific area including privacy information from an input image, generating an image in which the specific region has been processed; generating designation information indicating that processing has been performed on the specific region in the image; encoding the image and the designation information; Image encoding method.

32. Obtaining a first image from a first bitstream; acquiring, from a second bit stream, designation information designating a specific region in the first image and a second image including image data of the specific region; generating a third image based on the first image, the designation information, and the second image; Image decoding method.

33. Identifying a specific area in an input image, and generating designation information for the specific area; generating a first image by processing the specific region of the input image; generating a second image including image data corresponding to the identified region; generating a first bitstream by encoding the first image; generating a second bitstream by encoding the designation information and the second image; Image encoding method.