Image encoding device, image decoding device, image encoding method, and image decoding method

Abstract

The present invention is provided with: a circuit; and a memory connected to the circuit. In operation, the circuit decodes a first bitstream to acquire a first image, decodes a second bitstream to acquire specification information specifying a specific region in the first image and a second image including image data of the specific region, and generates a third image on the basis of the first image, the specification information, and the second image.

Description

Technical Field

[0001] The present disclosure relates to an image encoding device, an image decoding device, an image encoding method, and an image decoding method.

Background Art

[0002] Patent Document 1 discloses a video encoding and decoding method using an adaptive combined prefilter and a postfilter.

[0003] Patent Document 2 discloses an encoding method for image data to be loaded into an artificial intelligence (AI) integrated circuit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

[0005] An object of the present disclosure is to achieve both protection of personal privacy information and execution of a machine task or human vision using privacy information on the image decoding device side in the transmission of an image from an image encoding device to an image decoding device.

[0006] An image decoding device according to an aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit obtains a first image by decoding a first bitstream, and obtains designation information for designating a specific region in the first image and a second image including image data of the specific region by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

Brief Description of the Drawings

[0007] [Figure 1] This figure shows a simplified configuration of the image processing system according to the embodiment of this disclosure. [Figure 2] This is a flowchart showing the processing flow performed by the image encoding device. [Figure 3] This figure shows an example of an input image. [Figure 4] This figure shows a bounding box as an example of a specific region. [Figure 5A] This figure shows an example of the specified information. [Figure 5B] This figure shows an example of the specified information. [Figure 5C] This figure shows an example of the specified information. [Figure 5D] This figure shows an example of the specified information. [Figure 5E] This figure shows an example of the specified information. [Figure 5F] This figure shows an example of the specified information. [Figure 6] This is a diagram showing an example of the first image. [Figure 7A] This is a diagram showing an example of the second image. [Figure 7B] This is a diagram showing an example of the second image. [Figure 7C] This is a diagram showing an example of the second image. [Figure 7D] This is a diagram showing an example of the second image. [Figure 7E] This is a diagram showing an example of the second image. [Figure 7F] This is a diagram showing an example of the second image. [Figure 7G] This is a diagram showing an example of the second image. [Figure 7H] This is a diagram showing an example of the second image. [Figure 8] This figure shows the first example of the second image when parts of multiple bounding boxes overlap. [Figure 9A] This figure shows a second example of the second image where parts of multiple bounding boxes overlap. [Figure 9B]A diagram showing a second example of a second image when parts of a plurality of bounding boxes overlap. [Figure 10A] A diagram showing a first example of the data structure of a bitstream. [Figure 10B] A diagram showing a first example of the data structure of a bitstream. [Figure 11] A flowchart showing the flow of processing executed by an image decoding device. [Figure 12] A diagram showing a simplified implementation example of an image encoding device. [Figure 13] A diagram showing a simplified implementation example of an image decoding device.

Embodiments for Carrying Out the Invention

[0008] (Knowledge on which the present disclosure is based) Conventional coding methods aimed to provide optimal video under bitrate constraint conditions for human vision.

[0009] With the development of machine learning or neural network-based applications along with rich sensors, many intelligent platforms handling large amounts of data, including connected cars, video surveillance, or smart cities, etc., have been realized. Since a large amount of data is constantly generated, conventional methods including humans in the pipeline are inefficient and unrealistic in terms of latency and scale.

[0010] <000'0102>Furthermore, in transmission and archival systems, there is a concern that more compact data representation and low-latency solutions are required, and for this purpose, VCM (Video Coding for Machines) has been introduced. <{

[0011] In some cases, machines can communicate with each other and perform tasks without human intervention, while in other cases, additional processing by a human may be required on a specific decompressed stream. For example, in the case of surveillance cameras, a human "supervisor" may need to search for a specific person or scene in the footage.

[0012] In other cases, the corresponding bitstream may be used by both humans and machines. In connected cars, features can be used for image correction for humans and for object detection and segmentation for machines.

[0013] A typical system architecture includes a pair of image encoders and image decoders. The system's input can be video, still images, or feature data. Examples of machine tasks include object detection, object segmentation, object tracking, action recognition, pose estimation, or any combination thereof. Human vision is another potential use case that can be applied alongside machine tasks.

[0014] According to conventional technology, if a captured image contains private information such as an individual's face or a vehicle's license plate, even if this information is not necessary for the machine task, the image containing the private information is transmitted from the image encoding device to the image decoding device, potentially leading to the leakage of private information. On the other hand, it is conceivable to transmit an image with the private information obscured, such as by blurring, from the image encoding device to the image decoding device, but this presents the problem that the private information cannot be used when it is necessary for the machine task or human vision.

[0015] To solve this problem, the inventors have found that the problem can be solved by separately transmitting an image bitstream containing privacy information and an image bitstream without privacy information from an image encoding device to an image decoding device, and then combining them in the image decoding device when privacy information is required. This led to the invention of this disclosure.

[0016] Next, we will describe each aspect of this disclosure.

[0017] An image decoding device according to a first aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a first image by decoding a first bitstream, acquires designation information specifying a particular region within the first image and a second image including image data of the particular region by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

[0018] According to the first embodiment, personal privacy information can be protected by not including privacy information in the first bitstream. Furthermore, by transmitting a second image containing privacy information as a second bitstream and generating a third image based on the second image, a machine task or human vision using the privacy information can be executed on the image decoding device side.

[0019] In the first embodiment, the image decoding device according to the second aspect of this disclosure preferably includes an image in which a privacy region containing privacy information has been obscured.

[0020] According to the second embodiment, privacy information contained in a specific region within the first image can be appropriately protected by concealment processing.

[0021] In the third aspect of the present disclosure, the image decoding device preferably includes, in the second aspect, images to which the privacy region included in the specific region has not been subjected to the concealment process.

[0022] According to the third embodiment, by including images in the second and third images in which privacy regions within a specific area have not been obscured, the image decoding device can appropriately execute machine tasks or human vision using privacy information.

[0023] In the first embodiment, the image decoding device according to the fourth aspect of the present disclosure includes, in the first embodiment, an image in which a specific region in the first image has been processed to conceal a privacy region containing privacy information, and in the generation of the third image, in applications where an image with concealed privacy information is used, the third image is generated using the first image, and in applications where an image without concealed privacy information is used, the third image is generated using the first image and the second image.

[0024] According to the fourth embodiment, the image decoding device can switch between and execute a machine task or human vision using an image in which privacy information has been concealed and a machine task or human vision using an image in which privacy information has not been concealed.

[0025] In any one of the first to fourth embodiments of the present disclosure, the image decoding device may include information indicating the position and size of the specific region within the first image.

[0026] According to the fifth embodiment, the specified information allows for the appropriate identification of the position and size of a specific region within the first image.

[0027] In the sixth aspect of this disclosure, the image decoding device may, in the fifth aspect, further include information indicating the correspondence between the specific region in the first image and the second image.

[0028] According to the sixth aspect, the correspondence between a specific region in the first image and the second image can be appropriately identified by the specified information.

[0029] In the seventh aspect of the present disclosure, the image decoding device, in any one of the first to sixth aspects, may obtain the designated information by decoding the header area of ​​the second bitstream.

[0030] According to the seventh embodiment, since the specified information is stored in the header area of ​​the second bitstream, the specified information can be easily decoded from the second bitstream.

[0031] In the eighth aspect of the present disclosure, in any one of the first to seventh aspects, the image decoding device has information indicating the position and size of the specific region within the first image among the designated information stored in the header area of ​​the first bitstream, and the circuit may obtain this information by decoding the header area of ​​the first bitstream.

[0032] According to the eighth aspect, information inherently contained in the header area of ​​the first bitstream can be used as designation information.

[0033] In the ninth aspect of this disclosure, the image decoding device, in any one of the first to eighth aspects, may include, as image data, difference image data between the specific region in the first image and the specific region in the third image.

[0034] According to the ninth embodiment, the amount of code transmitted from the image encoding device to the image decoding device can be reduced compared to the case where image data of a specific region that has not been subjected to concealment processing is transmitted.

[0035] In the image decoding device according to the tenth aspect of this disclosure, in any one of the first to eighth aspects, the second image may include, as image data, the image data of the specific region within the third image.

[0036] According to the tenth embodiment, the circuit only needs to replace the image data of a specific region in the first image with the image data of the second image, thus reducing the processing load on the circuit.

[0037] In the eleventh aspect of this disclosure, the image decoding device may, in any one of the first to tenth aspects, have a specific region that includes a plurality of specific regions, and the picture of the second bitstream may include the image data of one of the plurality of specific regions.

[0038] According to the eleventh embodiment, since the picture of the second bitstream contains only image data of a specific region, it becomes easy to associate each specific region in the first image with each second image.

[0039] In the image decoding device according to the twelfth aspect of this disclosure, in any one of the first to tenth aspects, the specific region may include a plurality of specific regions, and the picture of the second bitstream may include the image data of the plurality of specific regions.

[0040] According to the twelfth embodiment, since the picture of the second bitstream contains image data of multiple specific regions, the number of pictures of the second bitstream that need to be transmitted from the image encoding device to the image decoding device can be reduced.

[0041] In the thirteenth aspect of the present disclosure, the image decoding device may, in any one of the first to twelfth aspects, have the specific region included within one or more encoding unit blocks constituting the picture of the second bitstream.

[0042] According to the 13th embodiment, since the specific region is contained within one or more coding unit blocks, the circuit can appropriately decode the specific region.

[0043] In the image decoding device according to the 14th aspect of this disclosure, in any one of the 1st to 12th aspects, the specific region may be included in a subpicture or tile that constitutes the picture of the second bitstream.

[0044] According to the 14th embodiment, since the specific region is contained within a subpicture or tile, the circuit can appropriately decode the specific region.

[0045] In the image decoding device according to the 15th aspect of this disclosure, in any one of the 1st to 14th aspects, the specific region may include a plurality of specific regions, and the second image may include the image data corresponding to the plurality of specific regions.

[0046] According to the 15th embodiment, even when multiple objects such as people move closer to or further away from each other in the captured image, multiple specific regions corresponding to multiple objects can be included in a single second image.

[0047] In the sixteenth aspect of this disclosure, the image decoding device is preferable in any one of the first to fifteenth aspects such that the size of the picture in the first bitstream and the size of the picture in the second bitstream are the same.

[0048] According to the 16th aspect, by having the same picture size in the first bitstream as the same picture size in the second bitstream, it becomes easier to associate a specific region in the first image with the second image.

[0049] In the image decoding device according to the 17th aspect of this disclosure, in any one of the 1st to 16th aspects, the picture of the second bitstream is preferably an intrapicture that performs in-screen prediction.

[0050] According to the 17th embodiment, since the picture in the second bitstream is an intrapicture, there is no need to consider the movement of the object over time, and therefore, there is no need to retain the previous frame or perform difference calculations with the previous frame.

[0051] In the image decoding device according to the 18th aspect of this disclosure, in any one of the 1st to 16th aspects, the picture of the second bitstream is an interpicture that performs inter-screen prediction using a reference picture obtained by decoding the second bitstream or the first bitstream.

[0052] According to the 18th aspect, the amount of encoding can be reduced because the picture of the second bitstream is an interpicture.

[0053] In the 19th aspect of the present disclosure, in any one of the 1st to 18th aspects, the first bitstream is transmitted from the image encoding device to the image decoding device using a multilayer base layer, and the second bitstream is transmitted from the image encoding device to the image decoding device using a multilayer enhancement layer.

[0054] According to the 19th aspect, the first bitstream and the second bitstream can be easily transmitted using a multilayer base layer and enhancement layer standardized by VVC, etc.

[0055] In the 20th aspect of this disclosure, the image decoding device is preferably such that, in any one of the 1st to 19th aspects, the first bitstream and the second bitstream are transmitted from the image encoding device to the image decoding device using different transmission paths.

[0056] According to the 20th aspect, by transmitting the first bitstream using a transmission path such as a public network and transmitting the second bitstream using a transmission path such as a private network, it is possible to achieve both a reduction in transmission costs and protection of private information.

[0057] An image encoding device according to a 21st aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, designates a specific region in an input image, generates designation information for the specific region, generates a first image obtained by processing the specific region from the input image, generates a second image including image data corresponding to the specific region, generates and outputs a first bitstream by encoding the first image, and generates and outputs a second bitstream by encoding the designation information and the second image.

[0058] According to the 21st aspect, by not including privacy information in the first bitstream, personal privacy information can be protected. Furthermore, by transmitting a second image containing privacy information as a second bitstream from the image encoding device to the image decoding device, and by the image decoding device generating an image containing privacy information based on the first and second images, a machine task or human vision using the privacy information can be executed on the image decoding device side.

[0059] In the image encoding apparatus according to the 22nd aspect of this disclosure, in the 21st aspect, the specific region in the first image may include an image in which a privacy region containing privacy information has been obscured.

[0060] According to the 22nd aspect, privacy information contained in a specific area within the first image can be appropriately protected by concealment processing.

[0061] In the 23rd aspect of this disclosure, the image encoding apparatus, in the 22nd aspect, may include an image in which the privacy region included in the specific region has not been subjected to the concealment process.

[0062] According to the 23rd embodiment, by including images in the second and third images that have not been obscured in the privacy region included in a specific area, a machine task or human vision using privacy information can be appropriately executed on the image decoding device side.

[0063] In any one of the 21st to 23rd embodiments of this disclosure, the designated information may include information indicating the position and size of the specific region within the first image.

[0064] According to the 24th aspect, the position and size of a specific region within the first image can be appropriately determined by the specified information.

[0065] In the 25th aspect of this disclosure, the image encoding device may, in the 24th aspect, further include information indicating the correspondence between the specific region in the first image and the second image.

[0066] According to the 25th aspect, the correspondence between a specific region in the first image and the second image can be appropriately identified by the specified information.

[0067] In the image encoding apparatus according to the 26th aspect of this disclosure, in any one of the 21st to 25th aspects, the circuit may store the designation information in the header area of ​​the second bitstream.

[0068] According to the 26th embodiment, since the specified information is stored in the header area of ​​the second bitstream, the image decoding device can easily decode the specified information from the second bitstream.

[0069] In the image encoding apparatus according to the 27th aspect of this disclosure, in any one of the 21st to 26th aspects, the circuit may store information indicating the position and size of the specific region within the first image, among the designated information, in the header area of ​​the first bitstream.

[0070] According to the 27th aspect, information inherently contained in the header area of ​​the first bitstream can be used as designation information.

[0071] In any one of the 21st to 27th embodiments of this disclosure, the image encoding device may include, as image data, difference image data between the specific region in the input image and the specific region in the first image.

[0072] According to the 28th embodiment, the amount of code transmitted from the image encoding device to the image decoding device can be reduced compared to the case where image data of a specific region that has not been subjected to concealment processing is transmitted.

[0073] In the image encoding apparatus according to the 29th aspect of this disclosure, in any one of the 21st to 27th aspects, the second image may include, as image data, image data of the specific region within the input image.

[0074] According to the 29th aspect, the image decoding device only needs to replace the image data of a specific region in the first image with the image data of the second image, thus reducing the processing load on the image decoding device.

[0075] In the 30th aspect of this disclosure, the image encoding device may, in any one of the 21st to 29th aspects, include a specific region comprising a plurality of specific regions, and the picture of the second bitstream may include the image data of one of the plurality of specific regions.

[0076] According to the 30th aspect, since the picture of the second bitstream contains only image data of a specific region, it becomes easy to associate each specific region in the first image with each second image.

[0077] In the 31st aspect of this disclosure, the image encoding device may, in any one of the 21st to 29th aspects, have the specific region include a plurality of specific regions, and the picture of the second bitstream may include the image data of the plurality of specific regions.

[0078] According to the 31st embodiment, since the picture of the second bitstream contains image data of multiple specific regions, the number of pictures of the second bitstream that need to be transmitted from the image encoding device to the image decoding device can be reduced.

[0079] In the 32nd aspect of this disclosure, the image encoding apparatus may, in any one of the 21st to 31st aspects, have the specific region included within one or more encoding unit blocks constituting the picture of the second bitstream.

[0080] According to the 32nd embodiment, since the specific region is contained within one or more coding unit blocks, the specific region can be appropriately decoded by the image decoding device.

[0081] In the 33rd aspect of this disclosure, the image encoding apparatus may, in any one of the 21st to 31st aspects, have the specific region included within a subpicture or tile constituting the picture of the second bitstream.

[0082] According to the 33rd embodiment, since the specific region is contained within a subpicture or tile, the image decoding device can appropriately decode the specific region.

[0083] In the 34th aspect of this disclosure, the image encoding device may, in any one of the 21st to 33rd aspects, include a plurality of specific regions, and the second image may include the image data corresponding to the plurality of specific regions.

[0084] According to the 34th aspect, even when multiple objects such as people move closer to or further away from each other in a captured image, multiple specific regions corresponding to multiple objects can be included in a single second image.

[0085] In the image encoding apparatus according to the 35th aspect of this disclosure, it is preferable that in any one of the 21st to 34th aspects, the size of the picture in the first bitstream and the size of the picture in the second bitstream are the same.

[0086] According to the 35th aspect, by having the same picture size in the first bitstream and the same picture size in the second bitstream, it becomes easier to associate a specific region in the first image with the second image.

[0087] In the 36th aspect of this disclosure, the image encoding apparatus may, in any one of the 21st to 35th aspects, be an intra-picture that performs in-screen prediction.

[0088] According to the 36th aspect, since the picture in the second bitstream is an intrapicture, there is no need to consider the movement of the object over time, and therefore, there is no need to retain the previous frame or perform difference calculations with the previous frame.

[0089] In the 37th aspect of this disclosure, the image encoding apparatus, in any one of the 21st to 35th aspects, is preferable in which the picture of the second bitstream is an interpicture that performs inter-screen prediction using a reference picture obtained by decoding the second bitstream or the first bitstream.

[0090] According to the 37th aspect, the amount of code can be reduced because the picture of the second bitstream is an interpicture.

[0091] In the 38th aspect of this disclosure, in any one of the 21st to 37th aspects, the circuit may transmit the first bitstream to the image decoding device using a multilayer base layer and transmit the second bitstream to the image decoding device using a multilayer enhancement layer.

[0092] According to the 38th aspect, the first bitstream and the second bitstream can be easily transmitted using a multilayer base layer and enhancement layer standardized by VVC, etc.

[0093] In the 39th aspect of this disclosure, the image coding device, in any one of the 21st to 38th aspects, may transmit the first bitstream and the second bitstream to the image decoding device using different transmission paths.

[0094] According to the 39th aspect, by transmitting the first bitstream using a transmission path such as a public network and transmitting the second bitstream using a transmission path such as a private network, it is possible to achieve both a reduction in transmission costs and protection of private information.

[0095] An image decoding method according to the fortieth aspect of this disclosure acquires a first image by decoding a first bitstream, acquires designation information specifying a particular region within the first image and a second image containing image data of the particular region by decoding a second bitstream, and generates a third image based on the first image, the designation information, and the second image.

[0096] According to the 40th aspect, by not including privacy information in the first bitstream, personal privacy information can be protected. Furthermore, by transmitting a second image containing privacy information as a second bitstream and generating a third image based on the second image, a machine task or human vision using privacy information can be executed on the image decoding device side.

[0097] An image encoding method according to the 41st aspect of this disclosure involves specifying a specific region in an input image, generating specification information for the specific region, generating a first image obtained by processing the specific region from the input image, generating a second image containing image data corresponding to the specific region, generating and outputting a first bitstream by encoding the first image, and generating and outputting a second bitstream by encoding the specification information and the second image.

[0098] According to the 41st aspect, by not including privacy information in the first bitstream, personal privacy information can be protected. Furthermore, by transmitting a second image containing privacy information as a second bitstream from the image encoding device to the image decoding device, and by the image decoding device generating an image containing privacy information based on the first and second images, a machine task or human vision using the privacy information can be executed on the image decoding device side.

[0099] (Embodiments of the present disclosure) Embodiments of this disclosure will be described in detail below with reference to the drawings. Elements denoted by the same reference numeral in different drawings are considered to be the same or corresponding elements.

[0100] The embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components. In addition, the contents of each embodiment can be combined.

[0101] Figure 1 is a simplified diagram showing the configuration of an image processing system according to an embodiment of this disclosure. The image processing system comprises an image encoding device 1, transmission lines NW1 and NW2, and an image decoding device 2.

[0102] The image encoding device 1 comprises a region selection unit 11, a first image generation unit 12, a second image generation unit 13, a first encoding unit 14, and a second encoding unit 15.

[0103] The region designation unit 11 designates a specific region in the input image D1 and generates designation information D2 for that specific region. The first image generation unit 12 generates a first image D3 by processing the specific region of the input image D1. The second image generation unit 13 generates a second image D4 that includes image data corresponding to the specific region. The first encoding unit 14 generates and outputs a first bitstream D5 by encoding the first image D3. The first bitstream D5 is transmitted to the image decoding device 2 via the transmission line NW1. The second encoding unit 15 generates and outputs a second bitstream D6 by encoding the designation information D2 and the second image D4. The second bitstream D6 is transmitted to the image decoding device 2 via a transmission line NW2 different from the transmission line NW1.

[0104] The transmission paths NW1 and NW2 are the Internet, WAN (Wide Area Network), LAN (Local Area Network), or any combination thereof. Furthermore, the transmission paths NW1 and NW2 are not necessarily limited to bidirectional communication networks, but may also be one-way communication networks that transmit broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. In addition, the transmission paths NW1 and NW2 may be recording media such as DVDs (Digital Versatile Discs) or BDs (Blu-Ray Discs) on which the first bitstream D5 or the second bitstream D6 is recorded. The transmission path NW1 is a public network, etc., and the transmission path NW2 is a private network, etc., where secure communication is ensured by access restrictions. Note that if different access restrictions can be set for the first bitstream D5 and the second bitstream D6, the transmission paths NW1 and NW2 may be physically the same communication network or recording media.

[0105] The image decoding device 2 comprises a first decoding unit 21, a second decoding unit 22, and an image generation unit 23.

[0106] The first decoding unit 21 receives the first bitstream D5 transmitted by the first encoding unit 14 via the transmission path NW1. The first decoding unit 21 decodes the received first bitstream D5 to obtain the first image D7, which corresponds to the first image D3.

[0107] The second decoding unit 22 receives the second bitstream D6 transmitted by the second encoding unit 15 via the transmission path NW2. By decoding the received second bitstream D6, the second decoding unit 22 obtains the specified information D8, which corresponds to the specified information D2, and the second image D9, which corresponds to the second image D4.

[0108] The image generation unit 23 generates a third image D10 corresponding to the input image D1 based on the first image D7, the specified information D8, and the second image D9. Specifically, for example, in use cases where images without privacy information are used (i.e., applications where images with concealed privacy information are used), the image generation unit 23 generates the third image D10 using the first image D7 as is. Use cases where images without privacy information are used include, for example, viewing by users without special access privileges, or use in machine tasks that do not require detailed information such as faces or vehicle license plates. Also, for example, in use cases where images containing privacy information are used (i.e., applications where images without concealed privacy information are used), the image generation unit 23 generates the third image D10 using the first image D7 and the second image D9 for image data within a specific area, and generates the third image D10 using the first image D7 for image data outside the specific area, based on the specified information D8. Use cases for images containing privacy information include, for example, viewing by users with special access privileges, or machine tasks that require detailed information such as faces or vehicle license plates. In this way, by switching the generation of the third image D10 according to the use of the image, the image decoding device 2 can switch between executing machine tasks or human vision using images that conceal privacy information and machine tasks or human vision using images that do not conceal privacy information. Machine tasks include object detection, object segmentation, object tracking, action recognition, or pose estimation.

[0109] Figure 12 is a simplified diagram showing an implementation example of the image encoding device 1. The image encoding device 1 is configured to include a processor 101 and a memory 102 connected to the processor 101. However, the memory 102 may be included within the processor 101. The processor 101 is a circuit that performs information processing. The processor 101 includes a CPU or GPU, etc. The memory 102 includes a semiconductor memory such as ROM or RAM, a magnetic disk, or an optical disk, etc. The memory 102 stores information necessary for the processor 101 to perform encoding processing, etc. For example, the memory 102 stores the input image D1, specified information D2, the first image D3, and the second image D4. The memory 102 also stores a program. By executing the program read from the memory 102, the processor 101 functions as a region specification unit 11, a first image generation unit 12, a second image generation unit 13, a first encoding unit 14, and a second encoding unit 15.

[0110] Figure 13 is a simplified diagram showing an implementation example of the image decoding device 2. The image decoding device 2 comprises a processor 201 and a memory 202 connected to the processor 201. However, the memory 202 may be included within the processor 201. The processor 201 is a circuit that performs information processing. The processor 201 includes a CPU or GPU, etc. The memory 202 includes a semiconductor memory such as ROM or RAM, a magnetic disk, or an optical disk, etc. The memory 202 stores information necessary for the processor 201 to perform decoding processing, etc. For example, the memory 202 stores the first bitstream D5, the second bitstream D6, the first image D7, the specified information D8, and the second image D9. The memory 202 also stores a program. By executing the program read from the memory 202, the processor 201 functions as a first decoding unit 21, a second decoding unit 22, and an image generation unit 23.

[0111] Figure 2 is a flowchart showing the processing flow performed by the image encoding device 1.

[0112] First, in step SP11, the region designation unit 11 designates a specific region for the input image D1 and generates designation information D2 for that specific region.

[0113] Next, in step SP12, the first image generation unit 12 generates a first image D3 by processing a specific region of the input image D1.

[0114] Next, in step SP13, the second image generation unit 13 generates a second image D4 which includes image data corresponding to a specific region.

[0115] Next, in step SP14, the first encoding unit 14 generates and outputs the first bitstream D5 by encoding the first image D3.

[0116] Next, in step SP15, the second encoding unit 15 generates and outputs a second bitstream D6 by encoding the specified information D2 and the second image D4.

[0117] Figure 3 shows an example of input image D1. Input image D1 includes an image of a person's face 31, a full-body image including the person's face 32, and a background image 33. The person's face is an example of personal privacy information. In addition to the person's face, privacy information may include vehicle license plates or house nameplates, etc.

[0118] Figure 4 shows bounding boxes 41 and 42 as an example of a specific region. The region designation unit 11 sets bounding box 41, which surrounds the privacy region including image 31 with a rectangle, as one of the specific regions. The region designation unit 11 also sets bounding box 42, which surrounds the privacy region including image 32 with a rectangle, as one of the specific regions. Since image 33 does not contain privacy information, no bounding box is set for it.

[0119] The specified information D2 includes information indicating the position and size of bounding boxes 41 and 42 within the first image D3.

[0120] Figures 5A to 5F show examples of specified information D2. As shown in Figure 5A, specified information D2 includes the position coordinates (X1, Y1) of a specific vertex of the bounding box, the vertical size H of the bounding box, and the horizontal size W of the bounding box. As shown in Figure 5B, specified information D2 includes the position coordinates (X1, Y1) of a specific vertex of the bounding box and the position coordinates (X4, Y4) of the vertex located diagonally opposite to that vertex. As shown in Figure 5C, specified information D2 includes the position coordinates (X0, Y0) of the center of the bounding box, the vertical size H of the bounding box, and the horizontal size W of the bounding box. As shown in Figure 5D, specified information D2 includes the position coordinates (X1, Y1)(X2, Y2)(X3, Y3)(X4, Y4) of the four vertices of the bounding box. In this case, as shown in Figure 5E, the position and size can also be specified for a tilted bounding box. As shown in Figure 5F, the specified information D2 includes mask information in which pixel positions within a specific region are represented by "1" and pixel positions outside the specific region are represented by "0".

[0121] Furthermore, the specified information D2 includes information indicating the correspondence between bounding boxes 41 and 42 in the first image D3 and the second image D4. If the second image D4 contains multiple bounding boxes 41 and 42, the specified information D2 includes information indicating the correspondence between bounding boxes 41 and 42 in the first image D3 and the bounding boxes 41 and 42 in the second image D4.

[0122] Figure 6 shows an example of the first image D3. The bounding boxes 41 and 42 in the first image D3 contain an image in which privacy information is hidden. The first image generation unit 12 performs a hiding process on the privacy area in the input image D1 based on the specified information D2 input from the area specification unit 11. The hiding process includes blurring, mosaic processing, or silhouette processing. Figure 6 shows an example in which silhouette processing has been applied.

[0123] Figures 7A to 7H show an example of the second image D4.

[0124] As shown in Figure 7A, the second image D4A has the same picture size as the input image D1. The rectangular area demarcated by the dashed line in the second image D4A corresponds to an encoding unit block. An encoding unit block is a CU (coding unit) or CTU (coding tree unit), etc., and has a size of, for example, 8 × 8 pixels. The image data of the area outside the bounding boxes 41 and 42 in the second image D4A may be the same as that of the input image D1, or it may be padded with meaningless data (e.g., "0"). The position and size of the bounding boxes 41 and 42 in the second image D4A are the same as the position and size of the bounding boxes 41 and 42 in the input image D1. Since the picture size of the input image D1 and the picture size of the second image D4A are the same, the first encoding unit 14 can transmit the first bitstream D5 to the image decoding device 2 using a multi-layer base layer standardized by H.266 / VVC (Versatile Video Coding), etc. Furthermore, the second encoding unit 15 can transmit the second bitstream D6 to the image decoding device 2 using a multi-layer enhancement layer. Note that when using a multi-layer method, the enhancement layer can be encoded as difference information from the base layer. Therefore, without setting a bounding box, the difference information of the entire picture between the first image D3, where privacy information is concealed, and the input image D1, where privacy information is not concealed, can be encoded as the second bitstream D6. The image decoding device 2 can perform machine tasks or human vision using images with concealed privacy information by allowing any user to access the base layer. Alternatively, by allowing only users with special privileges to access the enhancement layer, machine tasks or human vision using images with unconcealed privacy information can be performed.

[0125] The second image D4 includes an image in which the privacy region contained within the bounding boxes 41 and 42 has not been obscured. The second image includes image data corresponding to a specific region as an image in which the obscuring process has not been applied. The image data corresponding to the specific region may be the difference image data between the specific region in input image D1 and the specific region in the first image D3, or it may be the image data of the specific region in input image D1 itself. If the specific region in the second image D4 is the difference image data, the image before the obscuring process is applied to the privacy region is obtained by adding the values ​​obtained by multiplying the first image D3 and the second image D4 with respect to the specific region. If the specific region in the second image D4 is the image data of input image D1, the image before the obscuring process is applied to the privacy region is obtained by adding the value obtained by multiplying the first image D3 by the weight value α and the value obtained by multiplying the second image D4 by the weight value β with respect to the specific region. The weight values ​​α and β are both between 0 and 1, and α + β = 1. Furthermore, if α=0 and β=1, the specific region of the second image D4 becomes the image data of the input image D1 itself. In this case, the image is obtained before any concealment processing is applied to the privacy region, without using any image data of the specific region of the first image D3. Alternatively, β can be defined as 1-α, using only one weight value α.

[0126] Furthermore, the second image D4 may be an intra-picture encoded by in-screen prediction. In other words, the second encoding unit 15 may encode the second image D4 using in-screen prediction. Since the picture in the second bitstream D6 is an intra-picture, there is no need to consider the movement of objects over time, and therefore, there is no need to retain the previous frame or perform difference calculations with the previous frame.

[0127] On the other hand, the second image D4 may be an interpicture encoded by inter-screen prediction. In other words, the second encoding unit 15 may encode the second image D4 using inter-screen prediction. The reference picture used for inter-screen prediction may be the picture of the second image D4 obtained by decoding (local decoding) the second bitstream D6, or the picture of the first image D3 obtained by decoding (local decoding) the first bitstream D5. The amount of code can be reduced by having the picture of the second bitstream D6 be an interpicture.

[0128] Although the second image D4A shown in Figure 7A includes multiple bounding boxes 41 and 42, bounding boxes 41 and 42 may be included in different second images D4B and D4C.

[0129] As shown in Figure 7B, the second image D4B contains only the bounding box 41. The second image D4B has the same picture size as the input image D1. The image data of the area outside the bounding box 41 in the second image D4B may be the same as that of the input image D1, or it may be padded with meaningless data. The position and size of the bounding box 41 in the second image D4B are the same as the position and size of the bounding box 41 in the input image D1.

[0130] As shown in Figure 7C, the second image D4C contains only the bounding box 42. The second image D4C has the same picture size as the input image D1. The image data of the area outside the bounding box 42 in the second image D4C may be the same as that of the input image D1, or it may be padded with meaningless data. The position and size of the bounding box 42 in the second image D4C are the same as the position and size of the bounding box 42 in the input image D1.

[0131] As shown in Figure 7D, the second image D4D contains only the bounding box 41. Considering that the size of the bounding box 41 changes from frame to frame, the second image D4D is composed of the minimum number of encoding unit blocks (four in this example) that contain the bounding box 41, with the bounding box 41 packed into the upper left corner. The image data in the area of ​​the second image D4D outside the bounding box 41 is padded with meaningless data.

[0132] As shown in Figure 7E, the second image D4E contains only the bounding box 42. Considering that the size of the bounding box 42 changes from frame to frame, the second image D4E is composed of the minimum number of encoding unit blocks (15 in this example) that contain the bounding box 42, with the bounding box 42 packed into the upper left corner. The image data in the area of ​​the second image D4E outside the bounding box 42 is padded with meaningless data.

[0133] As shown in Figure 7F, the second image D4F contains bounding boxes 41 and 42. The 15 coding unit blocks that enclose bounding box 42 are arranged to the right of the 4 coding unit blocks that enclose bounding box 41. In addition, to define the second image D4F as a rectangular region, 6 coding unit blocks are arranged below the 4 coding unit blocks that enclose bounding box 41. The image data of the region outside bounding boxes 41 and 42 in the second image D4F is padded with meaningless data.

[0134] As shown in Figure 7G, the second image D4G contains bounding boxes 41 and 42. The picture in the second image D4G is divided into multiple subpictures, which are standardized by VVC, etc. Bounding box 41 is contained in the subpicture in the upper left corner, and bounding box 42 is contained in the subpicture adjacent to its right. Image data in the area of ​​the second image D4G outside of bounding boxes 41 and 42 is padded with meaningless data.

[0135] As shown in Figure 7H, the second image D4H contains bounding boxes 41 and 42. The picture in the second image D4H is divided into multiple tiles that are standardized by VVC, etc. Bounding box 41 is contained in the tile in the upper left corner, and bounding box 42 is contained in the adjacent tiles to its right and lower right. Image data in the area of ​​the second image D4H outside of bounding boxes 41 and 42 is padded with meaningless data.

[0136] Figure 8 shows a first example of the second image D4I when parts of multiple bounding boxes 41 and 42 overlap. As shown in Figure 8, the second image D4I contains bounding boxes 41 and 42. Parts of the bounding boxes 41 and 42 overlap because the two people have moved closer together. The second image D4I is composed of the minimum number of encoded unit blocks (24 in this example) that contain the bounding boxes 41 and 42, with the bounding box 41 packed into the upper left corner. Image data in the area of ​​the second image D4I outside of the bounding boxes 41 and 42 is padded with meaningless data. When the two people have moved further apart and the bounding boxes 41 and 42 no longer overlap, the second image D4I is divided into the second image D4D shown in Figure 7D and the second image D4E shown in Figure 7E. Even if parts of multiple bounding boxes 41 and 42 overlap, the image may be divided into a second image D4D shown in Figure 7D and a second image D4E shown in Figure 7E, including the overlapping portion, according to the specifications of each bounding box 41 and 42. Furthermore, the second image D4I may be defined by subpictures or tiles instead of multiple encoding unit blocks.

[0137] Figures 9A and 9B show a second example of the second image D4J and D4K when parts of multiple bounding boxes 41 and 42 overlap. As shown in Figure 9A, the second image D4J contains bounding boxes 41 and 42. Parts of the bounding boxes 41 and 42 overlap due to the two people getting closer. The second image D4I is composed of a subpicture that contains bounding boxes 41 and 42, with bounding box 41 packed into the upper left corner. The number of encoded unit blocks included in the subpicture is set to be greater than the minimum number of encoded unit blocks that contain bounding boxes 41 and 42 (24 in this example) (35 in this example). The image data of the area outside of bounding boxes 41 and 42 in the second image D4J is padded with meaningless data.

[0138] As the two figures move further apart, the bounding boxes 41 and 42 no longer overlap, and the overlap of the bounding boxes 41 and 42 is resolved in the second image D4K, as shown in Figure 9B. The second image D4K contains the bounding boxes 41 and 42 that do not overlap each other. As the two figures move further apart, the bounding box 42 moves out of the range of the second image D4K, and the second image D4K is divided into different subpictures, as shown in the second image D4G in Figure 7G. Note that the second images D4J and D4K may be defined by tiles or multiple encoding unit blocks instead of subpictures.

[0139] Figure 10A shows a first example of the data structure of bitstream D6. Bitstream D6 has a header area H where management information is stored and a payload area P where image data is stored. The second encoding unit 15 stores the encoded data of the second image D4 in the payload area P. The second encoding unit 15 also stores the encoded data 70 of the specified information D2 in a predetermined location in the payload area P.

[0140] Figure 10B shows a second example of the data structure of bitstream D6. Similar to Figure 10A, bitstream D6 has a header area H and a payload area P. The second encoding unit 15 stores the encoded data of the second image D4 in the payload area P. The second encoding unit 15 also stores the encoded data 70 of the specified information D2 in a predetermined location in the header area H. The predetermined location is, for example, the SEI (Supplemental Enhancement Information) area for storing additional information. The predetermined location may be VPS, SPS, PPS, PH, SH, APS, or a tile header, etc. By storing the specified information D2 in the header area H of bitstream D6, the second decoding unit 22 can easily obtain the specified information D8 by decoding. Furthermore, by storing the specified information D2 in the SEI area, the specified information D2 can be easily handled as additional information.

[0141] Furthermore, the first encoding unit 14 may store information indicating the position and size of bounding boxes 41 and 42 within the first image D3, from the specified information D2, in a predetermined location in the header area H of the first bitstream D5. The predetermined location may be, for example, the ARSEI (annotated region SEI) area for storing bounding box information. This makes it possible to use the bounding box information, which is inherently included in the header area H of the first bitstream D5, as part of the specified information D2.

[0142] Figure 11 is a flowchart showing the processing flow performed by the image decoding device 2.

[0143] First, in step SP21, the first decoding unit 21 decodes the first bitstream D5 received from the image encoding device 1 to obtain the first image D7, which corresponds to the first image D3.

[0144] Next, in step SP22, the second decoding unit 22 decodes the second bitstream D6 received from the image encoding device 1 to obtain the designated information D8 corresponding to the designated information D2 and the second image D9 corresponding to the second image D4.

[0145] Next, in step SP23, the image generation unit 23 generates a third image D10 corresponding to the input image D1 based on the first image D7, the specified information D8, and the second image D9.

[0146] Similar to the first image D3 in the image encoding device 1, the regions within bounding boxes 41 and 42 in the first image D7 in the image decoding device 2 include an image in which privacy regions containing privacy information have been obscured. The obscuring process includes blurring, mosaic processing, or silhouette processing.

[0147] Similar to the second image D4 and input image D1 in the image encoding device 1, the second image D9 and third image D10 in the image decoding device 2 contain images in which privacy regions included in bounding boxes 41 and 42 have not been obscured. The second image D9 may contain difference image data between the bounding boxes 41 and 42 in the first image D7 and the bounding boxes 41 and 42 in the third image D10, or it may contain image data of the bounding boxes 41 and 42 in the third image D10.

[0148] The second decoding unit 22 obtains the specified information D8 by decoding the payload area P or header area H of the second bitstream D6. Of the specified information D8, the information indicating the position and size of the bounding boxes 41 and 42 within the first image D7 may be stored in the header area H of the first bitstream D5. In this case, the first decoding unit 21 obtains this information by decoding the header area H of the first bitstream D5.

[0149] Similar to the designation information D2 in the image encoding device 1, the designation information D8 in the image decoding device 2 includes information indicating the position and size of bounding boxes 41 and 42 within the first image D7. The designation information D8 may further include information indicating the correspondence between the bounding boxes 41 and 42 within the first image D7 and the second image D9.

[0150] Similar to the second image D4 in the image encoding device 1, the second image D9 in the image decoding device 2 may contain image data for one of the bounding boxes 41, 42, or it may contain image data for multiple bounding boxes 41, 42. Furthermore, in the second image D9, the bounding boxes 41, 42 may be contained within one or more encoding unit blocks that constitute the picture of the second bitstream D6, or they may be contained within subpictures or tiles that constitute the picture of the second bitstream D6. In addition, the second image D9 may contain image data corresponding to multiple bounding boxes 41, 42.

[0151] Furthermore, the size of the picture in the first bitstream D5 and the size of the picture in the second bitstream D6 may be the same. The first bitstream D5 may be transmitted from the image encoding device 1 to the image decoding device 2 using a multi-layer base layer, and the second bitstream D6 may be transmitted from the image encoding device 1 to the image decoding device 2 using a multi-layer enhancement layer. The first bitstream D5 and the second bitstream D6 may be transmitted from the image encoding device 1 to the image decoding device 2 using different transmission paths NW1 and NW2. The picture in the second bitstream D6 may be an intra-picture for in-screen prediction, or an inter-picture for inter-screen prediction. As the reference picture used for inter-screen prediction, the picture of the second image D9 obtained by decoding the second bitstream D6 may be used, or the picture of the first image D7 obtained by decoding the first bitstream D5 may be used.

[0152] (Effects, etc.) According to the image encoding device 1 and image decoding device 2 of this embodiment, personal privacy information can be protected by not including privacy information in the first bitstream D5. Furthermore, by transmitting the second image D4 containing privacy information as the second bitstream D6, and having the image decoding device 2 generate the third image D10 based on the second image D9, a machine task or human vision using the privacy information can be executed on the image decoding device 2 side. [Industrial applicability]

[0153] This disclosure is particularly useful for application to an image processing system comprising an image encoding device for transmitting an image and an image decoding device for receiving an image.

Claims

1. Circuits and, A memory connected to the aforementioned circuit, Equipped with, In operation, the aforementioned circuit Retrieve the image and specified information from the bitstream. The aforementioned designated information indicates that processing has been performed on multiple specific regions in the image that include privacy information. Each of the aforementioned multiple specific regions has a rectangular shape, The circuit generates a packing image in which the plurality of specific regions are rearranged during operation. Image decoding device.

2. The aforementioned process includes at least one of blurring, mosaic processing, or silhouette processing. The image decoding device according to claim 1.

3. The aforementioned privacy information includes at least one of the following: a person's face, a vehicle's license plate number, or information indicating an address. The image decoding device according to claim 1.

4. The circuit generates a generated image based on the image and the specified information. The image decoding device according to claim 1.

5. The specified information includes information indicating the position and size of the plurality of specific regions within the image. The image decoding device according to claim 1.

6. The circuit obtains the specified information from the header area of ​​the bitstream. The image decoding device according to claim 1.

7. The header area includes SEI, The image decoding device according to claim 6.

8. The plurality of specific regions in the packing image are included within one or more encoding unit blocks that constitute the picture of the bitstream. The image decoding device according to claim 1.

9. The plurality of specific regions in the packing image are included within subpictures or tiles that constitute the picture of the bitstream. The image decoding device according to claim 1.

10. The specified information includes information indicating the correspondence between multiple specific regions in the image and multiple specific regions in the packing image. The image decoding device according to claim 1.

11. The aforementioned packing image shows that areas other than the aforementioned multiple specific regions are padded. The image decoding device according to claim 1.

12. Some of the aforementioned multiple specific regions have overlapping regions. The image decoding device according to claim 1.

13. Circuits and, A memory connected to the aforementioned circuit, Equipped with, In operation, the aforementioned circuit The first image is obtained from the first bitstream. From the second bitstream, specify information that designates a plurality of specific regions within the first image, and a second image containing image data of the plurality of specific regions are obtained. Based on the first image, the specified information, and the second image, a third image is generated. Each of the aforementioned multiple specific regions has a rectangular shape, The circuit generates a packing image in which the plurality of specific regions are rearranged during operation. Image decoding device.

14. The first image is transmitted from the image encoding device to the image decoding device using a multi-layer base layer. The second image is transmitted from the image encoding device to the image decoding device using a multi-layer enhancement layer. The image decoding device according to claim 13.

15. Circuits and, A memory connected to the aforementioned circuit, Equipped with, In operation, the aforementioned circuit For the input image, identify multiple specific regions containing privacy information. An image is generated in which processing has been applied to the aforementioned multiple specific regions. The system generates designation information indicating that processing has been performed on the multiple specific regions in the image. The image and the specified information are encoded into a bitstream. Each of the aforementioned multiple specific regions has a rectangular shape, The circuit generates a packing image in which the plurality of specific regions are rearranged during operation. Image encoding device.

16. The aforementioned process includes at least one of blurring, mosaic processing, or silhouette processing. The image encoding apparatus according to claim 15.

17. The aforementioned privacy information includes at least one of the following: a person's face, a vehicle's license plate number, or information indicating an address. The image encoding apparatus according to claim 15.

18. The specified information includes information indicating the position and size of the specific region within the image. The image encoding apparatus according to claim 15.

19. The circuit writes the specified information to the header area of ​​the bitstream. The image encoding apparatus according to claim 15.

20. The header area includes SEI, The image encoding apparatus according to claim 19.

21. The plurality of specific regions in the packing image are included within one or more encoding unit blocks that constitute the picture of the bitstream. The image encoding apparatus according to claim 15.

22. The plurality of specific regions in the packing image are included within subpictures or tiles that constitute the picture of the bitstream. The image encoding apparatus according to claim 15.

23. The specified information includes information indicating the correspondence between multiple specific regions in the image and multiple specific regions in the packing image. The image encoding apparatus according to claim 15.

24. The aforementioned packing image shows that areas other than the aforementioned multiple specific regions are padded. The image encoding apparatus according to claim 15.

25. Some of the aforementioned multiple specific regions have overlapping regions. The image encoding apparatus according to claim 15.

26. Circuits and, A memory connected to the aforementioned circuit, Equipped with, In operation, the aforementioned circuit The system identifies multiple specific regions in the input image and generates information specifying these multiple specific regions. A first image is generated by processing the plurality of specific regions from the input image. A second image is generated that includes image data corresponding to the aforementioned multiple specific regions. A first bitstream is generated by encoding the first image. A second bitstream is generated by encoding the specified information and the second image. Each of the aforementioned multiple specific regions has a rectangular shape, The circuit generates a packing image in which the plurality of specific regions are rearranged during operation. Image encoding device.

27. The first image is transmitted from the image encoding device to the image decoding device using a multi-layer base layer. The second image is transmitted from the image encoding device to the image decoding device using a multi-layer enhancement layer. The image encoding apparatus according to claim 26.

28. Retrieve the image and specified information from the bitstream. The aforementioned designated information indicates that processing has been performed on multiple specific regions in the image that include privacy information. Each of the aforementioned multiple specific regions has a rectangular shape, A packing image is generated in which the aforementioned multiple specific regions are rearranged. Image decoding method.

29. For the input image, identify multiple specific regions containing privacy information. An image is generated in which processing has been applied to the aforementioned multiple specific regions. The system generates designation information indicating that processing has been performed on the multiple specific regions in the image. The aforementioned image and the specified information are encoded, A packing image is generated in which the aforementioned multiple specific regions are rearranged. Image encoding method.

30. The first image is obtained from the first bitstream. From the second bitstream, specify information that designates a plurality of specific regions within the first image, and a second image containing image data of the plurality of specific regions are obtained. Based on the first image, the specified information, and the second image, a third image is generated. Each of the aforementioned multiple specific regions has a rectangular shape, A packing image is generated in which the aforementioned multiple specific regions are rearranged. Image decoding method.

31. The system identifies multiple specific regions in the input image and generates information specifying these multiple specific regions. A first image is generated by processing the plurality of specific regions from the input image. A second image is generated that includes image data corresponding to the aforementioned multiple specific regions. A first bitstream is generated by encoding the first image. A second bitstream is generated by encoding the specified information and the second image. Each of the aforementioned multiple specific regions has a rectangular shape, A packing image is generated in which the aforementioned multiple specific regions are rearranged. Image encoding method.

Images