Image encoding device, image decoding device, image encoding method, and image decoding method

Abstract

An image decoding device is provided with: a circuit that, during operation, acquires a first image and a plurality of first filter sets by decoding a bit stream, selects one first filter set from among the plurality of first filter sets on the basis of use information indicating the use of an image, and stores the selected first filter set on the basis of use information indicating the use of the image; a second image is generated and output by applying the selected first filter set to the first image.

Description

Technical Field

[0001] The present disclosure relates to an image encoding device, an image decoding device, an image encoding method, and an image decoding method. Background Art

[0002] Patent Document 1 discloses a video encoding and decoding method using an adaptive coupled pre-filter and a post-filter.

[0003] Patent Document 2 discloses a method for encoding image data for loading into an artificial intelligence (AI) integrated circuit.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent No. 9883207

[0007] Patent Document 2: U.S. Patent No. 10452955 Summary of the invention

[0008] An object of the present disclosure is to change a filter applied to a decoded image according to the image usage.

[0009] An image decoding device involved in one embodiment of the present invention comprises: a circuit; and a memory connected to the circuit, wherein the circuit, when in operation, obtains a first image and multiple first filter sets by decoding a bit stream, selects a first filter set from the multiple first filter sets based on usage information indicating the purpose of the image, and generates and outputs a second image by applying the selected first filter set to the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram that simply shows the configuration of an image processing system according to an embodiment of the present disclosure.

[0011] Figure 2 This is a diagram showing a simplified structure of a task processing unit.

[0012] Figure 3A This is a diagram showing an example of a partial configuration of an image processing system.

[0013] Figure 3B This is a diagram showing an example of a partial configuration of an image processing system.

[0014] Figure 4A It is a diagram showing an example of the filter conversion process performed by the conversion unit.

[0015] Figure 4BIt is a diagram showing an example of the filter conversion process performed by the conversion unit.

[0016] Figure 5 The figure shows object detection and object segmentation as an example of machine tasks.

[0017] Figure 6 FIG. 1 is a diagram showing target tracking, behavior recognition, and posture estimation as examples of machine tasks.

[0018] Fig. 7A FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0019] Figure 7B FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0020] Figure 7C FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0021] Fig.7D FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0022] Fig. 7E FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0023] Figure 7F FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0024] Figure 7G FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0025] Figure 7H FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0026] Fig.7I FIG. 1 is a diagram showing an example of a plurality of first filters of different types.

[0027] Figure 8 FIG. 4 is a diagram showing an example of a lookup table that defines selection of a first filter set based on image usage.

[0028] Fig. 9A This is a diagram showing an example of a sharpening filter with a strong filtering strength.

[0029] Fig. 9B This is a diagram showing an example of a sharpening filter with a weak filtering strength.

[0030] Fig. 10A This is a diagram showing an example of the second filter.

[0031] Fig. 10B This is a diagram showing an example of the second filter.

[0032] Fig. 10C This is a diagram showing an example of the second filter.

[0033] Fig.11A It is a diagram showing an example of application areas of the first filter and the second filter.

[0034] Fig. 11B It is a diagram showing an example of application areas of the first filter and the second filter.

[0035] Fig. 11C It is a diagram showing an example of application areas of the first filter and the second filter.

[0036] Fig. 12A This is a diagram showing a first example of a data structure of a bit stream.

[0037] Fig. 12B This is a diagram showing a second example of the data structure of a bit stream.

[0038] Fig.13 This is a diagram showing an example of the SEI message syntax related to the first filter set information.

[0039] Fig.14 This is a diagram showing an example of the SEI message syntax related to the second filter information.

[0040] Fig.15 It is a flowchart showing the flow of processing performed by the image encoding device.

[0041] Fig.16 This is a flowchart showing the flow of processing performed by the image decoding device.

[0042] Fig.17 This is a diagram showing a simplified implementation example of an image encoding device.

[0043] Fig.18 This is a diagram showing a simplified implementation example of an image decoding device. DETAILED DESCRIPTION

[0044] (Insights that form the basis of this disclosure)

[0045] The goal of conventional encoding methods is to provide optimal images for human vision under bit rate constraints.

[0046] The abundance of sensors and the development of machine learning or neural network based applications have enabled many intelligent platforms that process large amounts of data, including connected cars, video surveillance or smart cities, etc. As large amounts of data are always generated, the previous approach of including humans in the pipeline becomes inefficient and impractical in terms of latency and scale.

[0047] Furthermore, in transmission and archiving systems, there are concerns about the need for more compact data representation and low-latency solutions, so VCM (Video Coding for Machines) was introduced.

[0048] In some cases, machines can communicate with each other and perform tasks without human involvement, or humans need to perform additional processing on a specific decompressed stream. For example, a human "supervisor" searches for specific people or scenes in a surveillance camera image.

[0049] In another case, there are cases where both humans and machines use the corresponding bit streams. In the case of connected cars, for humans, the features can be used in image correction functions, and for machines, the features can be used in object detection and segmentation.

[0050] The general system architecture includes a pair of an image encoding device and an image decoding device. The input of the system is a dynamic image, a still image or a feature quantity. Examples of machine tasks include object detection, object segmentation, object tracking, behavior recognition, posture estimation, or any combination thereof. Human vision may be one of the cases that can be used together with machine tasks.

[0051] According to the conventional technology, there is a problem that it is impossible to dynamically change the filter applied to the decoded image in the image decoding device according to the image usage.

[0052] In order to solve such a problem, the inventor of the present invention obtained the following insights and finally came up with the present disclosure: a decoded image and multiple filter sets are obtained by decoding a bit stream, and a filter set is selected from multiple filter sets based on usage information indicating the image usage on the image decoding device side, thereby being able to dynamically change the filter set applied to the decoded image according to the image usage.

[0053] Next, each aspect of the present disclosure will be described.

[0054] The first method of the present disclosure involves an image decoding device comprising: a circuit; and a memory connected to the circuit, wherein the circuit, when in operation, obtains a first image and multiple first filter sets by decoding a bit stream, selects a first filter set from the multiple first filter sets based on usage information indicating the purpose of the image, and generates and outputs a second image by applying the selected first filter set to the first image.

[0055] According to the first aspect, the first filter set applied to the first image can be dynamically changed according to the image usage.

[0056] Regarding the image decoding device involved in the second mode of the present disclosure, it may be that in the first mode, the circuit obtains multiple second filters and multiple parameter values ​​by decoding the bit stream, selects a first filter set from the multiple first filter sets based on the usage information, selects a second filter from the multiple second filters, and selects a parameter value from the multiple parameter values, selects a first filter from the first filter set based on the feature value obtained by applying the selected second filter to the first image and the selected parameter value, and generates the second image by applying the selected first filter to the first image.

[0057] According to the second aspect, the first filter applied to the first image can be dynamically changed based on the feature value of the first image obtained by applying the second filter.

[0058] Regarding the image decoding device involved in the third embodiment of the present disclosure, it may be that in the second embodiment, the number of pixels of the first image area in the first image to which the first filter is applied is equal to the number of pixels of the second image area in the first image to which the second filter is applied, and the range of the second image area is larger than the range of the first image area.

[0059] According to the third aspect, it is possible to reduce the influence of local noise while suppressing an increase in the processing load associated with application of the second filter.

[0060] Regarding the image decoding device according to a fourth aspect of the present disclosure, in any one of the first to third aspects, one of the plurality of first filter sets may be a bypass filter that causes the first image to be output as the second image.

[0061] According to the fourth aspect, it is possible to avoid executing unnecessary filtering processing by selecting the bypass filter.

[0062] Regarding the image decoding device involved in the fifth embodiment of the present disclosure, it may be that in any one of the first to fourth embodiments, the circuit obtains a post-filter set corresponding to the pre-filter set applied to the input image in the image encoding device by decoding the bit stream received from the image encoding device, as one of the multiple first filter sets.

[0063] According to the fifth aspect, since the conversion process from the pre-filter set to the post-filter set is executed on the image encoding device side, it is possible to reduce the processing load on the image decoding device.

[0064] Regarding the image decoding device involved in the 6th mode of the present disclosure, it may be that in any one of the 1st to 4th modes, the circuit obtains a prefilter set applied to the input image in the image encoding device by decoding the bit stream received from the image encoding device, and obtains a postfilter set corresponding to the prefilter set by transforming the prefilter set as one of the multiple first filter sets.

[0065] According to the sixth aspect, since the conversion process from the pre-filter set to the post-filter set is performed on the image decoding device side, it is possible to reduce the processing load of the image encoding device.

[0066] The image decoding device according to the seventh aspect of the present disclosure may be configured such that, in any one of the first to sixth aspects, the image usage includes at least one machine task and human vision.

[0067] According to the seventh aspect, it is possible to select not only the first filter set suitable for machine tasks but also the first filter set suitable for human vision.

[0068] Regarding the image decoding device according to the eighth aspect of the present disclosure, in any one of the first to seventh aspects, the circuit may acquire the plurality of first filter sets by decoding a header of the bit stream.

[0069] According to the eighth aspect, by storing the first filter set in the header of the bit stream, the circuit can easily acquire the first filter.

[0070] Regarding the image decoding device involved in the 9th aspect of the present disclosure, it may be that, in the 8th aspect, the header has a SEI (Supplemental Enhancement Information) area, and the circuit obtains the plurality of first filter sets by decoding the SEI area.

[0071] According to the ninth aspect, by storing the first filter set in the SEI area, the first filter set can be easily handled as additional information.

[0072] The image encoding device involved in the 10th embodiment of the present disclosure comprises: a circuit; and a memory connected to the circuit, wherein the circuit, when in operation, generates a first image by applying a prefilter set corresponding to the image purpose to an input image, and generates a bit stream by encoding the first image and the prefilter set or a postfilter set corresponding to the prefilter set.

[0073] According to the tenth embodiment, the bit stream includes the first image and a plurality of pre-filter sets or a plurality of post-filter sets, so that the image decoding device can perform optimal post-filter processing corresponding to the pre-filter processing applied to the first image.

[0074] Regarding the image encoding device involved in the 11th aspect of the present disclosure, in the 10th aspect, the image usage may be an image usage in an image decoding device, and the circuit may send the bit stream to the image decoding device.

[0075] According to the eleventh aspect, a prefilter set according to the purpose of the image in the image decoding device can be applied to an input image.

[0076] The image decoding method involved in the 12th mode of the present disclosure obtains a first image and multiple first filter sets by decoding a bit stream, selects a first filter set from the multiple first filter sets based on usage information indicating the purpose of the image, and generates and outputs a second image by applying the selected first filter set to the first image.

[0077] According to the twelfth aspect, the first filter set applied to the first image can be dynamically changed according to the image usage.

[0078] The image encoding method involved in the 13th mode of the present disclosure generates a first image by applying a prefilter set corresponding to the image purpose to an input image, and generates a bit stream by encoding the first image and the prefilter set or a postfilter set corresponding to the prefilter set.

[0079] According to the thirteenth aspect, the bit stream includes the first image and the pre-filter set or the post-filter set, so that the image decoding device can perform the optimal post-filter processing according to the pre-filter processing applied to the first image.

[0080] (Embodiments of the present disclosure)

[0081] Hereinafter, the embodiments of the present disclosure will be described in detail using the drawings. In addition, elements with the same reference numerals in different drawings are assumed to represent the same or corresponding elements.

[0082] In addition, the embodiments described below all show a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples, and their main purpose is not to limit the present disclosure. In addition, for the constituent elements in the following embodiments that are not recorded in the independent claims representing the highest concept, they are described as arbitrary constituent elements. In addition, the respective contents can also be combined in all embodiments.

[0083] Figure 1 1 is a diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an image encoding device 10 , a network Nw, and an image decoding device 20 .

[0084] The image encoding device 10 includes a filter processing unit 11 and an encoding processing unit 12. Image data D1 of an input image is input to the filter processing unit 11. The input image includes a dynamic image, a still image, or a feature amount. The filter processing unit 11 has a plurality of prefilter sets of different types depending on the image usage on the image decoding device 20 side. The filter processing unit 11 selects a prefilter set from the plurality of prefilter sets according to the image usage, generates a first image by performing a filter process using the selected prefilter set on the input image, and outputs image data D2 of the first image. In addition, the filter processing unit 11 outputs filter information D3 related to the filter applied to the input image. The filter information D3 includes a plurality of first filter set information D3a related to the prefilter set (or a postfilter set complementary thereto) applied to the input image and other prefilter sets or postfilter sets, a plurality of second filter information D3b related to a plurality of feature extraction filters, and a plurality of parameter value information D3c. The filter processing unit 11 selects a prefilter to be applied to the input image based on a comparison result between a feature value obtained by applying a feature extraction filter to the input image and a parameter value such as a threshold.

[0085] The encoding processing unit 12 generates a bit stream D4 by performing encoding processing on the image data D2 and the filter information D3, and transmits the bit stream D4 to the image decoding device 20 via the network Nw.

[0086] The network Nw is the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), or any combination thereof. In addition, the network Nw is not necessarily limited to a two-way communication network, and may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. In addition, the network Nw may also be a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blue-Ray Disc) that records the bit stream D4.

[0087] The image decoding device 20 includes a decoding processing unit 21, a filtering processing unit 22, and a task processing unit 23. The decoding processing unit 21 receives a bit stream D4 from the image encoding device 10 via a network Nw, generates a first image by decoding the bit stream D4, and outputs image data D5 of the first image corresponding to the image data D2. In addition, the decoding processing unit 21 obtains filter information D6 corresponding to the filter information D3 by decoding the bit stream D4. The filter information D6 includes a plurality of first filter set information D6a corresponding to a plurality of first filter set information D3a, a plurality of second filter information D6b corresponding to a plurality of second filter information D3b, and a plurality of parameter value information D6c corresponding to a plurality of parameter value information D3c.

[0088] The filter processing unit 22 selects a first filter set from a plurality of first filter sets represented by the first filter set information D6a, selects a second filter from a plurality of second filters represented by the second filter information D6b, and selects a parameter value from a plurality of parameter values ​​represented by the parameter value information D3c, based on the use information D7 representing the use of the image in the task processing unit 23. The image use can be specified by the user, for example, or can be obtained by decoding the bit stream D4 as one of the information in the filter information D6. In addition, the filter processing unit 22 selects a first filter from the first filter set based on a comparison result between a feature value obtained by applying the selected second filter to the first image and the selected parameter value. The filter processing unit 22 generates a second image by applying the selected first filter to the first image, and outputs image data D8 of the second image.

[0089] In addition, one of the plurality of first filter sets may be a bypass filter that bypasses the filtering process and causes the filter processing unit 22 to output the first image (image data D5) as the second image (image data D8). It is possible to avoid performing useless filtering processes by selecting a bypass filter. For example, the bypass filter may be indicated by setting the values ​​of all filter coefficients to specific values ​​(e.g., 0), or it may be indicated by other information instead of setting the values ​​of the filter coefficients. In addition, it is also possible that, when there is no first filter set corresponding to the image purpose indicated by the purpose information D7 among the plurality of first filter sets indicated by the first filter set information D6a, the filtering process is bypassed and the filter processing unit 22 outputs the first image (image data D5).

[0090] The task processing unit 23 uses the second image represented by the image data D8 to execute task processing according to the usage information D7 indicating the usage of the image, and outputs result data D9 such as an inference result.

[0091] Figure 2 FIG. 2 is a diagram showing a simplified structure of the task processing unit 23. The image usage on the image decoding device 20 side includes at least one machine task and human vision. Figure 2 In the example of , the image usage includes a plurality of machine tasks 30A to 30C and human vision 30X. Regarding the usage information D7, for example, when the machine task 30B is selected, the task processing unit 23 executes the machine task 30B based on the image data D8, and outputs result data D9B such as an inference result. By including human vision in the image usage, the filter processing unit 22 can select not only the first filter set suitable for the machine task but also the first filter set suitable for human vision.

[0092] Figure 3A : is a diagram showing an example of a partial structure of an image processing system. The decoding processing unit 21 of the image decoding device 20 has a conversion unit 35. The conversion unit 35 converts the pre-filter set into a post-filter set complementary thereto. In the case of this example, the pre-filter set is included in the bit stream D4 as one of the first filter set information D3a. The decoding processing unit 21 obtains the pre-filter set applied to the input image in the image encoding device 10 by decoding the bit stream D4, and obtains the post-filter set corresponding to the pre-filter set as one of the plurality of first filter sets by converting the pre-filter set using the conversion unit 35. According to such a structure, since the conversion process from the pre-filter set to the post-filter set is performed on the image decoding device 20 side, the processing load of the image encoding device 10 can be reduced.

[0093] Figure 3B 1 is a diagram showing an example of a partial structure of an image processing system. The encoding processing unit 12 of the image encoding device 10 has a conversion unit 35. The conversion unit 35 converts the pre-filter set into a post-filter set complementary thereto. In the case of this example, the post-filter set is included in the bit stream D4 as one of the first filter set information D3a. The decoding processing unit 21 obtains the post-filter set corresponding to the pre-filter set applied to the input image in the image encoding device 10 as one of the plurality of first filter sets by decoding the bit stream D4. According to such a structure, the conversion process from the pre-filter set to the post-filter set is performed on the image encoding device 10 side, thereby reducing the processing load of the image decoding device 20. In addition, the filter processing unit 11 may also include the post-filter set generated according to the image usage on the image decoding device 20 side in one of the plurality of first filter set information D3a regardless of the pre-filter set applied to the input image. Furthermore, the encoding processing unit 12 may not include the conversion unit 35 , and may include a post filter set generated according to the image usage on the image decoding device 20 side in the bit stream D4 as one of the plurality of first filter set information D3 a .

[0094] Figure 4A : is a diagram showing an example of filter conversion processing performed by the conversion unit 35. The conversion unit 35 converts the pre-filter using the noise removal filter 35A with a filter strength of 1 / 2 into a post-filter using the sharpening filter 35B with a filter strength of 2. Alternatively, the conversion unit 35 converts the pre-filter using the sharpening filter 35B with a filter strength of 2 into a post-filter using the noise removal filter 35A with a filter strength of 1 / 2.

[0095] Figure 4B : is a diagram showing an example of filter conversion processing performed by the conversion unit 35. The conversion unit 35 converts the pre-filter using the noise removal filter 35C with a filter strength of 1 / 4 into a post-filter using the sharpening filter 35D with a filter strength of 4. Alternatively, the conversion unit 35 converts the pre-filter using the sharpening filter 35D with a filter strength of 4 into a post-filter using the noise removal filter 35C with a filter strength of 1 / 4.

[0096] Figure 5: is a diagram showing target detection and target segmentation as an example of machine tasks. In target detection, the attributes of the target contained in the input image (in this example, the TV and the person) are detected. In addition to the attributes of the target contained in the input image, the position and number of the targets in the input image can also be detected. Thus, for example, the position of the target of the identification object can also be screened to exclude targets other than the identification object. As specific uses, for example, the detection of faces in cameras and the detection of pedestrians in automatic driving can be considered. In target segmentation, the pixels of the area corresponding to the target are segmented (that is, divided). Thus, for example, in automatic driving, obstacles and roads can be separated to assist the safe driving of the car, detect defects of products in factories, and recognize terrain in satellite images can be considered.

[0097] Figure 6 This is a diagram showing target tracking, behavior recognition, and posture estimation as an example of machine tasks. In target tracking, the movement of a target to which an identification number ID is assigned is tracked. As applications, for example, counting the number of users of facilities such as stores, analyzing the movements of athletes, and the like can be considered. If the processing is further accelerated, the target can be tracked in real time, and can also be applied to camera processing such as autofocus. In behavior recognition, the category of the target's movement is detected (in this example, "riding a bicycle", "walking"). For example, by using anti-theft cameras, it can be applied to the prevention and detection of crimes such as robbery and theft, and the prevention of forgotten operations in factories. In posture estimation, the target's posture is detected by detecting key points and joints. For example, it can be considered in industrial fields such as improving work efficiency in factories, security fields such as detecting abnormal behaviors, and fields such as health care and sports.

[0098] The filter processing unit 11 has a plurality of first filter sets of different types depending on the image usage on the image decoding device 20 side. The types include at least one of the shape, size, and coefficient value of the filter. The first filter set corresponding to the machine task includes at least one of a noise removal filter, a sharpening filter, a bit depth conversion filter, a color space conversion filter, a resolution conversion filter, and a filter using a neural network. The noise removal filter includes at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bidirectional filter, and a median filter that removes noise by reducing the information of the details of the input image. The sharpening filter includes an edge detection filter or an edge emphasis filter, specifically including a Laplace filter, a Gaussian Laplace filter, a Sobel filter, a Prewitt filter, or a Canny edge detection filter. The bit depth conversion filter converts the bit depth of the luminance signal and / or the color signal between the input image and the first image. For example, the bit depth of the first image is converted to be smaller than the bit depth of the input image by discarding the low-order bits of the color signal of the first image to reduce the amount of symbols. The color space conversion filter converts the color space between the input image and the first image. For example, the amount of symbols is reduced by converting the color space of YUV444 in the input image into the color space of YUV422, YUV420 or YUV400 in the first image. The resolution conversion filter converts the resolution of the image between the input image and the first image. The resolution conversion filter includes a downsampling filter that reduces the resolution of the first image compared to the resolution of the input image. The resolution conversion filter may also include an upsampling filter that increases the resolution of the first image compared to the resolution of the input image. In addition, the first filter set corresponding to the machine task may also be, for example, a deblocking filter, an ALF filter, a CCALF filter, a SAO filter, a LMCS filter, or any combination thereof specified in H.266 / VVC (Versatile Video Codec).

[0099] The first filter set corresponding to human vision is a filter that does not perform an operation of reducing the amount of signs of the first image compared to the amount of signs of the input image through filtering processing. The first filter set corresponding to human vision includes a bypass filter that outputs the input image directly as the first image. The first filter set corresponding to human vision may also be a filter that reduces the amount of signs of the first image compared to the amount of signs of the input image through filtering processing, but the effect of reducing the amount of signs is suppressed compared to the first filter set corresponding to the machine task. In addition, the first filter set corresponding to human vision may also be a filter that emphasizes important areas of the input image, but the emphasis effect is suppressed compared to the first filter set corresponding to the machine task.

[0100] Figures 7A to 7I FIG. 1 is a diagram showing an example of a plurality of first filters of different types. Fig. 7A A one-dimensional horizontal filter is shown with N = 5. N is the size of the filter which corresponds to the number of filter coefficient values. Figure 7B A one-dimensional vertical filter with N=7 is shown. Figure 7C A two-dimensional cross filter with N=9 is shown. Fig.7D A two-dimensional square filter with N=25 is shown. Fig. 7E A two-dimensional diamond filter with N=25 is shown. Figure 7F as well as Figure 7G A noise removal filter using a 5×5 Gaussian filter is shown. Figure 7F The filtering strength of the filter is relatively strong. Figure 7G The filtering strength of the filter is relatively weak. Figure 7H The edge detection filter using the 7×7 Laplacian of Gaussian filter is shown. As described above, the plurality of first filters of different types may have different filter shapes, different filter sizes, or different filter coefficient values. Fig.7I The first filter using a neural network is shown. The neural network has a convolution layer L1, a pooling layer L2, and a fully connected layer L3. The image data of the convolution layer L1 is generated by applying the convolution filter F1 to the image data D1 of the input image, and the image data of the pooling layer L2 is generated by applying the pooling filter F2 to the image data of the convolution layer L1. The image data D0 of the prediction residual is output from the fully connected layer L3, and the image data D1 and the image data D0 are added to output the image data D3 of the first image.

[0101] Figure 8: is a diagram showing an example of a lookup table that specifies the selection of the first filter set based on the purpose of the image. The lookup table is maintained by the filter processing unit 11. In the case where the purpose of the image is a machine task of target detection, the detailed information of the target is important, so the filter processing unit 11 selects a weak noise removal filter and a weak sharpening filter. In the case where the purpose of the image is a machine task of target segmentation, the details of the image are important in the target segmentation model (Mask R-CNN) that performs pixel-level prediction, so the filter processing unit 11 selects a weak noise removal filter. In the case where the purpose of the image is a machine task of target tracking, the detailed information of the target is not important, so the filter processing unit 11 selects a strong noise removal filter and a weak sharpening filter. In the case where the purpose of the image is a machine task of behavior recognition, in order to remove redundant information of static content such as the background and emphasize dynamic content such as the edge of the target, the filter processing unit 11 selects a strong noise removal filter and a strong sharpening filter. When the image usage is a machine task of posture estimation, the posture estimation model (HRNet) learns high-resolution images to detect key points of a person such as ears and eyes, so the filter processing unit 11 selects a strong sharpening filter to emphasize the details of the image.

[0102] As described above, the filter processing unit 22 selects a first filter set from a plurality of first filter sets indicated by the first filter information set D6a based on the usage information D7 indicating the image usage in the task processing unit 23. The first filter set may be two or more filters having different filter strengths. Fig. 9A 3 is a diagram showing an example of a sharpening filter with a strong filter strength, and the filter strength is 9. Fig. 9B : is a diagram showing an example of a sharpening filter with a weak filter strength, and the filter strength is 5. In addition, the first filter set may be composed of only one filter.

[0103] In addition, as described above, the filter processing unit 22 selects a second filter from a plurality of second filters represented by the second filter information D6b based on the usage information D7 representing the usage of the image in the task processing unit 23. The second filter is a feature extraction filter for classifying the region in the image based on the image characteristics. As the second filter, any filter that can classify the region in the image, such as a differential filter, a saliency filter, or a segmentation filter, can be used.

[0104] Differentiation filters are used to calculate image gradients such as directional changes in brightness or color within an image. As a differential filter, for example, an edge detector can be used. The edge detector can be a first-order differential filter such as a Sobel filter or a Prewitt filter, or a second-order differential filter such as a Laplace filter or a Gaussian Laplace filter. Figures 10A to 10Cis a diagram showing an example of the second filter. Fig. 10A An example of an edge detector using a Laplace filter is shown in FIG. Fig. 10B An example of a horizontal edge detector using a Sobel filter is shown in FIG. Fig. 10C , an example of a vertical edge detector using a Sobel filter is shown.

[0105] Saliency filters are used to detect visually salient regions within an image that the human eye focuses more on. Visually salient regions can be used to improve human visual recognition scores or reduce computational complexity in machine tasks.

[0106] As the segmentation filter, a brightness-based image segmentation filter, a model-based image segmentation filter, or a hybrid segmentation filter can be used. The brightness-based image segmentation filter performs region segmentation on the image based on the brightness value of each pixel in the image. The model-based image segmentation filter uses a neural network model such as a lightweight object detection model to segment each region of the image.

[0107] As the hybrid segmentation filter, for example, a filter obtained by combining an existing filter and a model-based image segmentation filter can be used.

[0108] Figures 11A to 11C : is a diagram showing an example of the application area of ​​the first filter and the second filter. Each rectangle represents each pixel in the first image. The rectangle marked with hatching represents the pixel to which the first filter is applied, and the rectangle marked with the letter "S" represents the pixel to which the second filter is applied.

[0109] exist Fig.11A In the example shown, the number of pixels in the region (first image region) to which the first filter is applied in the first image is equal to the number of pixels in the region (second image region) to which the second filter is applied in the first image, and the range of the second image region is larger than the range of the first image region. According to this example, the increase in the processing load associated with the application of the second filter can be suppressed, and the influence of local noise can be reduced.

[0110] exist Fig. 11B In the example shown, the number of pixels in the first image region is greater than the number of pixels in the second image region, and the range of the second image region is equal to the range of the first image region.

[0111] think Fig. 11C In the example shown, the number of pixels in the first image region is equal to the number of pixels in the second image region, and the range of the second image region is equal to the range of the first image region.

[0112] As described above, the filter processing unit 22 selects a first filter from the first filter set based on the comparison result between the feature value obtained by applying the selected second filter to the first image and the selected parameter value. The feature value is, for example, the edge strength, and the parameter value is, for example, the threshold. When the edge strength associated with a certain second image area is greater than the threshold, the filter processing unit 22 applies the first filter with a weak filter strength to the first image area corresponding to the second image area. On the other hand, when the edge strength associated with a certain second image area is less than the threshold, the filter processing unit 22 applies the first filter with a strong filter strength to the first image area corresponding to the second image area. In addition, it is also possible to set a structure in which one first filter is selected from more than three first filters by setting more than two thresholds. In addition, the filter processing unit 22 may also select one first filter from more than two first filters based on the task type parameter indicated by the usage information D7. In addition, the first filter set may also be composed of only one first filter. In this case, the second filter may not be used, and the same first filter may be applied to all areas within the screen.

[0113] Fig. 12A 1 is a diagram showing a first example of the data structure of the bit stream D4. The bit stream D4 includes a header H storing management information and the like and a payload P storing image data D2. The encoding processing unit 12 stores the encoded data 70 of the filter information D3 in a predetermined position of the payload P.

[0114] Fig. 12B FIG. 2 is a diagram showing a second example of the data structure of the bit stream D4. Fig. 12A Similarly, the bitstream D4 has a header H and a payload P. The encoding processing unit 12 stores the encoded data 70 of the filter information D3 in a given part of the header H. The given part is, for example, a SEI (Supplemental Enhancement Information) area for storing additional information. The given part may also be a VPS, SPS, PPS, PH, SH, APS, or a tile header. By storing the filter information D3 in the header H of the bitstream D4, the decoding processing unit 21 can easily obtain the filter information D6. In addition, by storing the filter information D3 in the SEI area, the filter information D3 can be simply processed as additional information.

[0115] Fig.13 2 is a diagram showing an example of the syntax of an SEI message related to the first filter set information D3a indicating the post filter. This SEI message specifies the filter coefficients of the post filter or related information used for designing the post filter.

[0116] postfilter_hint_size_y specifies the vertical size of the filter coefficients or related arrays, and takes a value from "1" to "15", for example.

[0117] postfilter_hint_size_x specifies the horizontal size of the filter coefficients or related arrays, and takes a value from "1" to "15", for example.

[0118] num_of_postfilters specifies the total number of post filters, and takes a value from "1" to "15", for example.

[0119] postfilter_hint_type specifies the type of postfilter through a 2-bit flag information, for example, when its value is "0", it indicates a 2-dimensional FIR filter, when its value is "1", it indicates a 1-dimensional FIR filter, and when its value is "2", it indicates the cross-correlation matrix between the input image signal and the filtered image signal.

[0120] cIdx specifies the associated color component. chroma_format_idc specifies the chroma format, for example, when its value is "0" it indicates monochrome, when its value is "1" it indicates YUV420, and when its value is "2" it indicates YUV422. cy indicates a counter in the vertical direction, and cx indicates a counter in the horizontal direction. postfilter_hint_value [cIdx][cy][cx] indicates a filter coefficient or an element of a cross-correlation matrix.

[0121] Fig.14 : is a diagram showing an example of the syntax of the SEI message related to the second filter information D3b indicating the second filter. This SEI message specifies the filter coefficient of the second filter or related information used for designing the second filter.

[0122] derivative_filter_hint_size_y specifies the vertical size of the filter coefficients or related arrays, and can take values ​​from "1" to "15", for example.

[0123] derivative_filter_hint_size_x specifies the horizontal size of the filter coefficients or related arrays, and can take values ​​from "1" to "15", for example.

[0124] num_of_derivative_filters specifies the total number of second filters, and takes a value from "1" to "15", for example.

[0125] For example, derivative_filter_hint_type specifies the type of the second filter through 2-bit flag information. For example, when its value is "0", it indicates a 2-dimensional FIR filter; when its value is "1", it indicates a 1-dimensional FIR filter; and when its value is "2", it indicates the cross-correlation matrix between the input image signal and the filtered image signal.

[0126] cIdx specifies the associated color component. chroma_format_idc specifies the chroma format, for example, when its value is "0" it indicates monochrome, when its value is "1" it indicates YUV420, and when its value is "2" it indicates YUV422. cy indicates a counter in the vertical direction, and cx indicates a counter in the horizontal direction. derivative_filter_hint_value[cIdx][cy][cx] indicates a filter coefficient or an element of a cross-correlation matrix.

[0127] Fig.15 : is a flowchart showing the flow of processing performed by the image encoding device 10.

[0128] First, in step SP101, the filter processing unit 11 generates a first image by performing filter processing on an input image using a prefilter set, and outputs image data D2 of the first image. Furthermore, the filter processing unit 11 outputs filter information D3 on the filter applied to the input image.

[0129] Next, in step SP102, the encoding processing unit 12 generates a bit stream D4 by performing encoding processing on the first image. At this time, the encoding processing unit 12 encodes the filter information D3 and stores the encoded data 70 of the filter information D3 in the bit stream D4. The encoding processing unit 12 sends the generated bit stream D4 to the image decoding device 20 via the network Nw.

[0130] Fig.171 is a diagram showing a simplified example of an installation of an image coding device 10. The image coding device 10 is composed of a processor 101 and a memory 102 connected to the processor 101. However, the memory 102 may also be included in the processor 101. The processor 101 is a circuit that performs information processing. The processor 101 includes a CPU or a GPU, etc. The memory 102 includes a semiconductor memory such as a ROM or a RAM, a magnetic disk, or an optical disk, etc. In the memory 102, information required by the processor 101 to perform pre-filtering processing and encoding processing, etc. is stored. For example, in the memory 102, image data D1, D2 and filter information D3 are stored. In addition, a program is stored in the memory 102. The processor 101 executes the program read from the memory 102, so that the processor 101 functions as a filtering processing unit 11 and an encoding processing unit 12.

[0131] Fig.16 It is a flowchart showing the flow of processing performed by the image decoding device 20.

[0132] First, in step SP201, the decoding processing unit 21 receives the bit stream D4 from the image encoding device 10 via the network Nw, generates a first image by decoding the bit stream D4, and outputs image data D5 of the first image. In addition, the decoding processing unit 21 obtains filter information D6 by decoding the bit stream D4.

[0133] Next, in step SP202, the filtering processing unit 22 selects a first filter set from multiple first filter sets represented by the first filter set information D6a, selects a second filter from multiple second filters represented by the second filter information D6b, and selects a parameter value from multiple parameter values ​​represented by the parameter value information D3c based on the usage information D7.

[0134] Next, in step SP203 , the filter processing unit 22 generates a second image by applying the selected first filter set to the first image, and outputs image data D8 of the second image.

[0135] Next, in step SP204, the task processing unit 23 executes task processing according to the usage information D7 using the second image represented by the image data D8, and outputs result data D9 such as an inference result.

[0136] Fig.182 is a diagram showing a simplified example of an installation of an image decoding device 20. The image decoding device 20 is composed of a processor 201 and a memory 202 connected to the processor 201. However, the memory 202 may also be included in the processor 201. The processor 201 is a circuit that performs information processing. The processor 201 includes a CPU or a GPU, etc. The memory 202 includes a semiconductor memory such as a ROM or a RAM, a magnetic disk, or an optical disk, etc. In the memory 202, information required by the processor 201 to perform decoding processing, post-filtering processing, and task processing, etc. is stored. For example, in the memory 202, a bit stream D4, image data D5, D8, and filter information D6 are stored. In addition, a program is stored in the memory 202. The processor 201 executes the program read from the memory 202, so that the processor 201 functions as a decoding processing unit 21, a filtering processing unit 22, and a task processing unit 23.

[0137] According to this embodiment, the first filter set applied to the first image can be dynamically changed according to the image usage such as each machine task or human vision. Therefore, the optimal filter set can be selected and specified according to the properties of the image required for each image usage. In addition, according to the image usage, useless information is removed from the image transmitted by the bit stream D4, thereby reducing the transmission symbol amount from the image encoding device 10 to the image decoding device 20.

[0138] Furthermore, the first filter to be applied to the first image can be dynamically selected from the first filter set based on the feature value of the first image obtained by applying the second filter.

[0139] Industrial Applicability

[0140] The present disclosure is particularly useful when applied to an image processing system including an image encoding device that transmits an image and an image decoding device that receives an image.

Claims

1. An image decoding device, comprising: Circuits; and a memory connected to the circuit, The circuit is in action. A first image and a plurality of first filter sets are obtained by decoding the bit stream, Based on the usage information indicating the usage of the image, a first filter set is selected from the plurality of first filter sets, and a second image is generated and output by applying the selected first filter set to the first image.

2. The image decoding device according to claim 1, wherein: The circuit performs the following processing: Obtaining a plurality of second filters and a plurality of parameter values ​​by decoding the bit stream; Based on the usage information, select a first filter set from the plurality of first filter sets, select a second filter from the plurality of second filters, and select a parameter value from the plurality of parameter values; Based on the feature value obtained by applying the selected second filter to the first image and the selected parameter value, a first filter is selected from the first filter set, and the second image is generated by applying the selected first filter to the first image.

3. The image decoding device according to claim 2, wherein: The number of pixels of the first image region to which the first filter is applied in the first image is equal to the number of pixels of the second image region to which the second filter is applied in the first image, The range of the second image area is larger than the range of the first image area.

4. The image decoding device according to claim 1, wherein: One of the plurality of first filter sets is a bypass filter that causes the first image to be output as the second image.

5. The image decoding device according to claim 1, wherein: The circuit decodes the bit stream received from the image encoding device to obtain, as one of the plurality of first filter sets, a post filter set corresponding to a pre filter set applied to an input image in the image encoding device.

6. The image decoding device according to claim 1, wherein: The circuit obtains a pre-filter set applied to an input image in the image encoding device by decoding the bit stream received from the image encoding device, and obtains a post-filter set corresponding to the pre-filter set as one of the multiple first filter sets by transforming the pre-filter set.

7. The image decoding device according to claim 1, wherein: The image usage includes at least one of a machine task, and human vision.

8. The image decoding device according to claim 1, wherein: The circuit obtains the plurality of first filter sets by decoding a header of the bit stream.

9. The image decoding device according to claim 8, wherein: The header has a supplemental enhancement information SEI area, The circuit obtains the plurality of first filter sets by decoding the SEI region.

10. An image encoding device comprising: Circuits; and a memory connected to the circuit, The circuit is in action. A first image is generated by applying a set of prefilters corresponding to the purpose of the image to the input image, A bit stream is generated by encoding the first image and the pre-filter set or a post-filter set corresponding to the pre-filter set.

11. The image encoding device according to claim 10, wherein: The image usage is the image usage in the image decoding device, The circuit sends the bit stream to the image decoding device.

12. An image decoding method, comprising: Obtaining a first image and a plurality of first filter sets by decoding a bit stream; Based on the usage information indicating the usage of the image, a first filter set is selected from the plurality of first filter sets, and a second image is generated and output by applying the selected first filter set to the first image.

13. An image encoding method, comprising: Generate a first image by applying a set of pre-filters corresponding to the purpose of the image to the input image; A bit stream is generated by encoding the first image and the pre-filter set or a post-filter set corresponding to the pre-filter set.

Images