Image encoding device, image decoding device, image encoding method, and image decoding method
By selecting a filter based on the use information on the image decoding device side in the image encoding device, appropriate processing of the image is solved, and the problem of low bit efficiency in transmission of the image encoding device to the decoding device is realized, and the coexistence of machine tasks and human vision is realized. efficient bit streaming and task execution.
Patent Information
- Application Number
- CN202380031418.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-23
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has problems with low bit efficiency in the bit stream transmission of the image encoding device to the image decoding device, which leads to an increase in the transmission encoding amount and an increase in delay, which is difficult to optimize especially when machine tasks and human vision coexist.
By selecting an appropriate filter based on the image usage information on the image decoding device side in the image encoding device, filtering the input image and generating a bit stream, including using a noise de-noise filter, a sharpening filter, a bit depth conversion filter, Color space transformation filter and resolution transformation filter reduce the encoding amount of non-important areas and emphasize important areas.
The bit efficiency in bit stream transmission from the image encoding device to the image decoding device is improved, adapting to the image task needs of different purposes, and ensuring the effective execution of machine tasks and human vision.
Smart Images

Figure CN118947114A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image encoding device, an image decoding device, an image encoding method, and an image decoding method. Background Art
[0002] Patent Document 1 discloses a video encoding and decoding method using an adaptive coupled pre-filter and a post-filter.
[0003] Patent Document 2 discloses a method for encoding image data to be mounted on an artificial intelligence (AI) integrated circuit.
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent Document 1: U.S. Patent No. 9883207
[0007] Patent Document 2: U.S. Patent No. 10452955 Summary of the invention
[0008] An object of the present disclosure is to improve the bit efficiency of a bit stream transmitted from an image encoding device to an image decoding device.
[0009] An image encoding device involved in one embodiment of the present invention comprises: a filter processing unit, which generates a first image by filtering an input image; and an encoding processing unit, which generates a bit stream by encoding the first image and sends the bit stream to an image decoding device, wherein the filter processing unit has a plurality of filters of different types, and selects a filter from the plurality of filters based on usage information indicating the image usage on the image decoding device side and applies the filter to the input image. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 This is a diagram that simply shows the configuration of an image processing system according to an embodiment of the present disclosure.
[0011] Figure 2 This is a diagram showing a simplified structure of a task processing unit.
[0012] Figure 3 The figure shows object detection and object segmentation as an example of machine tasks.
[0013] Figure 4 The diagram shows target tracking, behavior recognition, and posture estimation as examples of machine tasks.
[0014] Figure 5 This is a diagram showing a simplified structure of a filter processing unit.
[0015] Fig. 6A FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0016] Figure 6B FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0017] Figure 6C FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0018] Fig.6D FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0019] Fig. 6E FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0020] Fig. 6F FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0021] Figure 6G FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0022] Figure 6H FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0023] Fig.6I FIG. 1 is a diagram showing examples of a plurality of filters of different types.
[0024] Figure 7 : is a diagram showing an example of a lookup table that defines the selection of a filter based on usage information.
[0025] Figure 8 It is a diagram showing a first structural example of a filter.
[0026] Fig. 9 It is a diagram showing a second structural example of a filter.
[0027] Fig.10 It is a diagram showing a third structural example of a filter.
[0028] Fig.11 It is a diagram showing the fourth structural example of the filter.
[0029] Fig.12 This is a diagram showing a first example of the data structure of a bit stream.
[0030] Fig.13 This is a diagram showing a second example of the data structure of a bit stream.
[0031] Fig.14 This is a diagram showing a first example of the syntax of the SEI message related to the filter information.
[0032] Fig.15This is a diagram showing a second example of the SEI message syntax related to filter information.
[0033] Fig.16 This is a diagram for explaining an example of a case where a setting unit sets a threshold value related to an IOU.
[0034] Fig.17 This is a flowchart showing the flow of processing performed by the image encoding device.
[0035] Fig.18 This is a flowchart showing the flow of processing performed by the image decoding device. DETAILED DESCRIPTION
[0036] (Insights that form the basis of this disclosure)
[0037] The goal of existing encoding methods is to provide optimal images for human vision under bit rate constraints.
[0038] Advances in machine learning or neural network-based applications coupled with a rich array of sensors will enable many intelligent platforms that handle large amounts of data, including connected cars, video surveillance, or smart cities. As large amounts of data are generated all the time, existing approaches that include humans in the pipeline become inefficient and impractical in terms of latency and scale.
[0039] Furthermore, in transmission and archiving systems, there are concerns about the need for more compact data representation and low-latency solutions, and for this purpose, VCM (Video Coding for Machines) has been introduced.
[0040] In some cases, machines can communicate with each other and perform tasks without human intervention, or additional processing by humans is required on a specific unfrozen stream, such as a human "supervisor" searching for specific people or scenes in a surveillance camera image.
[0041] In other cases, there are situations where both humans and machines use corresponding bit streams. In the case of connected cars, for humans, the features can be used in image correction functions, and for machines, the features can be used in object detection and segmentation.
[0042] The general system structure includes a pair of an image encoding device and an image decoding device. The input of the system is a dynamic image, a still image or a feature quantity. Examples of machine tasks include object detection, object segmentation, object tracking, behavior recognition, posture estimation or any combination of these. Human vision may be one of the use cases that can be used together with machine tasks.
[0043] According to the conventional technology, since a bit stream of the highest image quality assuming human vision is always transmitted from an image encoding device to an image decoding device, there is a problem that the transmission code amount increases and the bit efficiency is poor.
[0044] In order to solve the related problems, the inventors of the present invention came up with the following insight and conceived of the present disclosure: by performing appropriate filtering processing on the input image on the image encoding device side based on usage information indicating the image usage on the image decoding device side, the bit efficiency can be improved in the transmission of the bit stream from the image encoding device to the image decoding device.
[0045] Next, each aspect of the present disclosure will be described.
[0046] The image encoding device involved in the first embodiment of the present disclosure includes: a filter processing unit, which generates a first image by filtering an input image; and an encoding processing unit, which generates a bit stream by encoding the first image and sends the bit stream to an image decoding device, wherein the filter processing unit has a plurality of filters of different types, and selects a filter from the plurality of filters based on usage information indicating the image usage on the image decoding device side and applies it to the input image.
[0047] According to the first aspect, the filter processing unit includes a plurality of filters of different types, and selects one filter from the plurality of filters based on the usage information indicating the usage of the image on the image decoding device side and applies it to the input image. As a result, the bit efficiency can be improved in the transmission of the bit stream from the image encoding device to the image decoding device.
[0048] The image encoding device involved in the second embodiment of the present disclosure may be, in the first embodiment, that the multiple filters include at least one of a noise removal filter, a sharpening filter, a bit depth conversion filter, a color space conversion filter, a resolution conversion filter, and a filter utilizing a neural network.
[0049] According to the second aspect, the filter processing unit can apply an appropriate filter to the input image according to the image usage on the image decoding device side.
[0050] In the image encoding device according to the third aspect of the present disclosure, in the second aspect, the noise removal filter may include at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bilateral filter, and a median filter.
[0051] According to the third aspect, by applying at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bilateral filter, and a median filter to an input image, noise of the input image can be removed.
[0052] In the image encoding device according to the fourth aspect of the present disclosure, in the second aspect, the resolution conversion filter may include a down-sampling filter that reduces the resolution of the first image compared to the resolution of the input image.
[0053] According to the fourth aspect, the amount of code can be reduced by applying a downsampling filter to an input image.
[0054] In the image coding device according to the fifth aspect of the present disclosure, in any one of the first to fourth aspects, the image usage may include at least one machine task and human vision.
[0055] According to the fifth aspect, it is possible to select to use a filter that reduces the amount of coding when the image is used for machine tasks, and to use a filter that does not reduce the amount of coding more than in the case of machine tasks when the image is used for human vision.
[0056] The image encoding device involved in the 6th mode of the present disclosure may be, in the 5th mode, when the image purpose is the machine task, the filtering processing unit reduces the encoding amount of the first image compared with the encoding amount of the input image through the filtering processing.
[0057] According to the sixth aspect, by applying a filter that reduces the coding amount when the image usage is a machine task, it is possible to improve the bit efficiency in the transmission of the bit stream from the image encoding device to the image decoding device.
[0058] The image encoding device involved in the 7th mode of the present disclosure may be, in the 6th mode, that the filtering processing unit specifies a non-important area in the input image that is not important for the machine task, and reduces the encoding amount of the first image compared with the encoding amount of the input image by deleting information on the detail part of the non-important area.
[0059] According to the seventh method, the coding amount of the first image is reduced by deleting detail information of unimportant areas, thereby eliminating the need to reduce the coding amount of important areas important for machine tasks, thereby enabling machine tasks to be appropriately performed on the image decoding device side.
[0060] The image encoding device according to the eighth aspect of the present disclosure may be configured such that, in the sixth or seventh aspect, the filter processing unit specifies an important area that is important for the machine task within the input image, and emphasizes the important area through the filter processing.
[0061] According to the eighth aspect, since the filter processing unit emphasizes the important area through the filter processing, the machine task can be appropriately performed on the image decoding device side.
[0062] The image encoding device involved in the 9th mode of the present disclosure may be, in the 5th mode, that when the image is used for human vision, the filtering processing unit does not reduce the encoding amount of the first image through the filtering processing compared to the case where the image is used for the machine task.
[0063] According to the ninth aspect, when the image is used for human vision, a filter that does not reduce the coding amount more than in the case of a machine task is applied, so that human vision can be appropriately performed on the image decoding device side.
[0064] In the image encoding device according to the tenth aspect of the present disclosure, in any one of the first to ninth aspects, the encoding processing unit may store filter information related to the filter applied by the filter processing unit to the input image in the bit stream.
[0065] According to the tenth aspect, by storing filter information related to the filter applied to the input image in the bit stream, the filter information can be used in the machine task on the image decoding device side.
[0066] In the image encoding device according to the 11th aspect of the present disclosure, in the 10th aspect, the encoding processing unit may store the filter information in a header of the bit stream.
[0067] According to the eleventh aspect, by storing the filter information in the header of the bit stream, the image decoding apparatus can easily extract the filter information from the bit stream.
[0068] According to the image encoding device according to the twelfth aspect of the present disclosure, in the eleventh aspect, the header may include a SEI area, and the encoding processing unit may store the filter information in the SEI area.
[0069] According to the twelfth aspect, by storing the filter information in the SEI area, the filter information can be easily handled as additional information.
[0070] The image decoding device involved in the 13th mode of the present disclosure comprises: a decoding processing unit, which receives a bit stream containing an encoded image from an image encoding device, and generates a decoded image by decoding the bit stream; and a task processing unit, which performs a machine task based on the decoded image, and the bit stream also includes filter information related to the filter applied by the image encoding device to the input image corresponding to the machine task, and the image decoding device comprises: a setting unit, which extracts the filter information from the bit stream, and sets the parameter value used by the task processing unit when performing the machine task based on the filter information.
[0071] According to the thirteenth aspect, the setting unit extracts filter information from the bitstream and sets parameter values used by the task processing unit when executing the machine task based on the filter information. As a result, the task processing unit can perform appropriate task processing corresponding to the filter applied by the image encoding device to the input image.
[0072] In the image encoding method involved in the 14th mode of the present disclosure, the image encoding device generates a first image by filtering an input image, generates a bit stream by encoding the first image, and sends the bit stream to an image decoding device. In the filtering process, a filter is selected from a plurality of filters of different types based on usage information indicating the purpose of the image on the image decoding device side and applied to the input image.
[0073] According to the 14th aspect, in the filtering process, a filter is selected from a plurality of filters of different types based on the usage information indicating the usage of the image on the image decoding device side and applied to the input image. As a result, the bit efficiency can be improved in the transmission of the bit stream from the image encoding device to the image decoding device.
[0074] In the image decoding method involved in the 15th mode of the present disclosure, the image decoding device receives a bit stream including an encoded image from an image encoding device, generates a decoded image by decoding the bit stream, and performs a machine task based on the decoded image, wherein the bit stream also includes filter information related to the filter applied by the image encoding device to the input image corresponding to the machine task, the filter information is extracted from the bit stream, and the parameter value used when performing the machine task is set based on the filter information.
[0075] According to the twelfth aspect, filter information is extracted from a bitstream, and parameter values used when executing a machine task are set based on the filter information. As a result, it is possible to execute appropriate task processing corresponding to the filter applied by the image encoding device to the input image.
[0076] (Embodiments of the present disclosure)
[0077] Hereinafter, the embodiments of the present disclosure will be described in detail using the drawings. In addition, elements with the same reference numerals in different drawings represent the same or corresponding elements.
[0078] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples, and the main purpose is not to limit the present disclosure. In addition, among the constituent elements in the following embodiments, the constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In addition, the respective contents can also be combined in all embodiments.
[0079] Figure 1 A diagram showing a simplified configuration of an image processing system according to an embodiment of the present disclosure. The image processing system includes an image encoding device 10 , a network Nw, and an image decoding device 20 .
[0080] The image encoding device 10 includes a filter processing unit 11 and an encoding processing unit 12. The image data D1 of the input image and the usage information D2 indicating the usage of the image on the image decoding device 20 side are input to the filter processing unit 11. The input image includes a dynamic image, a still image or a feature quantity. The filter processing unit 11 has a plurality of filters of different types. The filter processing unit 11 selects a filter from the plurality of filters based on the usage information D2 and applies it to the input image. The filter processing unit 11 generates a first image by performing a filtering process on the input image using the selected filter, and outputs the image data D3 of the first image. The encoding processing unit 12 generates a bit stream D4 related to the encoded image by performing an encoding process on the first image, and sends the bit stream D4 to the image decoding device 20 via the network Nw.
[0081] The network Nw is the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), or any combination thereof. In addition, the network Nw is not necessarily limited to a two-way communication network, and may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. In addition, the network Nw may also be a recording medium such as a DVD (Digital Versatile Disc) or a BD (Blue-Ray Disc) that records the bit stream D4.
[0082] The image decoding device 20 includes a decoding processing unit 21, a setting unit 22, and a task processing unit 23. The decoding processing unit 21 receives a bit stream D4 from the image encoding device 10 via a network Nw, generates a decoded image by decoding the bit stream D4, and outputs image data D5 of the decoded image. The task processing unit 23 uses the decoded image to perform a machine task corresponding to the use information D8 indicating the use of the image. The bit stream D4 includes filter information D6 indicating the filter applied to the input image by the image encoding device 10 in response to the machine task. The setting unit 22 extracts the filter information D6 from the bit stream D4, sets the parameter value used by the task processing unit 23 when performing the machine task based on the filter information D6, and outputs its setting information D7. The parameter value includes a threshold value related to the credibility score used for machine task analysis or a threshold value related to IOU (Intersection Over Union). The task processing unit 23 uses the parameter value shown in the setting information D7 to perform the machine task, and outputs result data D9 such as the inference result. In addition, Figure 1 The structure of the image decoding device 20 shown is an example, and the present invention is not limited to this example. For example, the setting unit 22 may be omitted, and the function of the setting unit 22 may be implemented in the decoding processing unit 21 or the task processing unit 23.
[0083] Figure 2 2 is a diagram showing a simplified structure of the task processing unit 23. The image usage on the image decoding device 20 side includes at least one machine task and human vision. Figure 2 In the example of , the image usage includes a plurality of machine tasks 30A to 30C and human vision 30X. When the usage information D8 selects the machine task 30B, for example, the setting unit 22 outputs setting information D7B of parameter values used when the task processing unit 23 executes the machine task 30B. The task processing unit 23 executes the machine task 30B based on the image data D5 using the parameter values shown in the setting information D7B, and outputs result data D9B such as inference results.
[0084] Figure 3This is a diagram showing target detection and target segmentation as an example of a machine task. In target detection, the attributes of the target contained in the input image are detected (in this example, the TV and the person). In addition to detecting the attributes of the target contained in the input image, the position and number of the targets in the input image can also be detected. Thus, for example, the position of the target of the identification object can be narrowed down, or targets other than the identification object can be excluded. As specific uses, for example, consider the detection of faces in cameras, the detection of pedestrians in automatic driving, etc. In target segmentation, the pixels of the area corresponding to the target are segmented (i.e., partitioned). Thus, for example, the following uses are considered: separating obstacles and roads in automatic driving to assist in the safe driving of the car, or detecting defects in products in a factory, or recognizing terrain in satellite images.
[0085] Figure 4 This is a diagram showing target tracking, behavior recognition, and posture estimation as an example of a machine task. In target tracking, the movement of a target to which an identification number ID is assigned is tracked. As applications, for example, the counting of the number of users of a facility such as a store, and the analysis of the activities of athletes are considered. If the processing is further accelerated, the target can be tracked in real time, or it can also be applied to camera processing such as autofocus. In behavior recognition, the category of the target's action is detected (in this example, "riding a bicycle", "walking"). For example, by using it in a crime prevention camera, it can be applied to the prevention and detection of criminal actions such as robbery and theft, and the prevention of forgotten operations in factories. In posture estimation, the posture of the target is detected by detecting key points and joints. For example, consider the use in industrial fields such as improving the operating efficiency in factories, security fields such as detecting abnormal actions, and fields such as medical care and sports.
[0086] Figure 5 : is a diagram showing a simplified structure of the filter processing unit 11. The filter processing unit 11 includes a plurality of filters 40 of different types that can be selected according to the image usage on the image decoding device 20 side. The types include at least one of the shape, size, and coefficient value of the filter. Figure 5 In the example of , the filter processing unit 11 has filters 40A to 40C corresponding to machine tasks 30A to 30C and a filter 40X corresponding to human vision 30X. When the usage information D8 selects the machine task 30B, for example, the usage information D2 selects the filter 40B corresponding to the machine task 30B. In this case, the filter processing unit 11 generates a first image by performing a filter process using the filter 40B on the input image, and outputs image data D3B of the first image.
[0087] The filters 40A to 40C corresponding to the machine task are filters that reduce the amount of code of the first image by filtering compared to the amount of code of the input image. As described later, the filters 40A to 40C can also be filters that reduce the amount of code of non-important areas that are not important to the machine task while emphasizing important areas that represent features that are important to the machine task. The filters 40A to 40C corresponding to the machine task include at least one of a noise removal filter, a sharpening filter, a bit depth conversion filter, a color space conversion filter, a resolution conversion filter, and a filter using a neural network. The noise removal filter includes at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bilateral filter, and a median filter that removes noise by reducing information on the detail part of the input image. The sharpening filter includes an edge detection filter or an edge emphasis filter, specifically a Laplace filter, a Gaussian Laplace filter, a Sobel filter, a Prewitt filter, or a Canny edge detection filter. The bit depth conversion filter converts the bit depth of the brightness signal and / or the color signal between the input image and the first image. For example, the bit depth of the first image is transformed to be smaller than the bit depth of the input image by discarding the lower bits of the color signal of the first image, thereby reducing the amount of coding. The color space conversion filter converts the color space between the input image and the first image. For example, the color space of YUV444 in the input image is converted into the color space of YUV422, YUV420 or YUV400 in the first image to reduce the amount of coding. The resolution conversion filter converts the resolution of the image between the input image and the first image. The resolution conversion filter includes a downsampling filter that reduces the resolution of the first image compared to the resolution of the input image. The resolution conversion filter may also include an upsampling filter that increases the resolution of the first image compared to the resolution of the input image. In addition, the filters 40A to 40C corresponding to the machine task may also be, for example, a deblocking filter, an ALF filter, a CCALF filter, a SAO filter, a LMCS filter, or any combination of these specified in H.266 / VVC (Versatile Video Codec).
[0088] The filter 40X corresponding to human vision is a filter that does not reduce the amount of code of the first image by filtering compared to the amount of code of the input image. The filter 40X corresponding to human vision includes a bypass filter that outputs the input image as the first image without changing the amount of code. The filter 40X may be a filter that reduces the amount of code of the first image by filtering compared to the amount of code of the input image, but the effect of reducing the amount of code is more suppressed than that of the filters 40A to 40C. In addition, the filter 40X may be a filter that emphasizes an important area of the input image, but the emphasis effect is more suppressed than that of the filters 40A to 40C.
[0089] Figures 6A to 6I FIG. 4 is a diagram showing an example of a plurality of filters 40 of different types. Fig. 6A represents a one-dimensional horizontal filter with N = 5. N is the size of the filter which corresponds to the number of filter coefficient values. Figure 6B Represents a one-dimensional vertical filter with N=7. Figure 6C Represents a two-dimensional cross filter with N=9. Fig.6D Represents a two-dimensional square filter with N=25. Fig. 6E Represents a two-dimensional diamond filter with N=25. Fig. 6F as well as Figure 6G represents a noise removal filter using a 5×5 Gaussian filter, Fig. 6F The filter strength of the filter is relatively strong. Figure 6G The filter strength of the filter is relatively weak. Figure 6H 1 shows an edge detection filter using a 7×7 Laplacian of Gaussian filter. As described above, the filters 40 of different types may have different filter shapes, different filter sizes, or different filter coefficient values. Fig.6I Indicates a filter using a neural network. The neural network has a convolution layer L1, a pooling layer L2, and a fully connected layer L3. The image data of the convolution layer L1 is generated by applying the convolution filter F1 to the image data D1 of the input image, and the image data of the pooling layer L2 is generated by applying the pooling filter F2 to the image data of the convolution layer L1. The image data D0 of the prediction residual is output from the fully connected layer L3, and the image data D1 and the image data D0 are added to output the image data D3 of the first image.
[0090] Figure 7: is a diagram showing an example of a lookup table that specifies the selection of filters based on the usage information D2. The lookup table is held by the filter processing unit 11. When the image usage indicated by the usage information D2 is a machine task of target detection, the detailed information of the target is important, so the filter processing unit 11 selects a weak denoising filter and a weak sharpening filter. When the image usage indicated by the usage information D2 is a machine task of target segmentation, the detailed information of the image is important in the target segmentation model (Mask R-CNN) that performs pixel-level prediction, so the filter processing unit 11 selects a weak denoising filter. When the image usage indicated by the usage information D2 is a machine task of target tracking, the detailed information of the target is not important, so the filter processing unit 11 selects a strong denoising filter and a weak sharpening filter. When the image usage indicated by the usage information D2 is a machine task of behavior recognition, the filter processing unit 11 selects a strong denoising filter and a strong sharpening filter in order to emphasize dynamic content such as the edge of the target while removing redundant information of static content such as the background. When the image usage indicated by the usage information D2 is a machine task of posture estimation, the posture estimation model (HRNet) learns high-resolution images to detect key points of humans such as ears and eyes. Therefore, the filter processing unit 11 selects a strong sharpening filter to emphasize the details of the image.
[0091] As an example, the filter processing unit 11 selects a strong noise removal filter in the machine task of target tracking, and selects a weak noise removal filter in the machine task of target detection. Target detection is the process of detecting the object target in the image, and target tracking is the process of tracking the trajectory of the target in the continuous frames in the image. In this case, the edge and shape of the target are indispensable in target tracking, and the detailed information of the target is indispensable in target detection. For this reason, in target tracking, a strong noise removal filter is used to remove the detailed information, and in target detection, a weak noise removal filter is used to remove only the unnecessary information.
[0092] In other examples, the filter processing unit 11 selects a large-size noise removal filter in the machine task of target tracking, and selects a small-size noise removal filter in the machine task of target detection. The small-size noise removal filter has a lower control performance for frequency components, so it can remove noise of frequency components in a wide range, and the large-size noise removal filter has a higher control performance for frequency components, so it can remove noise of frequency components in a specific range. In some cases, the small-size filter has a smaller effect of reducing the amount of code than the large-size filter, and in other cases, the small-size filter has a greater effect of reducing the amount of code than the large-size filter.
[0093] In other examples, the filter processing unit 11 selects a filter with a wide output color range and a large bit depth in the target tracking machine task, and selects a filter with a narrow output color range and a small bit depth in the target detection machine task. By using a filter with a small bit depth, the effect of reducing the amount of coding can be improved.
[0094] In other examples, the filter processing unit 11 selects different color space filters in the machine task of target tracking and the machine task of target detection.
[0095] In other examples, the filter processing unit 11 selects a downsampling filter with a large output resolution and a small scale factor in a target tracking machine task, and selects a downsampling filter with a small output resolution and a large scale factor in a target detection machine task.
[0096] Figure 5 Each filter 40 shown may be composed of one filter, may be composed of a plurality of filters, or may be composed of at least one filter and at least one image processing unit.
[0097] Figure 8 : is a diagram showing a first configuration example of the filter 40. The filter 40 is composed of a single filter 51. The filter 51 is, for example, a Gaussian filter. The filter 51 applies a Gaussian filter to the image data D1, and outputs image data D3 from which noise has been removed.
[0098] Fig. 9 : is a diagram showing a second structural example of the filter 40. The filter 40 is composed of two filters 52 and 53, two image processing units 61 and 62, and an addition processing unit 63. The filter 52 is, for example, a Laplacian of Gaussian filter, the filter 53 is, for example, a Gaussian filter, the image processing unit 61 is, for example, an edge extraction processing unit, and the image processing unit 62 is, for example, a background extraction processing unit. The filter 52 outputs an edge map representing the location of edges in the image by applying a Laplacian of Gaussian filter to the image data D1 of the input image. The filter 53 outputs image data from which noise has been removed by applying a Gaussian filter to the image data D1 of the input image. The image processing unit 61 outputs an edge image by extracting edges from the input image based on the image data D1 and the edge map. The image processing unit 62 outputs a background image by extracting the background from the input image based on the image data from which noise has been removed and the edge map. The addition processing unit 63 outputs image data D3 of the first image by adding the edge image and the background image. In Fig. 9 In the example, the filtering processing unit 11 specifies the background area in the input image as a non-important area that is not important for the machine task, and deletes the information of the detail part (i.e., the high-frequency band) of the background area using the filter 53, thereby reducing the encoding amount of the first image compared to the encoding amount of the input image.
[0099] Fig.10 : is a diagram showing a third structural example of the filter 40. The filter 40 is composed of two filters 54 and 55, an image processing unit 64, and an addition processing unit 65. The filter 54 is, for example, a face detection filter using a neural network, the filter 55 is, for example, an averaging filter, and the image processing unit 64 is, for example, a region segmentation processing unit. The filter 54 detects the face of a person contained in the image data D1 of the input image, and outputs a face bounding box that defines the position coordinates of the face. The image processing unit 64 separates the input image into an area other than the face and an area of the face based on the image data D1 and the face bounding box. The filter 55 outputs a blurred face image by averaging the face area. The addition processing unit 65 outputs the image data D3 of the first image by adding the image of the area other than the face and the blurred face image. For example, in a machine task of target tracking, the details of the face are not important, so by blurring the face, the amount of coding can be reduced and privacy can be protected. In Fig.10 In the example, the filtering processing unit 11 specifies the face area in the input image as a non-important area that is not important for the machine task, and deletes the information of the detail part (i.e., the high-frequency band) of the face area using the filter 55, thereby reducing the encoding amount of the first image compared to the encoding amount of the input image.
[0100] Fig.11 : is a diagram showing the fourth structural example of the filter 40. The filter 40 is composed of three filters 56 to 58, an image processing unit 66, and an addition processing unit 67. Filter 56 is, for example, a foreground detection filter using a neural network, filter 57 is, for example, a sharpening filter, filter 58 is, for example, a noise removal filter, and image processing unit 66 is, for example, a region segmentation processing unit. Filter 56 detects an important foreground corresponding to a subject contained in the image data D1 of the input image, and outputs a foreground bounding box that specifies the position coordinates of the foreground. The image processing unit 66 separates the input image into a foreground area and a background area other than the foreground area based on the image data D1 and the foreground bounding box. Filter 57 outputs a foreground image that is sharpened by edge emphasis, etc. by sharpening the foreground area. Filter 58 outputs a background image from which noise has been removed by performing noise removal processing on the background area. The addition processing unit 67 outputs image data D3 of the first image by adding the sharpened foreground image and the background image from which noise has been removed. In Fig.11 In the example of , the filter processing unit 11 specifies the background area in the input image as a non-important area that is not important for the machine task, and deletes the information of the detail part (i.e., the high frequency band) of the background area using the filter 58, thereby reducing the encoding amount of the first image compared to the encoding amount of the input image. Fig.11In the example, the filter processing unit 11 specifies a foreground area in the input image as an important area representing features important to the machine task, and emphasizes the foreground area using the filter 57, thereby improving the accuracy of machine tasks such as behavior recognition or posture estimation.
[0101] Fig.12 1 is a diagram showing a first example of the data structure of a bit stream D4. The bit stream D4 includes a header H storing management information and the like, and a payload P storing image data. The encoding processing unit 12 stores the encoded data 70 of the filter information D6 indicating the filter applied to the input image by the image encoding device 10 in accordance with the machine task, in a predetermined position of the payload P.
[0102] Fig.13 is a diagram showing a second example of the data structure of the bit stream D4. Fig.12 Similarly, the bit stream D4 has a header H and a payload P. The encoding processing unit 12 stores the encoded data 70 of the filter information D6 in a given location of the header H. The given location is, for example, a SEI (Supplemental Enhancement Information) area for storing additional information. The given location may also be a VPS, SPS, PPS, PH, SH, APS, or a tile header.
[0103] Fig.14 This is a diagram showing a first example of the SEI message syntax related to the filter information D6.
[0104] prefilter_type_idc specifies the type of filter through 3-bit marking information, for example, when the value is "0", it indicates a noise removal filter, when it is "1", it indicates a sharpening filter, when it is "2", it indicates a bit depth transformation filter, when it is "3", it indicates a color space transformation filter, when it is "4", it indicates a resolution transformation filter, and when it is "5", it indicates other filters.
[0105] filter_strength_level_idc specifies the strength of the filter by, for example, 3-bit flag information. When the value is "0", it indicates the weakest filter strength, and a larger value indicates a stronger filter strength. The maximum value of the filter strength is "7" or an arbitrary integer.
[0106] input_bit_depth_minus8 specifies the bit depth of the input image before the filter process is applied, for example, by 3-bit flag information, and the bit depth of the input image is "8", "10", "12", or an arbitrary integer.
[0107] input_color_format_idc specifies the color space of the input image before the filter process is applied, for example, by 3-bit flag information. The color space that can be specified is monochrome, YUV444, YUV422, YUV420, YUV400, or an arbitrary color space.
[0108] scale_factor specifies the ratio of the resolution of the input image to the resolution of the first image. For example, when the resolution of the input image is 1920×1080 and the resolution of the first image is 960×540, the vertical and horizontal resolutions become 1 / 2, so scale_factor_nominator becomes "1" and scale_factor_denominator becomes "2". scale_factor_nominator and scale_factor_denominator are each, for example, 3-bit flag information, and can specify an arbitrary integer.
[0109] Fig.15 This is a diagram showing a second example of the SEI message syntax related to the filter information D6.
[0110] prefilter_hint_size_y specifies the size of the filter coefficients or the related array in the vertical direction, and is, for example, an arbitrary integer from "1" to "15".
[0111] prefilter_hint_size_x specifies the size of the filter coefficients or the related array in the horizontal direction, and is, for example, an arbitrary integer from "1" to "15".
[0112] prefilter_hint_type specifies the type of filter through 2-bit flag information, for example, when the value is "0", it indicates a two-dimensional FIR filter, when it is "1", it indicates two one-dimensional FIR filters, and when it is "2", it indicates a cross-correlation matrix.
[0113] prefilter_hint_value specifies the filter coefficients or elements of the cross-correlation matrix.
[0114] Fig.16 2 is a diagram for explaining an example of a case where the setting unit 22 sets a threshold value related to IOU as a parameter value used when executing a machine task. IOU is a value used to express the degree of overlap between a predicted bounding box and a true bounding box in object detection. Fig.16In the figure, the predicted bounding box is represented by a dotted line, and the true bounding box is represented by a dashed line. The IOU is calculated as the ratio of the area (numerator) of the region containing both bounding boxes to the area (denominator) of the region containing at least one bounding box. When the threshold associated with the IOU is set close to "1", the detection result becomes "true" only when the bounding boxes of both sides roughly overlap. When the threshold associated with the IOU is set close to "0", the detection result becomes "true" even when the bounding boxes of both sides slightly overlap. The setting unit 22 extracts the filter information D6 from the bitstream D4, sets the threshold associated with the IOU based on the filter information D6, and inputs the setting information D7 to the task processing unit 23. The task processing unit 23 uses the threshold associated with the IOU shown in the setting information D7 to execute the machine task.
[0115] Fig.17 It is a flowchart showing the flow of processing executed by the image encoding device 10.
[0116] First, in step SP101 , the filter processing unit 11 selects one filter from a plurality of filters based on the usage information D2 .
[0117] Next, in step SP102 , the filter processing unit 11 applies the filter selected in step SP101 to the input image to perform filter processing, thereby generating a first image.
[0118] Next, in step SP103, the encoding processing unit 12 generates a bit stream by encoding the first image. At this time, the encoding processing unit 12 encodes the filter information D6 indicating the filter applied by the filter processing unit 11 to the input image, and stores the encoded data 70 of the filter information D6 in the bit stream D4. The encoding processing unit 12 transmits the generated bit stream D4 to the image decoding device 20 via the network Nw.
[0119] Fig.18 It is a flowchart showing the flow of processing executed by the image decoding device 20.
[0120] First, in step SP201, the decoding processing unit 21 receives the bit stream D4 from the image encoding device 10 via the network Nw, and generates a decoded image by decoding the bit stream D4.
[0121] Next, in step SP202 , the setting unit 22 extracts the filter information D6 from the decoded bit stream D4 , and sets the parameter value used when the task processing unit 23 executes the machine task based on the filter information D6 .
[0122] Next, in step SP203, the task processing unit 23 executes the machine task using the decoded image decoded in step SP201 and the parameter values set in step SP202, and outputs result data D9 such as inference results.
[0123] According to the present embodiment, the filter processing unit 11 includes a plurality of filters of different types, and selects one filter from the plurality of filters and applies it to the input image based on the usage information indicating the usage of the image on the side of the image decoding device 20. As a result, the bit efficiency can be improved in the transmission of the bit stream D4 from the image encoding device 10 to the image decoding device 20.
[0124] Furthermore, according to the present embodiment, the filter processing unit 11 can apply an appropriate filter to an input image in accordance with the image usage on the image decoding device 20 side.
[0125] Furthermore, according to the present embodiment, at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bilateral filter, and a median filter is applied to an input image as a noise removal filter, thereby removing noise from the input image.
[0126] Furthermore, according to the present embodiment, the amount of coding can be reduced by applying a down-sampling filter to an input image.
[0127] Furthermore, according to the present embodiment, it is possible to select to use a filter that reduces the amount of coding when the image is used for machine tasks, and to use a filter that does not reduce the amount of coding more than in the case of machine tasks when the image is used for human vision.
[0128] Furthermore, according to the present embodiment, by applying a filter that reduces the amount of coding when the image usage is a machine task, it is possible to improve the bit efficiency in the transmission of the bit stream D4 from the image coding device 10 to the image decoding device 20 .
[0129] Furthermore, according to the present embodiment, the coding amount of the first image is reduced by deleting detail information of unimportant areas, thereby eliminating the need to reduce the coding amount of important areas important for machine tasks, and thus the machine tasks can be appropriately performed on the image decoding device 20 side.
[0130] Furthermore, according to the present embodiment, since the filter processing unit 11 emphasizes the important region through the filter processing, it is possible to appropriately perform the machine task on the image decoding device 20 side.
[0131] Furthermore, according to the present embodiment, when the image is used for human vision, a filter that does not reduce the coding amount more than in the case of machine tasks is applied, so that human vision can be appropriately performed on the image decoding device 20 side.
[0132] Furthermore, according to the present embodiment, by storing the filter information D6 related to the filter applied to the input image in the bit stream D4, the filter information D6 can be used in the machine task on the image decoding device 20 side.
[0133] Furthermore, according to the present embodiment, by storing the filter information D6 in the header H of the bit stream D4, the image decoding device 20 can easily extract the filter information D6 from the bit stream D4.
[0134] Furthermore, according to the present embodiment, by storing the filter information D6 in the SEI area, the filter information D6 can be easily handled as additional information.
[0135] Furthermore, according to the present embodiment, the setting unit 22 extracts the filter information D6 from the bit stream D4, and sets the parameter value used when the task processing unit 23 executes the machine task based on the filter information D6. As a result, the task processing unit 23 can execute appropriate task processing in accordance with the filter applied to the input image by the image encoding device 10.
[0136] Industrial Applicability
[0137] The present disclosure is particularly useful in the application of an image processing system including an image encoding device for transmitting images and an image decoding device for receiving images.
Claims
1. An image encoding device, comprising: a filter processing unit that generates a first image by performing a filter process on an input image; and an encoding processing unit that generates a bit stream by encoding the first image and sends the bit stream to an image decoding device, The filter processing unit includes a plurality of filters of different types, and selects one filter from the plurality of filters based on usage information indicating usage of the image on the image decoding device side, and applies the selected filter to the input image.
2. The image encoding device according to claim 1, wherein: The plurality of filters include at least one of a noise removal filter, a sharpening filter, a bit depth conversion filter, a color space conversion filter, a resolution conversion filter, and a filter using a neural network.
3. The image encoding device according to claim 2, wherein: The noise removal filter includes at least one of a low-pass filter, a Gaussian filter, a smoothing filter, an averaging filter, a bilateral filter, and a median filter.
4. The image encoding device according to claim 2, wherein: The resolution conversion filter includes a down-sampling filter that reduces the resolution of the first image compared to the resolution of the input image.
5. The image encoding device according to claim 1, wherein: The image usage includes at least one of a machine task, and human vision.
6. The image encoding device according to claim 5, wherein: When the image usage is the machine task, the filter processing unit reduces the code amount of the first image compared with the code amount of the input image by the filter processing.
7. The image encoding device according to claim 6, wherein: The filter processing unit specifies a non-important area that is not important for the machine task in the input image, and reduces the code amount of the first image compared to the code amount of the input image by deleting information on detail portions of the non-important area.
8. The image encoding device according to claim 6, wherein: The filter processing unit specifies an important area that is important for the machine task in the input image, and emphasizes the important area through the filter processing.
9. The image encoding device according to claim 5, wherein: When the image usage is the human vision, the filter processing unit does not reduce the encoding amount of the first image by the filter processing compared to the case where the image usage is the machine task.
10. The image encoding device according to claim 1, wherein: The encoding processing unit stores filter information related to the filter applied by the filter processing unit to the input image in the bit stream.
11. The image encoding device according to claim 10, wherein: The encoding processing unit stores the filter information in a header of the bit stream.
12. The image encoding device according to claim 11, wherein: The header has a supplementary enhancement information, i.e., a SEI area, The encoding processing unit stores the filter information in the SEI area.
13. An image decoding device comprising: a decoding processing unit that receives a bit stream including an encoded image from the image encoding device and generates a decoded image by decoding the bit stream; and a task processing unit, which executes a machine task based on the decoded image, The bitstream also includes filter information related to a filter applied by the image encoding device to the input image in response to the machine task. The image decoding device includes a setting unit that extracts the filter information from the bit stream and sets a parameter value used by the task processing unit when executing the machine task based on the filter information.
14. A method for encoding an image, wherein an image encoding device performs the following processing: The first image is generated by filtering the input image. By encoding the first image to generate a bit stream, the bit stream is sent to an image decoding device, In the filtering process, one filter is selected from a plurality of filters of different types based on usage information indicating the usage of the image on the image decoding device side and applied to the input image.
15. A method for decoding an image, wherein the image decoding device performs the following processing: receiving a bit stream including an encoded image from an image encoding device, and generating a decoded image by decoding the bit stream, performing a machine task based on the decoded image, wherein: The bitstream also includes filter information related to a filter applied by the image encoding device to the input image in response to the machine task. The filter information is extracted from the bitstream, and a parameter value used when executing the machine task is set based on the filter information.
Citation Information
Cited By
Video coding preprocessing method and device, electronic equipment, storage medium and computer program product
CN120499379A
