Image processing method for extending an image
Patent Information
- Application Number
- CA3323862
- Authority / Receiving Office
- CA · CA
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-14
- Filing Date
- 2025-03-12
- Publication Date
- 2026-10-08
AI Technical Summary
Existing video coding methods extend image edges by repeating edge pixels, leading to unnatural representations and image errors.
An image processing method that adds an extension region by determining a target area smaller than the expansion area, using a template area within the image to find similar areas, and generating image information from source areas to fill the target areas, optimizing the expansion process with minimal computational effort.
The method provides a high-quality image expansion with reduced computational effort, improving the quality of image edges and reducing errors, suitable for video coding applications.
Abstract
Description
[0001] Image processing method for enhancing an image
[0002] Technical area
[0003] The present invention relates to an image processing method for enhancing an image, which is used in particular in a video coding method, and to an associated computer program.
[0004] State of the art
[0005] Particularly in the field of video coding, it is necessary to extend images beyond their edges. This is necessary, for example, when video coding requires access to information at the edge of, for example, a reference image as part of a so-called interprediction. To implement such an extension, the edge pixels are usually repeated multiple times in the current state of the art. However, this leads to image errors, since simply repeating the edge pixels does not result in a natural representation of the edge.
[0006] Description of the invention
[0007] The present invention therefore relates, in a first aspect, to an image processing method for enhancing an image by adding an additional extension region.
[0008] Such an image processing method can be used in particular in the field of video coding and here in particular in the field of padding reference pictures or sub-pictures. An extension region is understood to be a region which is arranged on one or more sides of the image and supplements the image with an additional image region. An image can, for example, be the aforementioned reference image or the aforementioned sub-image in the field of video coding. In the first step of the method, a first target region is determined in the extension region, wherein the first target region borders the image and the extent of the first target region perpendicular to the image boundary is smaller than the extent of the extension region in the section of the first target region.Thus, a target area is first determined, which is understood to be an area that is to be filled with additional image information, but which is not part of the original image. As described above, the now processed target area is designed such that its extent in a direction away from the image edge, i.e. perpendicular to the image boundary, is smaller than the extent of the expansion area. If, for example, the expansion area in this section is to have an extent of four image units, for example pixels, away from the image boundary, i.e. the image is to be expanded by four pixels, then the target area defined here can, for example, have an extent in the direction away from the boundary of just one image unit, for example one pixel. In the direction along the boundary, i.e. parallel to the image boundary, the extent can be one or more image units.
[0009] In the next step, a template area is defined. The template area is located within the image and borders the first target area. This means that the template area is directly adjacent to the target area, but is located within the image and thus also on the other side of the image boundary. The template area can have any specified size, such as 8 x 8 image units, for example, pixels.
[0010] The next step involves searching for an area within the image that is similar to the template area. Similarity can be determined here, for example, using a similarity metric such as the sum of absolute differences (SAD). This means that for a large number of different areas in the image, such as all areas in a predefined search area, a sum of the absolute differences can be calculated and an area can be identified that is particularly similar to the template area. A further step then involves determining a first source area that has the same relative position to the similar area as the first target area has to the template area. As a result, a first source area is determined that is located in the same way to the similar area as the template area is to the first target area.
[0011] The image information from the first source area is then used, in a next step, to generate image information for the first target area based on the image information from the first source area. As will be described in more detail later, in the simplest form, the image information from the first source area can be transferred to the first target area.
[0012] As explained above, the first target area is smaller than the extension area in an extent perpendicular to the image boundary, which means that additional information perpendicular to the image boundary must be added in the extension area in order to fill the entire extension area.
[0013] For this purpose, a second target area is then determined in the extension area in the next step, whereby the second target area adjoins the first target area perpendicular to the image boundary.
[0014] Image information for the second target area is then determined based on image information from a second source area. This second source area is an area that has the same relative position to the second target area as the first source area has to the first target area. In other words, the displacement vector between the first source area and the first target area is equal to the displacement vector between the second source area and the second target area.
[0015] Depending on the position of the first source area relative to the image boundary, it may be necessary to fill other sections located parallel to the image boundary next to the first target area by executing the aforementioned method before processing the second target area.
[0016] Thus, the once derived displacement vector, or the once derived ratio between the first source area and the first target area, can be used to generate the further image information in the direction perpendicular to the image boundary.
[0017] The step of generating image information for a second target area can be repeated as often as desired in order to fill additional target areas in the extension area adjacent to the previously filled target areas perpendicular to the image boundary.
[0018] Thus, using the above-mentioned method, the expansion area away from the image boundary can be successively filled with image information until the entire desired expansion area is filled. A similar procedure can also be used for the areas in the expansion area along the image boundary.
[0019] This makes it possible, with minimal computational effort, to fill the extension region in a way that is significantly higher quality than simply repeating the edge pixels. Furthermore, by selecting a target region that is smaller in extent away from the image boundary than the extension itself, the search area that can be used to search for a region similar to the original region can be increased. This is because choosing a larger target region would enlarge an area of the image that would not be usable for the search, as otherwise the source region could end up outside the image.
[0020] In one embodiment, generating image information for the first target area based on image information from the first source area can involve copying the image information from the first source area to the first target area. The same applies to the second target area and the second source area. Thus, in a simple embodiment, the image information is transferred from the first source area to the first target area.
[0021] In a further aspect, searching for an area in the image that is similar to the original area can comprise searching for a plurality of similar areas. Thus, not only the most similar area in the image is searched for, but also a plurality of further similar areas, wherein the number of areas to be used can either be predetermined or, as will be described below, variably determined. Then, in the step of generating image information for the first target area based on image information of the first source area, the method can comprise averaging respective source areas of the plurality of similar areas, wherein the respective source areas of the plurality of similar areas each have the same relative position to the respective one of the plurality of similar areas as the original area has to the first target area.In other words, a plurality of similar regions are determined, for each of which an associated source region is then determined, which has the same relative position to the respective similar region as the original region has to the first target region. An average of the image information of these source regions is then calculated, and this average is then used as image information for the first target region.
[0022] Since the use of a fixed number of source regions could lead to the use of similar regions and associated source regions which only have a low similarity to the template region, it is possible in a further aspect to variably determine the number of similar regions or source regions to be used. For this purpose, the search for several similar regions comprises the following further steps: First, a similarity value is determined for each of the several regions. Then, the best similarity value is determined from these similarity values. This can, for example, be the smallest value or the largest value, depending on the metric used to determine the similarity. This best similarity value is then multiplied by a predetermined value. This resulting value is then used as the threshold for selecting the similar regions and their associated source regions.To do this, those of the several similar areas are selected whose similarity value is better than, or equal to, the best similarity value multiplied by the predetermined value. This can be a smaller or larger value, depending on the similarity value and factor used.
[0023] The multiple similar regions selected in this way are then used, as described above, to determine the respective source regions and then to average the respective image information of the respective source regions and to generate the image information for the first target region. Furthermore, the maximum number of similar regions to be used can be limited by a threshold value to limit the computational effort.
[0024] The quality of the image information of the extension area is thus further improved since no source areas are used for averaging whose associated similar areas have only a low similarity to the original area.
[0025] Furthermore, in one aspect, the search for similar areas can be accelerated by variably defining a corresponding search area.
[0026] To do this, finding an area in the image similar to the template area involves the following additional steps: First, a search area is defined in the image in which to search for a similar area. This could, for example, be an area in the image a certain distance from the template area.
[0027] A termination criterion is then defined. This could, for example, be a threshold for a similarity value.
[0028] It is then determined whether the specified search area contains an area that meets the termination criterion. For example, it checks whether the search area contains an area with a similarity value equal to or better than the threshold previously defined as the termination criterion. If this is the case, the search is terminated and the process described above is performed. However, if no area meets the termination criterion, the search area is enlarged, for example, by increasing the distance from the template area, and a new search is performed in the now larger search area.
[0029] Thus, by initially selecting a smaller search area and successively enlarging this search area, the process of finding a similar area can be accelerated and the computational intensity of the procedure can be reduced.
[0030] According to a further aspect, the determination of similarity when searching for a region of the image similar to the template region may be performed based on the luminance component of the similar region and the search region.
[0031] If the searches were to be performed on both the luminance and chrominance components, this would result in increased computational effort. In addition, the images usually exhibit what is known as color subsampling. For example, digital images in the JPEG standard or digital video material in the MPEG standard often use what is known as 4:2:0 sampling, whereby a corresponding set of chrominance components is available for only four luminance values. If one wanted to perform a corresponding search based on the luminance component and the chrominance component, one would first have to increase the sampling rate, known as upsampling, in order to obtain a chrominance value for each luminance value. After the process is complete, a downsampling would then have to be performed to generate the chrominance components again. This generates a large computational effort.However, studies have shown that a search based solely on the luminance components can also be used to find similar regions without any loss of quality. Accordingly, determining a similarity when searching for a region in the image similar to the original region based on the luminance components of the similar region can be understood as searching based on the luminance components without considering the chrominance components.
[0032] In order to be able to determine the target range of the chrominance components in addition to the target range of the luminance component after finding the similar range based on the luminance components, without performing upsampling and downsampling, a luminance shift vector can first be determined from the first source range to the first target range for the luminance component. This means that a luminance shift vector is determined that maps the shift between the source range and the target range of the luminance component. Then, by simply applying this luminance shift vector, the target range of the luminance component can be derived based on the first source range of the luminance component.
[0033] The chrominance components for the first destination region can also be derived based on the chrominance components of the first source region using the luminance shift vector.
[0034] In one aspect, for example, the luminance shift vector can be followed in reverse from the target area to the source area and then, using an interpolation method, for example a nearest neighbor method, the source values of the chrominance component in the vicinity of the source area of the luminance component can be selected and, based on this, the chrominance component for the target area can be determined.
[0035] As already explained above, this method is particularly suitable for use in the field of video coding, and in particular in the area of padding reference pictures or sub-pictures. Accordingly, a further aspect comprises a video coding method that includes one of the aforementioned image processing methods. A further aspect further relates to a computer program that, when executed on a computer, executes a method according to one of the preceding claims.
[0036] Method steps described above that are not directly dependent on one another are not to be understood as being limited to the illustrated order of execution in one aspect, but can also be executed in a different order according to one aspect. However, according to another aspect, they can also be executed in the order defined above.
[0037] Short description of the characters
[0038] Figure 1 shows schematically filling a target area by copying a source area.
[0039] Figure 2 shows schematically the filling of a target area by averaging several source areas.
[0040] Figure 3 shows schematically a successive creation of an extension area.
[0041] Figure 4 shows a derivation of a chrominance component based on a luminance vector.
[0042] Figure 5 shows a method according to one aspect.
[0043] Detailed description of embodiments
[0044] Figure 1 first describes the general method of expanding image 101 by copying information from a source area 107. Figure 1 shows image 101, which is to be expanded by adding further information at the edge, and thus beyond image boundary 102. If, for example, image information is to be added to the expansion area above image boundary 102, a target area 103 is first determined, which borders image boundary 102. Then, a template area 104 adjacent to this target area 103 is determined, which lies within image 101 and borders target area 103 across image boundary 102. It is evident that target area 103 and template area 104 have a specific relationship to one another. Then, as shown in Figure 1 b), an area 106 similar to the template area 104 is searched within the search area 105.As can be seen from Figures 1a) and 1b), the search area 105 is not directly adjacent to the image boundary 102, but is spaced from the image boundary 102 according to the extent of the target area 103. This is the case because, if a similar area were searched up to the image boundary 102, the corresponding source area 107 would lie outside the image 101 and thus could not contain any information.
[0045] Accordingly, the search area 105 is now searched for areas that have a high degree of similarity to the template area 104. In one embodiment, a similarity metric such as the sum of absolute differences (SAD) can be used. If an area with a particularly high degree of similarity, such as the similar area 106 shown here, is found, a source area 107 is then determined which lies in the same spatial relationship to the similar area as the target area 103 is to the template area 104. This results in the displacement vector 108, which can be seen in Figure 1 b). The image information of the source area 107 is then transferred to the target area 103. The image 101 has thus been expanded to include the target area 103.Accordingly, this can be carried out along the image boundary 102 in order to extend the image 101 over its entire length, or in its entire periphery, by a corresponding extension area.
[0046] Instead of simply copying the information from one source region, an average of several source regions can also be performed, as shown in Figure 2. First, the target region 103 and the template region 104 are again determined, as shown in Figure 2a). Then, a search is carried out in the search region 105 for several similar regions, each labeled 106 in Figure 2. The number can either be predetermined, or those regions can be selected as similar regions that are not significantly worse than a most similar of the similar regions. For this purpose, for example, a similarity metric can be determined for each region in the search region 105. This similarity metric can then be multiplied by a specific factor, and this resulting value can be used as a threshold value.All regions with a similarity value less than this threshold are then selected as similar regions 106. An associated source region 107 is then determined for each of the similar regions. The similar region, in turn, has the same spatial relationship to the associated source region as the original region 104 has to the target region 103. The image information of the plurality of source regions 107 can then be averaged, and based on this averaging, the image information for the target region 103 can be derived.
[0047] With reference to Figure 3, the successive construction of an extension area will now be described.
[0048] If an expansion area 110 in Figure 3a), which extends between an image boundary 102 and an expansion area boundary 109, is to be filled with image information, it is advantageous, as already explained above, to select the extent of a target area in the direction perpendicular to an image boundary 102 as small as possible, so that the area of the image 101 that is not usable for a search and is delimited by the image boundary 102 is kept as small as possible and thus the search area 105, which is delimited by the search area boundary 118, extends as far as possible to the edge of the image 101, i.e., to the image boundary 102. Thus, in a first step of the present example, a first target area 103 is defined, which in this example is four units wide but only one unit deep, vertically to the image boundary 102.The search thus results in a search area 105 that is only one image unit, for example, one pixel, away from the image boundary 102 and can thus be designed as large as possible. The search area boundary 118 and the image boundary 102 are therefore only one image unit, for example, one pixel, apart from each other.
[0049] In contrast, if the full depth of four image units between the image boundary 102 and the extension area boundary 109 were selected as the size of the target area, the search area 105 would be significantly restricted, since the distance of the search area 105 in the image 101 to the image boundary 102 would also have to be four image units.
[0050] Then, as described above, a template area 104 is determined as the target area 103 and then a similar area is searched for, with the help of which a corresponding source area 107 is determined.
[0051] To simplify the drawings, the similar area is not shown in Figure 3a), but only the resulting source area 107.
[0052] The image information in the target area 103 can then be supplemented using the image information in the source area 107. If this is also carried out for the areas adjacent horizontally along the image boundary 102 using a corresponding method, this results in a first extended image line 111. The displacement vector 108 already determined as the displacement vector 108 between the first source area 107 and the first target area 103, as shown in Figure 3b), can then be applied again to fill a second target area 112, which is located below the first target area 103, with image information from a second source area 113. In this case, an area is selected as the second source area 113 which has the same spatial relationship to the second target area 112 as the first source area 107 has to the first target area 103. In other words, the displacement vector 108 is applied to the second target area 112.
[0053] If this is now also carried out for the horizontally adjacent regions of the second target region 112 in Figure 3b), the second extended image line 114 in Figure 3c) results. This can now be carried out until the entire extension region 110 is filled up to the extension region boundary 109, as shown in Figure 3d). The end result is the extended image 101 shown in Figure 3d), which can be used for the purposes described above in the field of video coding. Figure 4 shows an example of the application in the field of an image with 4:2:0 color subsampling. First, the method described above can be carried out in such a way that the search for similar regions is carried out only on the basis of the luminance components 115.This then results in the luminance shift vectors 116 between the source region and the target region, which represent the shift in the luminance values between the source region and the target region. If the additional chrominance component is now to be determined for a corresponding target region, the shift vector can be followed along the luminance shift vector 116 to the corresponding section in the source region, represented by a dashed arrow in Fig. 4, and from this, for example, using a nearest neighbor method, the corresponding value of the chrominance component, here the value 117, can be determined, again represented by a dashed arrow.
[0054] It is therefore not necessary to perform a complete upsampling or downsampling of the entire image. Both the search and the determination of image information for the extension region can be performed without corresponding upsampling and downsampling using only the aforementioned method, also known as virtual upsampling.
[0055] Accordingly, a basic method according to one aspect of the present invention, as shown in Fig. 5, first comprises the step S1 of determining a first target area 103 in the extension area 110, wherein the first target area 103 borders the image 101 and the extent of the first target area 103 perpendicular to the image boundary 102 is smaller than the extent of the extension area 110 in the section of the first target area 103. This can also be seen from Fig. 3 a) shown above.
[0056] The method then comprises the second step S2 of determining a template area 104, wherein the template area 104 lies in the image 101 and borders the first target area 103. The method then comprises the third step S3 of searching for an area 106 in the image 101 that is similar to the template area 104.
[0057] The method then comprises step S4 of determining a first source area 107 which has the same relative position to the similar area 106 as the template area 104 has to the first target area 103. The first source area 107 is again shown, for example, in Fig. 3a) discussed above.
[0058] The method then comprises the next step S5 of generating image information for the first target area 103 based on the image information of the first source area 107.
[0059] In order to now also fill the further area of the extension area 110 in Fig. 3a), a second target area 112 in the extension area 110 is then determined in step S6, wherein the second target area 113 borders on the first target area 103 perpendicular to the image boundary 102.
[0060] Then, in step S7, image information for the second target area 112 is determined based on image information from the second source area 113, wherein the second source area 113 has the same relative position to the second target area 112 as the first source area has to the first target area. This is shown, for example, in Figure 3d). This step can be performed with additional adjacent target areas if necessary, until the entire expansion area 109 is filled.
[0061] The result of the process is the expanded image shown in Figure 3d), which is supplemented by a completely filled expansion area 110 up to the expansion area boundary 109. Reference symbol
[0062] Picture
[0063] Image boundary first target area
[0064] Template area
[0065] Search area similar area first source area
[0066] Displacement vector
[0067] Expansion area limit
[0068] Expansion area first expanded image line second target area second source area second expanded image line
[0069] Luminance components
[0070] Luminance shift vector
[0071] Chrominance component
[0072] Search area boundary
[0073] Determining a first target area
[0074] Specifying a template range
[0075] Find an area in the image similar to the template area
[0076] Determining a first source area
[0077] Generating image information for the first target area
[0078] Determining a second target area
[0079] Generating image information for the second target area
Claims
Patent claims 1. An image processing method for extending an image (101) by adding an additional extension area (110), comprising the steps of: - determining (S1) a first target area (103) in the extension area (110), wherein the first target area (103) borders the image (101) and the extent of the first target area (103) perpendicular to the image boundary (102) is smaller than the extent of the extension area in the (110) section of the first target area (103); - determining (S2) a template area (104), wherein the template area (104) lies in the image (101) and borders on the first target area (103); - searching (S3) for an area in the image (101) similar to the template area (104); - determining (S4) a first source area (107) which has the same relative position to the similar area (106) as the template area (104) has to the first target area (103); - generating (S5) image information for the first target area (103) based on image information of the first source area (107); - determining (S6) a second target area (112) in the extension area (110), wherein the second target area (112) borders the first target area (103) perpendicular to the image boundary (102); - generating (S7) image information for the second target area (112) based on image information of a second source area (113), wherein the second source area (113) has the same relative position to the second target area (112) as the first source area (107) to the first target area (103).
2. Image processing method according to claim 1, wherein - generating (S5) image information for the first target area (103) based on image information of the first source area (107) is copying the image information from the first source area (107) to the first target area (103); and / or - generating (S7) image information for the second target area (112) based on image information of a second source area (113) is copying the image information from the second source area (113) to the second target area (112).
3. Image processing method according to claim 1, wherein - searching (S3) for an area in the image (101) similar to the template area (104) comprises searching for several similar areas, and - generating (S5) image information for the first target area (103) based on image information of the first source area (107) comprises averaging respective source areas of the plurality of similar areas, wherein the respective source areas of the plurality of similar areas each have the same relative position to the respective one of the plurality of similar areas as the template area (104) to the first target area (103), and wherein - searching (S3) for several similar areas includes: - determining a similarity value for each of the plurality of similar regions; - Determine the best similarity value; - Multiplying the best similarity value by a predetermined value; - Selecting the one of the plurality of similar regions whose similarity value is better than the best similarity value multiplied by the predetermined value.
4. Image processing method according to one of the preceding claims, wherein - searching (S3) for an area in the image (101) similar to the template area (104) comprises: - Specify a search area in which to search for a similar area; - Specifying a termination criterion; - Determine whether there is an area in the specified search area that meets the termination criterion; - if there is no area that meets the termination criterion, enlarge the search area; 5. Image processing method according to one of the preceding claims, wherein - searching (S3) for an area in the image (101) similar to the template area (104) comprises: - Specify a search area in which to search for a similar area; - Setting a threshold for a similarity value; - Determine whether there is an area in the specified search area with a similarity value equal to or better than the threshold; - if there is no area with a similarity value equal to or better than the threshold, increase the search area; 6. Image processing method according to one of the preceding claims, wherein - the determination of a similarity is carried out when searching for an area (106) in the image (101) similar to the template area (104) based on luminance components of the similar area (106).
7. Image processing method according to claim 6, further comprising - determining a luminance shift vector from the first source area (107) to the first target area (103), wherein - Generating (S5) image information for the first target area (103) based on image information of the first source area (107) comprises: - determining values of the luminance component for the first target area (103) based on the values of the luminance component of the first source area (107) by applying the luminance shift vector; - determining values of the chrominance components for the first target area (103) based on the values of the chrominance components of the first source area (107) using the luminance shift vector.
8. Image processing method according to claim 7, wherein - determining values of the chrominance components for the first target area (103) based on the values of the chrominance components of the first source area (107) using the luminance shift vector comprises: - Determining the values of the chrominance component by following along the luminance shift vector and then interpolating, in particular by selecting the nearest neighbor.
9. A video coding method comprising the image processing method according to any one of the preceding claims.
10. A computer program which, when executed on a computer, carries out a method according to any one of the preceding claims.