Image processing methods for image enhancement

Abstract

Image processing method for extending an image (101) by adding an additional extension area (110), comprising the steps: - Determining (S1) a first target area (103) in the extension area (110), wherein the first target area (103) borders the image (101) and the extent of the first target area (103) perpendicular to the image boundary (102) is smaller than the extent of the extension area in the (110) section of the first target area (103); - Determining (S2) a template area (104), wherein the template area (104) lies in the image (101) and borders the first target area (103); - Search (S3) for an area in the image (101) similar to the template area (104); - Determining (S4) a first source area (107) which has the same relative position to the similar area (106) as the template area (104) has to the first target area (103); - Generating (S5) image information for the first target area (103) based on image information from the first source area (107); - Determining (S6) a second target area (112) in the extension area (110), wherein the second target area (112) is adjacent to the first target area (103) perpendicular to the image boundary (102); - Generating (S7) image information for the second target area (112) based on image information from a second source area (113), wherein the second source area (113) has the same relative position to the second target area (112) as the first source area (107) has to the first target area (103).

Description

Technical field

[0001] The present invention relates to an image processing method for enhancing an image, which is used in particular in a video coding method, and to an associated computer program. State of the art

[0002] Particularly in the field of video encoding, it is necessary to extend images beyond their edges. This is required, for example, when video encoding requires accessing information at the edge of a reference image during interpretation. To achieve such an extension, the state of the art typically involves repeating the edge pixels multiple times. However, this leads to image errors, as simply repeating the edge pixels does not result in a natural representation of the edges.

[0003] EP 2 719 179 A1 concerns the filling of edge pixels for intraprediction in video coding.

[0004] DE 10 2021 121 560 A1 concerns an object image completion. Description of the invention

[0005] The present invention therefore relates in a first aspect to an image processing method for extending an image by adding an additional extension area.

[0006] Such an image processing method can be applied particularly in the field of video encoding, and especially in the padding of reference pictures or sub-pictures. An extension area is understood to be an area located on one or more sides of the image, supplementing it with an additional image area. In this context, the image in question could be, for example, the aforementioned reference picture or the aforementioned sub-picture in the context of video encoding.

[0007] In the first step of the process, an initial target area is defined within the extension area. This initial target area borders the image, and its extent perpendicular to the image boundary is smaller than the extent of the extension area within that section. Thus, a target area is defined—a region to be filled with additional image information, but which is not part of the original image. As previously described, this target area is designed such that its extent in a direction away from the image edge, i.e., perpendicular to the image boundary, is smaller than the extent of the extension area.For example, if the extension area in this section is to have a dimension of four image units, for example pixels, away from the image boundary, that is, if the image is to be extended by four pixels, then the target area defined here can have a dimension away from the boundary of only one image unit, for example one pixel. In the direction along the boundary, that is, parallel to the image boundary, the dimension can be one or more image units.

[0008] In the next step, a template area is defined. This template area lies within the image and borders the first target area. This means that the template area is directly adjacent to the target area, but within the image and therefore on the other side of the image boundary. The template area can have any defined dimensions, such as 8 x 8 image units, for example, pixels.

[0009] In the next step, an area within the image similar to the reference area is sought. Similarity can be determined using a similarity metric such as the sum of absolute differences (SAD). This means that for a large number of different areas in the image, such as all areas within a predefined search area, a sum of absolute differences can be calculated, and an area that is particularly similar to the reference area can be identified.

[0010] In a further step, a first source area is then determined, which has the same relative position to the similar area as the first target area has to the template area. As a result, a first source area is determined that is located in the same way to the similar area as the template area is to the first target area.

[0011] The image information from the first source area is then used to generate image information for the first target area based on the image information from the first source area. As will be described in more detail later, this can, for example, in its simplest form, involve transferring the image information from the first source area to the first target area.

[0012] As explained above, the first target area in an extent perpendicular to the image boundary is smaller than the extension area; that is, additional information perpendicular to the image boundary must be added in the extension area to fill the entire extension area.

[0013] In the next step, a second target area is then determined in the extension area, with the second target area adjoining the first target area perpendicular to the image boundary.

[0014] Next, image information for the second target area is determined based on image information from a second source area. This second source area is an area that has the same relative position to the second target area as the first source area has to the first target area. In other words, the displacement vector between the first source area and the first target area is equal to the displacement vector between the second source area and the second target area.

[0015] Depending on the position of the first source area relative to the image boundary, it may be necessary to fill other sections, which are parallel to the image boundary next to the first target area, by performing the aforementioned procedure before processing the second target area.

[0016] Thus, the derived displacement vector, or the derived ratio between the first source area and the first target area, can be used to generate further image information in a direction perpendicular to the image boundary.

[0017] The step of generating image information for a second target area can be repeated any number of times in order to fill further target areas in the extension area adjacent to the previously filled target areas perpendicular to the image boundary.

[0018] Thus, using the method described above, the extension area away from the image boundary can be successively filled with image information until the entire desired extension area is filled. A similar procedure can be used for the areas within the extension area along the image boundary.

[0019] This makes it possible to fill the extension area with significantly higher quality than simply repeating the edge pixels, using minimal computational effort. Furthermore, by choosing a target area that extends less far from the image boundary than the extension itself, the search area—used to find a similar area to the original—can be enlarged. This is because a larger target area would enlarge a portion of the image unusable for the search, as the source area could then extend beyond the image boundaries.

[0020] In one embodiment, generating image information for the first target area based on image information from the first source area can involve copying the image information from the first source area to the first target area. The same applies to the second target area and the second source area. Thus, in a simple embodiment, the image information is transferred from the first source area to the first target area.

[0021] In another aspect, searching for an area in the image similar to the template area can involve searching for several similar areas. Thus, not only the most similar area in the image is sought, but also a multitude of other similar areas, whereby the number of areas to be used can either be predetermined or, as will be described below, determined variably. Furthermore, in the step of generating image information for the first target area based on image information from the first source area, the process can involve averaging the respective source areas of the several similar areas, where each of the respective source areas of the several similar areas has the same relative position to its respective similar area as the template area has to the first target area.In other words, a large number of similar areas are identified, for each of which a corresponding source area is determined that has the same relative position to the respective similar area as the template area has to the first target area. Then, an average of the image information from these source areas is calculated, and this average is then used as the image information for the first target area.

[0022] Since using a fixed number of source areas could lead to the use of similar areas and associated source areas that only bear a slight resemblance to the template area, it is possible to variably determine the number of similar areas or source areas to be used. Searching for multiple similar areas involves the following steps: First, a similarity value is determined for each of the multiple areas. Then, the best similarity value is determined from these similarity values. This could be, for example, the smallest or largest value, depending on the metric used to determine similarity. This best similarity value is then multiplied by a predetermined value. The resulting value is then used as a threshold for selecting the similar areas and their associated source areas.For this purpose, those of several similar areas are selected whose similarity score is better than, or equal to, the best similarity score multiplied by the predetermined value. Depending on the similarity score and factor used, this can be a smaller or a larger value.

[0023] The selected similar areas are then used, as described above, to determine the respective source areas and subsequently to average the image information of the respective source areas and to generate the image information for the first target area. Furthermore, the maximum number of similar areas to be used can be limited by a threshold value to restrict the computational effort.

[0024] The quality of the image information in the extension area is thus further improved, since no source areas are used for averaging whose associated similar areas show only a low similarity to the template area.

[0025] Furthermore, in one aspect, the search for similar areas can be accelerated by defining a corresponding search area variably.

[0026] To search for an area in the image similar to the template area, the following additional steps are involved: First, a search area is defined within the image, in which a similar area is sought. This could, for example, be an area in the image at a certain distance from the template area.

[0027] Then, a termination criterion is defined. This could, for example, be a threshold value for a similarity score.

[0028] The process then determines whether an area within the defined search range meets the termination criterion. For example, it checks whether the search range contains an area with a similarity value equal to or better than the threshold previously defined as the termination criterion. If so, the search is terminated and the procedure described above is carried out. However, if no area meets the termination criterion, the search range is enlarged, for example, by increasing the distance to the template area, and a new search is performed within the now larger search range.

[0029] Thus, by selecting a smaller search area initially and successively increasing this search area, the finding of a similar area can be accelerated and the computational intensity of the procedure reduced.

[0030] According to another aspect, the determination of similarity when searching for an area of ​​the image similar to the template area can be carried out based on the luminance component of the similar area and the search area.

[0031] Performing searches on both the luminance and chrominance components would result in increased computational effort. Furthermore, images often exhibit chroma subsampling. For example, digital images in the JPEG standard or digital video material in the MPEG standard often use 4:2:0 sampling, where only four luminance values ​​have a corresponding set of chrominance components. To perform a search based on both the luminance and chrominance components, one would first have to increase the sampling rate (upsampling) to obtain a chrominance value for each luminance value. After this process, a subsequent downsampling would be necessary to generate the chrominance components again. This results in significant computational overhead.Studies have shown that searching solely based on luminance components can also be used to find similar areas without loss of quality. Accordingly, determining similarity when searching for an area in the image similar to the reference area based on the luminance components of that area can be understood as searching based on luminance components without considering chrominance components.

[0032] To determine the target region of the chrominance components, in addition to the target region of the luminance component, without performing upsampling and downsampling, a luminance shift vector from the first source region to the first target region for the luminance component can first be determined. This means a luminance shift vector is determined that represents the shift between the source region and the target region of the luminance component. Then, by simply applying this luminance shift vector, the target region of the luminance component can be derived based on the first source region of the luminance component.

[0033] The chrominance components for the first target region can also be derived based on the chrominance components of the first source region using the luminance shift vector.

[0034] In one aspect, for example, the luminance shift vector can be followed in reverse from the target area to the source area, and then, using an interpolation method, such as a nearest neighbor method, the source values ​​of the chrominance component in the vicinity of the source area of ​​the luminance component can be selected, and based on this, the chrominance component for the target area can be determined.

[0035] As previously explained, this method is particularly useful in the field of video coding, and especially in the padding of reference pictures or sub-pictures. Accordingly, another aspect encompasses a video coding method that includes one of the aforementioned image processing methods.

[0036] Another aspect concerns a computer program which, when executed on a computer, performs a method according to one of the preceding claims.

[0037] The previously described process steps, which are not directly dependent on each other, are not restricted to the presented execution sequence in one aspect, but can also be executed in a different order according to another aspect. However, according to another aspect, they can also be executed in the order defined above. Brief description of the characters Fig. Figure 1 schematically shows how to fill a target area by copying a source area. Fig. Figure 2 schematically shows how to fill a target area by averaging several source areas. Fig. Figure 3 schematically shows the successive creation of an extension area. Fig. Figure 4 shows the derivation of a chrominance component based on a luminance vector. Fig. Figure 5 shows a procedure based on one aspect. Detailed description of embodiments

[0038] In Fig. 1. First, the general procedure for extending the image 101 by copying the information from a source area 107 is described. Here, the Fig. 1. Image 101 is extracted, which is to be extended by adding further information at the edge, and thus beyond the image boundary 102. If, for example, image information is to be added in the extension area above the image boundary 102, a target area 103 is first defined, which borders the image boundary 102. Then, a template area 104 adjacent to this target area 103 is defined, which lies within image 101 and borders the target area 103 across the image boundary 102. Clearly, the target area 103 and the template area 104 have a specific relationship to each other. Then, as in the Fig. 1b) shows that a similar area 106 to the template area 104 is sought within the search area 105. As shown, Fig. 1a) and Fig. As can be seen from 1b), the search area 105 is not directly adjacent to the image boundary 102, but is spaced away from the image boundary 102 according to the extent of the target area 103. This is the case because, if a similar area were searched for up to the image boundary 102, the corresponding source area 107 would lie outside the image 101 and thus could not contain any information.

[0039] Accordingly, the search area 105 is now searched for areas that exhibit a high similarity to the template area 104. In one embodiment, for example, a similarity metric such as the sum of absolute differences (SAD) can be used. If an area with a particularly high similarity, such as the similar area 106 shown here, is found, a source area 107 is then determined, which lies in the same spatial relation to the similar area as the target area 103 lies to the template area 104. This results in the displacement vector 108, which is the Fig. As can be seen in section 1b). The image information from the source area 107 is then transferred to the target area 103. Thus, the image 101 has been extended to include the target area 103. This can be done accordingly along the image boundary 102 to extend the image 101 along its entire length, or around its entire periphery, by a corresponding extension area.

[0040] Instead of simply copying the information from a source area, it is possible, as in the Fig. As shown in 2, averaging of several source areas can also be performed. First, as in 2, the following steps are taken: Fig. 2a) shows that the target area 103 and the template area 104 are determined. Then, a search is performed in the search area 105 for several similar areas, each of which is labeled with 106 in Fig. The process is carried out for areas marked 2. The number of similar areas can either be predetermined, or those areas can be selected as similar areas that are not significantly worse than the most similar of the similar areas. For example, a similarity metric can be determined for each area in the search area 105, this can then be multiplied by a specific factor, and the resulting value can be used as a threshold. All areas that have a similarity value below this threshold are then selected as similar areas 106. A corresponding source area 107 is then determined for each of the similar areas. The similar area, in turn, has the same spatial relationship to the corresponding source area as template area 104 has to target area 103.The image information from the multitude of source areas 107 can then be averaged and, based on this averaging, the image information for the target area 103 can be derived.

[0041] With reference to Fig. Section 3 will now describe the gradual development of an expansion area.

[0042] Should an expansion area of ​​110 be built in Fig. 3a), which extends between an image boundary 102 and an extension area boundary 109, is to be filled with image information. As explained above, it is advantageous to choose the extent of a target area in the direction perpendicular to an image boundary 102 to be as small as possible, so that the area of ​​the image 101, which is bounded by the image boundary 102 and is not usable for a search, is kept as small as possible, and thus the search area 105, which is bounded by the search area boundary 118, extends as far as possible to the edge of the image 101, i.e., to the image boundary 102. Therefore, in a first step of the present example, a first target area 103 is defined, which in this example is four units wide but only one unit deep, perpendicular to the image boundary 102.This results in a search area 105, which is only one image unit, for example one pixel, away from the image boundary 102 and can therefore be made as large as possible. The search area boundary 118 and the image boundary 102 are therefore only one image unit, for example one pixel, apart.

[0043] In contrast, if the full depth of four image units between the image boundary 102 and the extension area boundary 109 were chosen as the size of the target area, the search area 105 would be significantly restricted, since the distance of the search area 105 in image 101 to the image boundary 102 would also have to be four image units.

[0044] Then, as described above, a template area 104 is determined for the target area 103, and then a similar area is sought with the help of which a corresponding source area 107 is determined.

[0045] To simplify the drawings, the similar area is shown in the Fig. 3a) not shown, but only the resulting source area 107.

[0046] Then, using the image information in the source area 107, the image information in the target area 103 can be supplemented. If this is also carried out for the areas horizontally adjacent along the image boundary 102 using a corresponding method, a first extended image line 111 results. Then, the displacement vector 108, already determined as a displacement vector 108 between the first source area 107 and the first target area 103, can be used as described in Fig. The process shown in Figure 3b) can be applied again to populate a second target area 112, located below the first target area 103, with image information from a second source area 113. Here, the second source area 113 is selected as an area that has the same spatial relationship to the second target area 112 as the first source area 107 has to the first target area 103. In other words, the displacement vector 108 is applied to the second target area 112.

[0047] Will this now also apply to the horizontal direction in Fig. 3b) adjacent areas of the second target area 112, the second extended image line 114 results in Fig. 3c). This can now be carried out until the entire extension area 110 is filled up to the extension area boundary 109, as in Fig. 3d). The final result is thus shown in Fig. 3d) shown extended image 101, which can be used for the purposes described above in the field of video coding.

[0048] In Fig. Figure 4 shows an example of the application in the area of ​​an image with 4:2:0 chroma subsampling. First, the procedure described above can be carried out such that the search for similar areas is performed solely based on the luminance components 115. This then yields the luminance shift vectors 116 between the source and target areas, which represent the shift of the luminance values ​​between these two areas. If the additional chrominance component is also to be determined for a corresponding target area, the shift vector can be followed along the luminance shift vector 116 to the corresponding section in the source area, represented by a dashed arrow in Figure 4. Fig. 4, and from this, for example using a nearest neighbor method, the corresponding value of the chrominance component, here the value 117, can be determined, again represented by a dashed arrow.

[0049] Therefore, it is not necessary to perform a complete upsampling or downsampling of the entire image. Both the search and the determination of the image information for the extension area can be carried out without corresponding upsampling and downsampling, using only the aforementioned method, also described as virtual upsampling.

[0050] Accordingly, a basic method according to one aspect of the present invention comprises, as in Fig. Figure 5 shows, first step S1 of determining a first target area 103 in the extension area 110, where the first target area 103 borders the image 101 and the extent of the first target area 103 perpendicular to the image boundary 102 is smaller than the extent of the extension area 110 in the section of the first target area 103. This is also the one shown above. Fig. 3 a) can be found here.

[0051] The procedure then includes the second step S2 of determining a template area 104, wherein the template area 104 is located in the image 101 and borders the first target area 103.

[0052] The procedure then includes the third step S3 of searching for an area 106 similar to the template area 104 in the image 101.

[0053] The procedure then includes step S4 of determining a first source area 107, which has the same relative position to the similar area 106 as the template area 104 has to the first target area 103. The first source area 107 is again, for example, in the area discussed above. Fig. 3a) shown.

[0054] The procedure then includes the next step S5 of generating image information for the first target area 103 based on the image information of the first source area 107.

[0055] To now also include the further area of ​​expansion area 110 in Fig. 3a) To fill in, a second target area 112 is then determined in step S6 in the extension area 110, whereby the second target area 113 borders the first target area 103 perpendicular to the image boundary 102.

[0056] Then, in step S7, image information for the second target area 112 is determined based on image information from the second source area 113, where the second source area 113 has the same relative position to the second target area 112 as the first source area has to the first target area. This is, for example, Fig. 3d). This step can be repeated with other adjacent target areas until the entire extension area 109 is filled.

[0057] The result of the process is then obtained in Fig. 3d) shown extended image, which is supplemented by a fully filled extension area 110 up to the extension area boundary 109. Reference sign 101 pictures 102 Image boundary 103 first target area 104 Template area 105 Search area 106 similar areas 107 first source area 108 Displacement vector 109 Extension area boundary 110 Extension area 111 first extended image line 112 second target area 113 second source area 114 second extended image line 115 luminance components 116 Luminance displacement vector 117 Chrominance component 118 Search area limit S1 Determining a first target area S2 Defining a template area S3 Search for an area in the image similar to the template area S4 Determining a first source area S5 Generating image information for the first target area S6 Determining a second target area S7 Generating image information for the second target area

Claims

[1] Image processing method for extending an image (101) by adding an additional extension area (110), comprising the steps: - Determining (S1) a first target area (103) in the extension area (110), wherein the first target area (103) borders the image (101) and the extent of the first target area (103) perpendicular to the image boundary (102) is smaller than the extent of the extension area in the (110) section of the first target area (103); - Determining (S2) a template area (104), wherein the template area (104) lies in the image (101) and borders the first target area (103); - Search (S3) for an area in the image (101) similar to the template area (104); - Determining (S4) a first source area (107) which has the same relative position to the similar area (106) as the template area (104) has to the first target area (103); - Generating (S5) image information for the first target area (103) based on image information from the first source area (107); - Determining (S6) a second target area (112) in the extension area (110), wherein the second target area (112) is adjacent to the first target area (103) perpendicular to the image boundary (102); - Generating (S7) image information for the second target area (112) based on image information from a second source area (113), wherein the second source area (113) has the same relative position to the second target area (112) as the first source area (107) has to the first target area (103). [2] Image processing method according to claim 1, wherein - Generating (S5) image information for the first target area (103) based on image information from the first source area (107) is copying the image information from the first source area (107) to the first target area (103); and / or - Generating (S7) image information for the second target area (112) based on image information from a second source area (113) is the copying of image information from the second source area (113) to the second target area (112). [3] Image processing method according to claim 1, wherein - searching (S3) for an area in the image (101) similar to the template area (104) includes searching for several similar areas, and - Generating (S5) image information for the first target area (103) based on image information of the first source area (107) comprises averaging the respective source areas of the multiple similar areas, wherein the respective source areas of the multiple similar areas each have the same relative position to the respective of the multiple similar areas as the template area (104) has to the first target area (103), and wherein - the search (S3) includes multiple similar areas: - Determining a similarity score for each of the several similar areas; - Determining the best similarity value; - Multiplying the best similarity score by a predetermined value; - Selecting the one of several similar areas whose similarity value is better than the best similarity value multiplied by the predetermined value. [4] Image processing method according to any of the preceding claims, wherein - searching (S3) for an area in the image (101) similar to the template area (104) includes: - Defining a search range in which to search for a similar area; - Defining a termination criterion; - Determine whether there is an area within the specified search range that meets the termination criterion; - if no area meets the abort criterion, increase the search area; [5] Image processing method according to any of the preceding claims, wherein - searching (S3) for an area in the image (101) similar to the template area (104) includes: - Defining a search range in which to search for a similar area; - Setting a threshold for a similarity value; - Determine whether, within the specified search area, there is a range with a similarity value equal to or better than the threshold; - if no area with a similarity value equal to or better than the threshold exists, increase the search area; [6] Image processing method according to any of the preceding claims, wherein - the determination of a similarity is carried out when searching for an area (106) similar to the template area (104) in the image (101) based on luminance components of the similar area (106). [7] Image processing method according to claim 6, further comprising - Determining a luminance shift vector from the first source region (107) to the first target region (103), wherein - Generating (S5) image information for the first target area (103) based on image information from the first source area (107) includes: - Determining values ​​of the luminance component for the first target area (103) based on the values ​​of the luminance component of the first source area (107) by applying the luminance shift vector; - Determining values ​​of the chrominance components for the first target region (103) based on the values ​​of the chrominance components of the first source region (107) using the luminance shift vector. [8] Image processing method according to claim 7, wherein - Determining the values ​​of the chrominance components for the first target region (103) based on the values ​​of the chrominance components of the first source region (107) using the luminance shift vector includes: - Determining the values ​​of the chrominance component by following along the luminance shift vector and then interpolating, in particular by selecting the nearest neighbor. [9] Video coding method comprising the image processing method according to any of the preceding claims. [10] Computer program which, when executed on a computer, performs a method according to any of the preceding claims.

Images