Image inpainting method and device

Abstract

An image filling method and device. The method comprises: translating foreground objects in a reference image according to a preset direction to obtain a target image containing a blank area, each blank pixel point in the blank area is in a first pixel block corresponding to the blank pixel point, and the similarity of any blank pixel point is used to represent the similarity between the first pixel block and a second pixel block corresponding to the first pixel block in the target image; and each blank pixel point is iterated to fill, which comprises: determining the maximum similarity in the similarity of any blank pixel point and at least one adjacent blank pixel point of the blank pixel point, and searching for an optimal pixel block in a preset range of the corresponding second pixel block; and taking the color value of an optimal pixel point in the optimal pixel block and at the same position as the color value of the any blank pixel point. The method can fill the blank area generated after the movement of the foreground objects, and improve the display effect of the target image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image filling method and apparatus. Background Technology

[0002] In related technologies, new images can be generated from existing images. For example, in a binocular vision scenario, any image can be used as a reference image observed by one eye of the observer, and the target image observed by the other eye can be generated by translating the foreground objects in that image.

[0003] However, since the depth of field (or depth) of each foreground object in the reference image is often different, the translation distance of different foreground objects is also different. This results in the translation of the foreground objects into blank areas without color values ​​in the target image, that is, there is a screen break in the target image, which affects the display effect of the target image. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide an image filling method and apparatus to address the shortcomings of related technologies.

[0005] According to a first aspect of the present disclosure, an image filling method is provided, comprising:

[0006] The foreground object in the reference image is translated in a preset direction to obtain the target image. The target image includes a blank area generated after the foreground object is moved. Each blank pixel in the blank area is located in its corresponding first pixel block. The similarity of any blank pixel is used to characterize the degree of similarity between the first pixel block where the blank pixel is located and the second pixel block corresponding to the first pixel block in the target image.

[0007] Iterate through and fill in each of the blank pixels, wherein filling in any blank pixel includes:

[0008] Determine the maximum similarity among the similarities of any blank pixel and at least one adjacent blank pixel, and find the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block in which the blank pixel is located.

[0009] The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

[0010] According to a second aspect of the present disclosure, an image filling apparatus is provided, the apparatus comprising one or more processors configured to:

[0011] The foreground object in the reference image is translated in a preset direction to obtain the target image. The target image includes a blank area generated after the foreground object is moved. Each blank pixel in the blank area is located in its corresponding first pixel block. The similarity of any blank pixel is used to characterize the degree of similarity between the first pixel block where the blank pixel is located and the second pixel block corresponding to the first pixel block in the target image.

[0012] Iterate through and fill in each of the blank pixels, wherein filling in any blank pixel includes:

[0013] Determine the maximum similarity among the similarities of any blank pixel and at least one adjacent blank pixel, and find the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block in which the blank pixel is located.

[0014] The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the image filling method described in the first aspect above.

[0016] According to a fourth aspect of the present disclosure, a non-transient computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the image filling method described in the first aspect above.

[0017] According to embodiments of this disclosure, a target image is obtained by translating a foreground object in a reference image in a preset direction. This image includes blank areas created after the foreground object is moved. To this end, a first pixel block corresponding to each blank pixel in the blank area is determined. The similarity of any blank pixel is used to characterize the degree of similarity between the first pixel block containing the blank pixel and the second pixel block corresponding to the first pixel block in the target image. Based on the aforementioned pixel blocks and similarity, each blank pixel is filled. Filling any blank pixel includes: determining the maximum similarity among the similarities of any blank pixel and at least one adjacent blank pixel; and searching for an optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block containing the blank pixel. Then, the color value of the optimal pixel in the optimal pixel block is used as the color value of the blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of the blank pixel in its first pixel block, and the optimal pixel block is the same size as the first pixel block.

[0018] It is understood that the blank areas in the target image are the areas to be filled. This solution utilizes the redundancy of the target image itself, introducing concepts such as first pixel blocks, second pixel blocks, and similarity between pixel blocks to fill blank pixels. When filling any blank pixel in a blank area, the maximum similarity is first determined among the similarities between any blank pixel and at least one adjacent blank pixel. Then, the optimal pixel block is searched within a preset range of the second pixel block corresponding to the maximum similarity, thereby ensuring that the optimal pixel block has the maximum similarity with the first pixel block where the blank pixel is located in the preset area, that is, the optimal pixel block is closest to the first pixel block. Furthermore, because the optimal pixel block and the first pixel block are the same size, and the position of the optimal pixel in the optimal pixel block is the same as the position of the blank pixel in its first pixel block, it is ensured that the optimal pixel is the pixel whose pixel environment in the target image is closest to that of the blank pixel. Thus, by using the color value of the optimal pixel in the optimal pixel block as the color value of the blank pixel, the color value filling of the blank pixel is achieved.

[0019] This method not only fills in each blank pixel in the blank area, but also makes the color value after filling as close as possible to the color value of the surrounding pixels. This ensures that the color value of the filled blank area is not too abrupt, achieving a smooth transition of color values ​​and resulting in a better display effect for the filled target image.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating the imaging principle of binocular vision according to an embodiment of the present disclosure.

[0023] Figure 2 This is a schematic diagram illustrating the parallax effect in a binocular vision scene according to an embodiment of the present disclosure.

[0024] Figure 3 This is a flowchart illustrating an image filling method according to an embodiment of the present disclosure.

[0025] Figure 4 This is a schematic diagram of a blank area of ​​a target image according to an embodiment of the present disclosure.

[0026] Figure 5 This is a schematic diagram illustrating a pixel block initialization effect according to an embodiment of the present disclosure.

[0027] Figure 6 This is a schematic diagram illustrating a similarity comparison process according to an embodiment of the present disclosure.

[0028] Figure 7 This is a schematic diagram illustrating an optimal pixel block search process according to an embodiment of the present disclosure.

[0029] Figure 8 This is a schematic diagram illustrating a filling effect according to an embodiment of the present disclosure.

[0030] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0031] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0032] Because of the interpupillary distance between the two eyes, the angle of any observed object relative to the left and right eyes is usually not the same, thus enabling the two eyes to form a stereoscopic image.

[0033] Figure 1 This is a schematic diagram illustrating the imaging principle of binocular vision according to an embodiment of this disclosure. Figure 1 As shown, relative to human eyes, the observed objects, from near to far (i.e., from shallow to large depth of field), are successively triangles, circles, and parallelograms (the distance difference between any object and the left and right eyes is negligible). The left-eye view and the right-eye view are shown below. Figure 1 As shown, both the left-eye view and the right-eye view contain the three objects mentioned above.

[0034] Figure 2 This is a schematic diagram illustrating the parallax effect in a binocular vision scene according to an embodiment of this disclosure. For example... Figure 2 As shown, the relative positions of objects are not the same in the left-eye view and the right-eye view. This is because the angles at which each object is located relative to the left and right eyes are different, respectively. Furthermore, due to... Figure 2 It is evident that the closer an object is to the human eye (i.e., the smaller the depth of field), the greater the difference in the specific value between the left and right eye views. For example, the distance difference of a triangle is greater than that of a circle.

[0035] Based on the above principle, any view can be used as a reference view formed by observation from one eye. The object in that view is translated to obtain a new view, which can be used as the target view formed by observation from the other eye. Thus, a new image can be generated from a planar image. These two images can then be used to construct a stereoscopic image under binocular vision.

[0036] In related technologies, the offset distance of each object in the reference image is usually calculated directly based on the object's depth of field, and each object is translated according to this distance to obtain the target image. However, because the depth of field of each object is different, their offset distances are not the same. That is, objects with a smaller depth of field have a larger offset distance, and objects with a larger depth of field have a smaller offset distance. Therefore, discontinuities may appear between the translated objects, that is, blank areas may appear between different objects or between objects and the background, resulting in poor display effect of the target image.

[0037] To address this issue, this disclosure proposes an image filling method to fill in pixel blocks appearing in a target image, thereby improving the display effect of the target image. The following detailed description of this solution, in conjunction with embodiments and related accompanying drawings, further illustrates the method.

[0038] Figure 3 This is a flowchart illustrating an image filling method according to an embodiment of the present disclosure. Figure 3As shown, the method may include the following steps 302-304, wherein step 304 includes sub-steps 3042-3044.

[0039] Step 302: The foreground object in the reference image is translated in a preset direction to obtain the target image. The target image includes the blank area generated after the foreground object is moved. Each pixel in the target image is located in its corresponding first pixel block. The similarity of any pixel is used to characterize the degree of similarity between the first pixel block where the pixel is located and the second pixel block corresponding to the first pixel block in the target image.

[0040] The image inpainting method described in this disclosure can be applied to an image processing device, which may be a server-side device or a terminal device. The server-side device may be a physical server containing an independent host, or a virtual server or cloud server hosted in a host cluster. The terminal device may be a mobile phone, or a tablet, laptop, PDA (Personal Digital Assistants), wearable device (such as smart glasses, smartwatches, etc.), VR (Virtual Reality) device, AR (Augmented Reality) device, etc., and one or more embodiments of this disclosure do not limit this to these categories.

[0041] Furthermore, the color value of any pixel in the embodiments of this disclosure is the value of that pixel under a certain color model, such as RGB value, grayscale value, CIE-XYZ value, etc., and the embodiments of this disclosure do not limit it.

[0042] In the embodiments described in this disclosure, the reference image can be any planar image, which may contain at least one foreground object. The foreground object can be any object in the reference image that needs to be translated. The image region formed by all the pixels of the foreground object is the foreground region. Translating the foreground object essentially translates each pixel within the foreground region. Therefore, the foreground object exists in the translated target image, meaning a corresponding foreground region exists in both images; only the positions of the foreground regions differ between the two images.

[0043] In one embodiment, the image processing device can determine the movement distance of each foreground object based on its depth value (i.e., depth of field) in the reference image. The movement distance of any foreground object can be negatively correlated with its depth value. Then, each foreground object is moved in the same preset direction according to its respective movement distance. For example, the image processing device can acquire the depth value of each foreground object in the reference image and then calculate the movement distance of each foreground object based on a preset negative correlation between the depth value and the movement distance. Inevitably, blank areas will exist in the target image obtained after this movement; these blank areas are the areas to be filled in by the present invention.

[0044] The image processing device can use a pre-trained monocular depth estimation model to perform monocular depth estimation on the reference image, calculating the depth value of each pixel in the image based on the color value of each pixel in the reference image. For example, to meet the accuracy or speed requirements of the depth estimation process, any monocular depth estimation module such as the MiDaSNet network can be used for depth estimation. Of course, in the implementation of the scheme, other monocular depth estimation models can also be selected to calculate the depth value of each pixel in the reference image according to the specific situation, and this embodiment does not limit this.

[0045] Specifically, the depth value of the foreground object can be the depth value of each pixel in the foreground region. The image processing device can then calculate the movement distance of each pixel and translate each pixel in the foreground region accordingly. This method allows for precise translation of each pixel in the reference image, resulting in a more refined and realistic display of the foreground object in the translated target image. Alternatively, the image processing device can perform edge segmentation on each foreground object in the reference image, then calculate the average depth of all pixels in the foreground region of each foreground object, and calculate the overall movement distance of each foreground object based on this average. This allows for the simultaneous translation of all pixels in the foreground region corresponding to each foreground object according to the overall movement distance (pixels within the same foreground region are translated by the same distance). This method reduces the computational workload for large-scale movement distances and improves the efficiency of target image generation.

[0046] Furthermore, during the calculation of the movement distance based on the depth value, the image processing device can determine the user's interpupillary distance (IPD). For example, if the image processing device is a terminal device, it can acquire the user's IPD through hardware such as an integrated camera, or it can obtain the IPD input by the user. If the image processing device is a client device, it can receive the IPD uploaded by the corresponding client device after being acquired in the aforementioned manner, or it can use a pre-set default IPD; this embodiment does not limit this. The IPD can affect the maximum pixel offset rate (maxShiftRatio) during object translation. For example, when the resolution of the reference view is width*height, the maximum pixel offset value corresponding to horizontal translation (left or right) is approximately maxShiftRatio*width. Therefore, by setting an appropriate maximum pixel offset rate (maxShiftRatio), the display effect of the target image can be made closer to the view actually observed by the user's eyes, which helps to improve the user experience.

[0047] The target image obtained by translating the foreground object in the aforementioned manner may include a background region distinct from both the blank and foreground regions. Each pixel in this background region is a background pixel in the target image. During the generation of the target image from the reference image, the pixels in the background region may not be translated; that is, the reference and target images may have the same background region. It is understood that each pixel in both the foreground and background regions of the target image is a pixel present in the reference image, and therefore each pixel has a corresponding color value. To distinguish them from the blank areas to be filled, the foreground and background regions can be collectively referred to as non-blank regions; that is, the target image is composed of both blank and non-blank regions.

[0048] Taking the reference image before translation as the right-eye view and the target image generated after translation as the left-eye view as an example, based on the positional relationship between the left and right eyes, it can be determined that during the generation of the target image, each foreground object in the reference image is translated to the right. The resulting target image is as follows: Figure 4 As shown, the items being translated are the individual cookies on the plate and the items placed on top of them. Due to the different depths of field of the different items, blank areas are generated to the left of the translated objects in the reference image, such as black block-shaped blank areas 401 and black strip-shaped blank areas 402. The purpose of the solution described in this disclosure is to fill in these blank areas. Figure 4 The blank areas shown.

[0049] After identifying the blank areas in the target image, the image processing device can begin filling in these blank areas. The image processing device can first initialize each pixel in the target image, that is, assign each pixel its own first pixel block and determine the corresponding second pixel block for each first pixel block.

[0050] In one embodiment, the image processing device can use a pixel block of a preset size containing any pixel as the first pixel block for that pixel. The size of the first pixel blocks for all pixels in the target image can be the same; this embodiment does not limit the size of the first pixel block. To simplify subsequent calculations, any pixel can be positioned at the center of its own first pixel block, meaning a pixel block of a preset size centered on that pixel can be used as the first pixel block for that pixel. For example, if the first pixel block for any pixel in the target image is a 3x3 grid, the pixel can be the center point of the first pixel block, which will not be elaborated further. Of course, the first pixel block containing pixels near the image edge may have some missing pixels. In this case, subsequent calculations can treat the pixels present in the first pixel block as present, without adversely affecting the final calculation result. Figure 4 As shown, the first pixel block 403 is the first pixel block where a blank pixel is located, the first pixel block 404 is the first pixel block where a background pixel is located in the background area, and the first pixel block 405 is the first pixel block where a foreground pixel is located in the foreground area.

[0051] In another embodiment, the image processing device can assign a corresponding second pixel block to the first pixel block where each pixel is located. For example, the image processing device can randomly assign a corresponding second pixel block to the first pixel block in a preset area of ​​the target image. It should be noted that the first pixel block where any pixel is located and the second pixel block corresponding to the first pixel block (hereinafter referred to as the first pixel block and the second pixel block corresponding to any pixel) can be the same size, such as 3*3 grids. This embodiment of the present disclosure does not limit this.

[0052] The preset region can be the entire region of the target image, meaning that any first pixel block is randomly assigned a corresponding second pixel block within the entire region of the target image. Alternatively, the preset region can be a non-blank region in the target image that is distinct from the blank region. In this case, all pixels in any second pixel block can belong to the non-blank region, or some pixels in the second pixel block can belong to the blank region, etc., which will not be elaborated further. This ensures that the first pixel block containing each pixel in the target image contains at least one non-blank pixel, thereby ensuring that the subsequent similarity calculation results have high discriminative power and minimizing the selection of blank pixels as the optimal pixel corresponding to any blank pixel. Furthermore, considering that foreground objects often occlude the background, the background should likely be displayed after translating the foreground object. Therefore, the preset region can also be set to the background region in the non-blank region that is distinct from the foreground region, to maximize the accuracy of finding the optimal pixel block and thus improve image filling efficiency.

[0053] like Figure 2 As shown, for any three first pixel blocks in the target image, the image processing device can determine the corresponding second pixel blocks respectively. For example, first pixel block 501 corresponds to second pixel block 504, first pixel block 502 to the left of first pixel block 501 corresponds to second pixel block 505, and first pixel block 503 above first pixel block 501 corresponds to second pixel block 506.

[0054] For any pixel point, the first pixel block and the second pixel block determined in the aforementioned manner, the image processing device can calculate the similarity between the first pixel block and the second pixel block as the similarity of that pixel point. For example, the image processing device can calculate the similarity by using the depth values ​​of the pixels contained in the first pixel block and the learned perceptual image patch similarity in the second pixel block, respectively. The depth values ​​of each pixel in the second pixel block can be estimated using the aforementioned monocular depth estimation model. The learned perceptual image patch similarity is LPIPS (Learned Perceptual Image Patch Similarity), or perceptual loss. The specific calculation method of LPIPS can be found in the relevant art and will not be elaborated further.

[0055] The following explains how similarity is calculated: Assume that peach(x,y) represents any pixel block with center coordinates (x,y). The coordinates of the center point are: For any pixel block, the similarity between the two pixel blocks is... The calculation formula can be:

[0056]

[0057] in, The depth attenuation coefficient is 0 < α < β. It should also be noted that the depth values ​​described in the above formula are expressed in grayscale form; the smaller the depth value of a pixel, the farther away that pixel is from the camera.

[0058] Represents peach(x,y) and The difference between the average depths of each of the individual pixels. Clearly, if This indicates that peach(x,y) is more than Closer to the camera; conversely, if This indicates that peach(x,y) is more than It's further away from the camera.

[0059] If we take the first pixel block and the second pixel block corresponding to any pixel point as the aforementioned peac(hx,y) and The similarity (first pixel block, second pixel block) between the two pixel blocks can be calculated using the above formula, and this value can be used as the similarity of any pixel. After calculating the similarity of each pixel in the target image in the above manner, the image processing device completes the initialization process and can then begin traversing and filling in the blank pixels in the blank areas.

[0060] Step 304: Iterate through and fill in each blank pixel in the blank area.

[0061] As mentioned earlier, during the initialization phase, the image processing device determines the corresponding first pixel block and second pixel block for each pixel in the target image. Based on this, the image processing device can iterate and fill in each blank pixel in the blank area. It should be noted that the image processing device can perform the filling process on each blank pixel in the blank area only once, and use the target image obtained after the iteration as the filled image. Of course, the filling effect of only performing the filling process once may still need improvement, so the image processing device can also perform multiple iterations on all blank pixels, that is, the traversal process of each blank pixel is executed iteratively multiple times.

[0062] In any iteration, the image processing device may only traverse each blank pixel; or, to ensure that the similarity relationship between pixel blocks can gradually be passed to blank pixels deep within the blank area, so that the color value of each blank pixel in the blank area gradually approaches the theoretical optimal value as the number of iterations increases, the image processing device may also traverse all pixels in the target image in each iteration: for blank pixels, not only is the mapping relationship between the first pixel block and the second pixel block updated according to the optimal pixel block, but the pixel value of the blank pixel is also updated using the pixel value of the optimal pixel in the optimal pixel block; while for non-blank similarity, since no filling is required, it is only necessary to update the mapping relationship between its first pixel block and the second pixel block using the optimal pixel block. The specific iteration process can be found in the description of the embodiments below, and will not be repeated here. It should be noted that in the case of multiple iterations, except for the first iteration, the pixel value of any blank similarity traversed in the subsequent iterations may not be zero or a preset value such as 255, but may have been updated in the previous iteration. However, in order to illustrate the scheme and avoid contradictions, the pixels in the blank area of ​​the newly generated target image are always recorded as blank pixels.

[0063] The image processing device can start by filling in blank pixels located at the edge of a blank area, sequentially traversing and filling each blank pixel in the blank area. For example, if the target image includes multiple independent (or isolated, i.e., not all blank pixels are adjacent) blank areas, the image processing device can, after filling in the blank pixels in one independent blank area, directly jump to the first blank pixel in the next independent blank area according to a preset traversal order, and begin traversing each blank pixel in that independent blank area. This method ensures the continuity of the pixel traversal process, so that when any blank pixel is traversed, its first pixel block contains at least one non-blank pixel.

[0064] Furthermore, as mentioned earlier, since the foreground object being translated usually occludes the background or other foreground objects before translation (i.e., in the reference image), the blank areas created by translating this object should theoretically be filled with the color values ​​of the background or other foreground object pixels to achieve a better display effect after filling. Therefore, the image processing device can determine the corresponding traversal order based on the translation direction of the foreground object (i.e., the preset direction) to avoid using the color values ​​of a foreground object's own pixels to fill blank pixels. For example, when the translation direction is from right to left, the image processing device can traverse each blank pixel sequentially from top to bottom and from left to right; while when the translation direction is from left to right, the image processing device can traverse each blank pixel sequentially from bottom to top and from right to left, and so on.

[0065] During any of these traversal filling processes, the image processing device may fill any blank pixel by the following sub-steps 3042-3044.

[0066] Sub-step 3042: Determine the maximum similarity among the similarities of any blank pixel and at least one neighboring pixel, and search for the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block where the blank pixel is located.

[0067] For any blank pixel, the image processing device can first determine at least one pixel adjacent to it. It should be noted that for the concept of "pixels adjacent to any blank pixel," a minimum distance (or step size) k can be preset to determine the pixels "adjacent" to any blank pixel based on this minimum distance k. The distance between any blank pixel and any adjacent pixel (in a straight line) is k-1 pixels. For example, when k=1, at least one of the four adjacent pixels in the four directions (up, down, left, and right) of any blank pixel can be selected; while when k=3, pixels two pixels away from the blank pixel can be selected. Further details will be provided below using k=1 as an example.

[0068] Similar to the aforementioned traversal order, considering that the maximum similarity belongs to any pixel among at least one pixel adjacent to any blank pixel, and that the optimal pixel for filling any blank pixel is ultimately determined from a preset range of the second pixel block corresponding to the maximum similarity value, the image processing device can also select at least one adjacent pixel according to the translation direction of the foreground object (i.e., the preset direction) to avoid using the color value of the foreground pixel of the foreground object corresponding to the blank pixel (i.e., the blank pixel was created by moving the object) to fill the blank pixel. For example, when the preset direction is from left to right, any pixel adjacent to the blank pixel can be selected from above or to the left of the blank pixel; while when the preset direction is from right to left, any pixel adjacent to the blank pixel can be selected from below or to the right of the blank pixel.

[0069] like Figure 5 As shown, if any blank pixel corresponds to the first pixel block 501, then when k=1, the pixels to the left and above the blank pixel can be selected as its two adjacent pixels (of course, either of these two pixels can be a blank pixel or a non-blank pixel). Let's assume that its left adjacent pixel corresponds to the first pixel block 502 and the second pixel block 505, and its upper adjacent pixel corresponds to the first pixel block 503 and the second pixel block 506.

[0070] Using the above method, when at least one pixel adjacent to any blank pixel is determined, the image processing device can determine the maximum similarity from the similarity of the blank pixel and the similarity of the at least one pixel. Then, an optimal pixel block can be found within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block where the blank pixel is located. Here, a pixel block adjacent to the second pixel block corresponding to the maximum similarity can be considered an adjacent second pixel block. The relative positional relationship between the adjacent second pixel block and the second pixel block corresponding to the maximum similarity is the same as the relative positional relationship between the blank pixel and the first pixel block corresponding to the maximum similarity. Based on this, the preset range of the second pixel block can include the adjacent second pixel block. Specifically, the preset range can be a preset area centered on the second pixel block corresponding to the maximum similarity, or it can be a preset area centered on the adjacent second pixel block.

[0071] Following the foregoing embodiments, in Figure 5In the scenario shown, for any blank pixel and its two adjacent pixels, if the similarity between the adjacent pixels to the left of the blank pixel is the highest (i.e., similarity (first pixel block 502, second pixel block 505) > similarity (first pixel block 501, second pixel block 504) and similarity (first pixel block 502, second pixel block 505) > similarity (first pixel block 503, second pixel block 506), then the optimal pixel block can be found within a preset range of the second pixel block 505. Since the first pixel block 501 corresponding to the blank pixel is located to the left of the first pixel block 502 with the highest similarity and they are adjacent, the pixel block to the right of the second pixel block 505 and adjacent to it can be denoted as the adjacent second pixel block 601. In this case, the preset range of the second pixel block 505 should include the adjacent second pixel block 601. This preset range can be centered on the second pixel block 505 or on the adjacent second pixel block 601.

[0072] Taking the preset range centered on the adjacent second pixel block 601 as an example, the corresponding preset range can be as follows: Figure 7 The range box 701 is shown in the image. Based on this, the image processing device can traverse each pixel block contained in the range box 701 that is the same size as the second pixel block 601, and determine the similarity between each pixel block and the first pixel block 501 in turn, so as to find the pixel block with the highest similarity as the optimal pixel block within the preset range.

[0073] As mentioned earlier, the target image may also include a foreground region and a background region distinct from the blank region. Therefore, if multiple maximum similarity values ​​exist during the process of determining the maximum similarity among any blank pixel and at least one adjacent pixel, the image processing device can determine the maximum similarity value of the corresponding second pixel block located in the background region as the maximum similarity. This method allows for the selection of the optimal pixel in the background region to fill any blank pixel, thus contributing to a better filling effect. Alternatively, given the gradation of pixel color values ​​in the target image, to ensure a good display effect for the filled blank pixel, the filled color value should be close to the color values ​​of other nearby pixels. This method aims to make the finally determined optimal pixel as close as possible to the blank pixel, thereby contributing to a better filling effect.

[0074] Similarly, in the process of finding the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity, if multiple pixel blocks within the preset range have the same maximum similarity, the image processing device can determine the pixel block in the background area corresponding to the second pixel block as the optimal pixel block; or, it can determine the pixel block corresponding to the second pixel block that is closest to any blank pixel point as the optimal pixel block, which will not be elaborated further.

[0075] Sub-step 3044: The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

[0076] After determining the optimal pixel block, the image processing device can further determine the optimal pixel point within it and use the color value of that pixel point to fill any blank pixel point. The optimal pixel block is the same size as the first pixel block, and the position of the optimal pixel point within the optimal pixel block is the same as the position of any blank pixel point within its first pixel block. This ensures that the optimal pixel block is positionally identical to the blank pixel point, meaning the surrounding pixel environment of the optimal pixel block is similar to that of the blank pixel point, thus ensuring the displayed effect after filling. For example, if any pixel block is a 3x3 grid, and the blank pixel point is the center point of its first pixel block, then the image processing device can determine the center point of the optimal pixel block as the optimal pixel point.

[0077] After determining the optimal pixel, the image processing device can use the color value of the optimal pixel as the color value of any blank pixel, that is, use the color value of the optimal pixel to fill any blank pixel, and the filled blank pixel will have the same color value as the optimal pixel.

[0078] This concludes the description of the filling process for any blank pixel. Similarly, the image processing device can use the above method to sequentially fill each blank pixel in the blank area, thereby completing the filling of the blank area in the target image. At this time, the image processing device can use the target image after completing the filling of each blank pixel in the blank area as the filled image. Alternatively, as mentioned earlier, the process of filling each blank pixel in the target image by the image processing device belongs to the same iteration process, and the image processing device can perform multiple similar iterations on the target image. Specifically, the image processing device can iteratively execute the process of filling each blank pixel in the blank area until the number of iterations reaches a threshold, or stop iterating when the color values ​​of each blank pixel after a certain iteration are the same as those after the previous iteration. The target image at the point where iteration stops is used as the filled image. The threshold number used as the iteration stopping condition can be set according to actual conditions, and this embodiment does not limit this.

[0079] In the case where the target image also includes non-blank regions distinct from the blank regions, for any of the above iterations, the image processing device can further traverse each non-blank pixel in the non-blank regions to update the second pixel block corresponding to the first pixel block where each non-blank pixel is located. This updates the mapping relationship between the first and second pixel blocks of each non-blank similarity to the mapping relationship between the first pixel block and the optimal pixel block. The updated second pixel blocks corresponding to each non-blank pixel can then be used in the next iteration, ensuring that the display effect of the blank regions is better after each iteration than the previous one.

[0080] like Figure 8 As shown, in the filled image, the original (relative to) Figures 4-7 The blank area has been effectively filled, and there are no longer any holes or faults in the area corresponding to blank area 401, indicating a good repair effect. Furthermore, according to... Figure 8 As can be seen from the reference image and the repaired image, the distance that the foreground object closer to the camera moves (the horizontal interval between positioning lines 805 and 804) is greater than the distance that the foreground object farther from the camera moves (the horizontal interval between positioning lines 803 and 802), which more intuitively reflects the parallax in a binocular vision scene.

[0081] In one embodiment, the image processing device can also generate a stereoscopic image based on the movement distance when translating the foreground object, using the reference image and the filled target image (i.e., the filled image). For example, it can... Figure 8The reference image shown is used as the left-eye view, and the infilled image is used as the right-eye view to generate the corresponding stereoscopic image. In this way, a stereoscopic image can be generated from two planar images, thus completing the 2D to 3D conversion. The infilled image is effectively used to present users with a more realistic and immersive stereoscopic display effect, thereby improving the user experience.

[0082] As seen in the foregoing embodiments, the image processing device can determine the first pixel block and the second pixel block corresponding to each pixel in the target image at the initialization node. Based on the pixel blocks determined in the above process, the image processing device can traverse all pixels in the target image—naturally, during this traversal, all blank pixels and all non-blank pixels will be traversed. However, it should be noted that during the traversal of any blank pixel, after determining the optimal pixel block according to the first pixel block and the second pixel block corresponding to that pixel, it is not only necessary to record the optimal pixel block, that is, update the correspondence between the first pixel block and the second pixel block to the correspondence between the first pixel block and the optimal pixel block, but also to update the blank pixel with the color value of the optimal pixel—that is, the traversal process for blank pixels requires updating the correspondence between pixel blocks and filling in the color value of the blank pixels. During the traversal of any non-blank pixel, after determining the optimal pixel block according to the first pixel block and the second pixel block corresponding to that pixel, the image processing device only needs to update the correspondence between the first pixel block and the second pixel block to the correspondence between the first pixel block and the optimal pixel block; it is not necessary to update the color value of the non-blank pixel.

[0083] In fact, since the non-blank pixels in the target image do not need to be filled, the image processing device can determine only the first pixel block and the second pixel block corresponding to each blank pixel in the target image, without determining the first pixel block and the second pixel block corresponding to each non-blank pixel, which will not be elaborated further.

[0084] Corresponding to the aforementioned embodiments of the image filling method, this disclosure also provides embodiments of the image filling apparatus.

[0085] This disclosure provides an image filling apparatus, which is applied to an image filling device, and the apparatus includes one or more processors configured to:

[0086] The foreground object in the reference image is translated in a preset direction to obtain the target image. The target image includes the blank area generated after the foreground object is moved. Each pixel in the target image is located in its corresponding first pixel block. The similarity of any pixel is used to characterize the degree of similarity between the first pixel block where the pixel is located and the second pixel block corresponding to the first pixel block in the target image.

[0087] Iterate through and fill each blank pixel in the blank area, wherein filling any blank pixel includes:

[0088] Determine the maximum similarity among the similarities of any blank pixel and at least one neighboring pixel, and find the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block where the blank pixel is located.

[0089] The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

[0090] In one embodiment, the processor is configured to:

[0091] The movement distance of each foreground object is determined based on the depth value of each foreground object in the reference image. The movement distance of any foreground object is negatively correlated with its depth value.

[0092] Each foreground object moves its own distance in the same preset direction.

[0093] In one embodiment, the target image further includes a foreground region and a background region distinct from the blank region, wherein the foreground region is the region formed by the translated foreground object; the processor is configured to:

[0094] In the case of multiple maximum similarity values, the maximum similarity value of the corresponding second pixel block located in the background region is determined as the maximum similarity value, or the maximum similarity value of the corresponding second pixel block closest to any of the blank pixels is determined as the maximum similarity value; and / or,

[0095] If multiple pixel blocks within the preset range have the same maximum similarity, the pixel block whose corresponding second pixel block is located in the background area is determined as the optimal pixel block, or the pixel block whose corresponding second pixel block is closest to any blank pixel is determined as the optimal pixel block.

[0096] In one embodiment, the processor is configured to:

[0097] Starting from the blank pixel located at the edge of the blank area, sequentially traverse and fill each blank pixel in the blank area.

[0098] In one embodiment, the target image after traversing and filling each blank pixel in the blank area is used as the filled image; or...

[0099] The processor is further configured to: iteratively execute the process of traversing and filling each blank pixel in the blank area until the number of iterations reaches a threshold, or stop iterating when the color values ​​of each blank pixel after a certain iteration are the same as those of each blank pixel after the previous iteration, wherein the target image at the time of stopping iteration is used as the filled image.

[0100] In one embodiment, the target image further includes a non-blank region distinct from the blank region, and the processor is configured to: during any iteration, traverse each non-blank pixel in the non-blank region to update the second pixel block corresponding to the first pixel block where each non-blank pixel is located, and the updated second pixel block corresponding to each non-blank pixel is used in the next iteration.

[0101] In one embodiment, the processor is configured to:

[0102] A corresponding second pixel block is randomly assigned to the first pixel block within a preset area of ​​the target image;

[0103] The preset region includes: a non-blank region in the target image that is distinct from the blank region, or a background region in the non-blank region that is distinct from the foreground region, wherein the foreground region is the region formed by the translated foreground object.

[0104] In one embodiment, the processor is further configured to:

[0105] Based on the translation process and the corresponding movement distance, a stereoscopic image is generated using the reference image and the completed target image.

[0106] In one embodiment, when the preset direction is from left to right, any pixel adjacent to any blank pixel is located above or to the left of any blank pixel; or,

[0107] When the preset direction is from right to left, any pixel adjacent to any blank pixel is located below or to the right of any blank pixel.

[0108] In one embodiment, the similarity between the first pixel block where any pixel is located and its corresponding second pixel block is calculated by the depth value of the pixel contained in the first pixel block and the similarity of the learned perceptual image block in the second pixel block.

[0109] In one embodiment, any pixel is located at the center of the first pixel block in which it is located.

[0110] Embodiments of this disclosure also provide an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the image filling method described in any of the above embodiments.

[0111] Embodiments of this disclosure also provide a non-transient computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the image filling method described in any of the above embodiments.

[0112] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0113] Figure 9 This is a schematic block diagram illustrating an apparatus 900 for determining data storage or driving mode according to embodiments of the present disclosure. For example, apparatus 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0114] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0115] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the image filling method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0116] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0117] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 900.

[0118] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0119] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0120] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0121] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in position of device 900 or a component of device 900, the presence or absence of user contact with device 900, orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0122] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 6G NR, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0123] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image filling method described above.

[0124] In an exemplary embodiment, a non-transient computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to perform the image filling method described above. For example, the non-transient computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0125] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0126] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0127] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0128] The methods and apparatus provided in the embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this disclosure should not be construed as a limitation of this disclosure.

Claims

1. An image filling method, comprising: A target image is obtained by translating a foreground object in a reference image in a preset direction. The target image includes a blank area created by the translation of the foreground object. Each pixel in the target image is located within its corresponding first pixel block. The similarity of any pixel is used to characterize the degree of similarity between the first pixel block in which the pixel is located and the second pixel block in which the first pixel block corresponds in the target image. Determining the second pixel block in the target image corresponding to the first pixel block includes: randomly assigning a corresponding second pixel block to the first pixel block in a preset region of the target image. The preset region includes: a non-blank region in the target image that is different from the blank area, or a background region in the non-blank region that is different from the foreground region. The foreground region is the region formed by the translated foreground object. Iterate through and fill each blank pixel in the blank area, wherein filling any blank pixel includes: Determine the maximum similarity among the similarities of any blank pixel and at least one neighboring pixel, and find the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block where the blank pixel is located. The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

2. The method according to claim 1, wherein translating the foreground object in the reference image according to a preset direction to obtain the target image comprises: The movement distance of each foreground object is determined based on the depth value of each foreground object in the reference image. The movement distance of any foreground object is negatively correlated with its depth value. Each foreground object moves its own distance in the same preset direction.

3. The method according to claim 1, wherein the target image further includes a foreground region and a background region distinct from the blank region, the foreground region being the region formed by the translated foreground object; Determining the maximum similarity among the similarities of any blank pixel and at least one neighboring pixel includes: if multiple maximum similarity values ​​exist, determining the maximum similarity value of the corresponding second pixel block located in the background region as the maximum similarity value, or determining the maximum similarity value of the corresponding second pixel block that is closest to any blank pixel as the maximum similarity value; and / or, Finding the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity includes: when multiple pixel blocks within the preset range have the same maximum similarity, determining the pixel block in the background area corresponding to the second pixel block as the optimal pixel block, or determining the pixel block closest to any blank pixel point corresponding to the second pixel block as the optimal pixel block.

4. The method according to claim 1, wherein traversing and filling each blank pixel in the blank area comprises: Starting from the blank pixel located at the edge of the blank area, sequentially traverse and fill each blank pixel in the blank area.

5. The method according to claim 1, The target image after filling in each blank pixel in the blank area is used as the filled image; or, The method further includes: The process of iteratively filling each blank pixel in the blank area is executed until the number of iterations reaches a threshold, or the iteration stops when the color values ​​of each blank pixel after a certain iteration are the same as those of each blank pixel after the previous iteration. The target image at the time of stopping the iteration is used as the filled image.

6. The method according to claim 5, wherein the target image further includes a non-blank region distinct from the blank region, and in any iteration, the method further includes: Traverse each non-blank pixel in the non-blank region to update the second pixel block corresponding to the first pixel block where each non-blank pixel is located. The updated second pixel blocks corresponding to each non-blank pixel are used in the next iteration process.

7. The method according to claim 1, further comprising: Based on the translation process and the corresponding movement distance, a stereoscopic image is generated using the reference image and the completed target image.

8. The method according to claim 1, When the preset direction is from left to right, any pixel adjacent to any blank pixel is located above or to the left of any blank pixel; or, When the preset direction is from right to left, any pixel adjacent to any blank pixel is located below or to the right of any blank pixel.

9. The method according to claim 1, wherein the similarity between the first pixel block where any pixel point is located and its corresponding second pixel block is calculated by the depth value of the pixel point contained in the first pixel block and the similarity of the learned perceptual image block in the second pixel block.

10. The method according to claim 1, wherein any pixel is located at the center of the first pixel block in which it is located.

11. An image filling apparatus, the apparatus comprising one or more processors, the processors being configured to: A target image is obtained by translating a foreground object in a reference image along a preset direction. The target image includes the blank area created by the movement of the foreground object. Each pixel in the target image is located within its corresponding first pixel block. The similarity of any pixel is used to characterize the degree of similarity between the first pixel block in which the pixel is located and the corresponding second pixel block in the target image. Determining the second pixel block corresponding to the first pixel block in the target image includes: randomly assigning a corresponding second pixel block to the first pixel block in a preset area of ​​the target image, wherein the preset area includes: a non-blank area in the target image that is different from the blank area, or a background area in the non-blank area that is different from the foreground area, wherein the foreground area is the area formed by the translated foreground object; Iterate through and fill each blank pixel in the blank area, wherein filling any blank pixel includes: Determine the maximum similarity among the similarities of any blank pixel and at least one neighboring pixel, and find the optimal pixel block within a preset range of the second pixel block corresponding to the maximum similarity. The optimal pixel block is the pixel block within the preset range that has the highest similarity to the first pixel block where the blank pixel is located. The color value of the optimal pixel in the optimal pixel block is used as the color value of any blank pixel. The position of the optimal pixel in the optimal pixel block is the same as the position of any blank pixel in its first pixel block. The optimal pixel block and the first pixel block are the same size.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 10.

13. A non-transient computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 10.

Images