Method, device and system for filling hole areas in depth images
By acquiring the color and depth images of the target scene, using the pixel value and neighborhood range information in the color image to fill the hollow area of the depth image, the problem of low filling accuracy in the prior art is solved, and higher filling accuracy is achieved Sex and depth information richness.
Patent Information
- Application Number
- CN202010987112.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-09-18
AI Technical Summary
In the prior art, the accuracy of filling hole areas of depth images is low, and it is difficult to effectively use the time domain and spatial domain information for accurate filling.
By acquiring the target color image and the target depth image of the target scene, the pixel value in the hollow area is determined based on the pixel value in the color image, and the pixel value in the first neighborhood and the second neighborhood are filled, combining the similarity degree Differential analysis is performed with smoothing degree to improve the accuracy of filling.
Improve the filling accuracy of the depth image void area, generate color panoramic images containing richer depth information, and enhance the accuracy of image processing.
Smart Images

Figure CN112102199B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image processing technology, and in particular to a method, device, system, electronic device and storage medium for filling a hole area of a depth image. Background Art
[0002] In the prior art, methods for filling hole areas in depth images can be mainly divided into the following two categories:
[0003] The first method uses time domain information to perform real-time hole filling in depth images. For example, by using several consecutive frames of images in a depth video, the motion information of the object and the pixel value change of each pixel position in the depth image are obtained, and the pixel value of the hole to be filled in the depth image is set to the median value of all pixel values in the same position as the hole to be filled in the depth image in several consecutive frames.
[0004] The second method uses spatial information to fill holes in depth images. This method uses the neighborhood pixels of the depth image to fill holes. This method fills holes in the depth image by obtaining the depth value correlation between the hole pixels of the depth image and its neighborhood pixels.
[0005] In the prior art, the accuracy of filling the hole area of the depth image is usually low. Summary of the invention
[0006] Embodiments of the present invention provide a method, device, system, electronic device and storage medium for filling hole areas of a depth image, so as to enrich the filling methods of hole areas of a depth image and improve the accuracy of filling hole areas of a depth image.
[0007] According to one aspect of an embodiment of the present invention, a method for filling a hole area of a depth image is provided, comprising:
[0008] Acquire a target color image and a target depth image obtained by photographing the target scene;
[0009] In response to a hole region existing in the target depth image, determining pixel values of pixels in the hole region based on the target color image;
[0010] The hole area is filled based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0011] Optionally, in the method of any embodiment of the present invention, determining the pixel values of the pixels in the hole area based on the target color image includes:
[0012] Based on the pixel values of the pixels in the target color image, the pixel values of the pixels in the hole area are determined.
[0013] Optionally, in the method of any embodiment of the present invention, determining the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image includes:
[0014] For a pixel in the hole area, a pixel value of the pixel is determined based on pixel values of pixels in a first neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
[0015] Optionally, in the method of any embodiment of the present invention, determining the pixel values of the pixels in the hole area based on the target color image includes:
[0016] For a pixel in the hole area, a pixel value of the pixel is determined based on a degree of smoothness of an image in a second neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
[0017] Optionally, in the method of any embodiment of the present invention, the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following:
[0018] The similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area.
[0019] Optionally, in the method of any embodiment of the present invention, the hole area corresponds to at least one of a weak texture area and a repeated texture area in the target color image.
[0020] According to a second aspect of an embodiment of the present invention, a hole area filling device for a depth image is provided, comprising:
[0021] An acquisition unit is configured to acquire a target color image and a target depth image obtained by photographing a target scene;
[0022] A first determining unit is configured to determine, in response to the presence of a hole region in the target depth image, pixel values of pixels in the hole region based on the target color image;
[0023] The filling unit is configured to fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0024] Optionally, in the device of any embodiment of the present invention, the first determining unit includes:
[0025] The first determining subunit is configured to determine the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image.
[0026] Optionally, in the device of any embodiment of the present invention, the first determining subunit includes:
[0027] The determination module is configured to determine, for a pixel in the hole area, a pixel value of the pixel based on pixel values of pixels in a first neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
[0028] Optionally, in the device of any embodiment of the present invention, the first determining unit includes:
[0029] The second determination subunit is configured to determine, for a pixel in the hole area, a pixel value of the pixel based on a smoothness of an image in a second neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
[0030] Optionally, in the device of any embodiment of the present invention, the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following:
[0031] The similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area.
[0032] Optionally, in the device of any embodiment of the present invention, the hole area corresponds to at least one of a weak texture area and a repeated texture area in the target color image.
[0033] According to a third aspect of an embodiment of the present invention, a system for filling a hole area of a depth image is provided, comprising an image processing unit, a color image shooting device and a depth image shooting device communicatively connected to the image processing unit, wherein:
[0034] The color image shooting device is configured to: shoot a target scene to obtain a target color image;
[0035] The depth image shooting device is configured to: shoot the target scene to obtain a target depth image;
[0036] The image processing unit is configured to: acquire the target color image and the target depth image; in response to the presence of a hole area in the target depth image, determine the pixel values of the pixels in the hole area based on the target color image; and fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0037] Optionally, in the system of any embodiment of the present invention, the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following:
[0038] The similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area.
[0039] Optionally, in the system of any embodiment of the present invention, the hole area corresponds to at least one of a weak texture area and a repeated texture area in the target color image.
[0040] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, including:
[0041] Memory for storing computer programs;
[0042] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method described in any of the above embodiments of the present disclosure is implemented.
[0043] According to a fifth aspect of the embodiments of the present invention, a computer-readable medium is provided, and when the computer program is executed by a processor, the method of any embodiment of the method for filling hole areas in a depth image of the first aspect described above is implemented.
[0044] Based on the method, device, system, electronic device and storage medium for filling the hole area of the depth image provided by the above-mentioned embodiments of the present invention, a target color image and a target depth image obtained by shooting the target scene can be obtained. Then, in the case where there is a hole area in the target depth image, the pixel values of the pixels in the hole area are determined based on the target color image. Finally, based on the pixel values of the pixels in the hole area, the hole area is filled to obtain a filled depth image. In the disclosed embodiment, the hole area of the target depth image of the target scene can be filled based on the target color image of the target scene, thereby enriching the filling method of the hole area of the depth image, improving the accuracy of filling the hole area of the depth image, and facilitating the generation of a color panoramic image containing richer depth information.
[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.
[0047] The present invention may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:
[0048] Figure 1 The present invention is a flowchart of an embodiment of a method for filling a hole area in a depth image.
[0049] Figure 2 The present invention is a flowchart of another embodiment of a method for filling a hole region in a depth image.
[0050] Figure 3 The present invention is a flowchart of another embodiment of the method for filling hole areas in a depth image.
[0051] Figure 4 The figure is a schematic structural diagram of an embodiment of a device for filling a hole region of a depth image according to the present invention.
[0052] Figure 5 The figure is a schematic structural diagram of an embodiment of a system for filling hole regions in a depth image according to the present invention.
[0053] Figure 6 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0054] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0055] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0056] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0057] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0058] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0059] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0060] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0062] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0063] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0064] Embodiments of the present invention may be applied to at least one electronic device in a terminal device, a computer system, and a server, and may operate with numerous other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with at least one electronic device in a terminal device, a computer system, and a server include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0065] At least one of the terminal device, computer system, and server may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.
[0066] Please refer to Figure 1 , shows a process 100 of an embodiment of a method for filling a hole region of a depth image according to the present disclosure. The method for filling a hole region of a depth image comprises the following steps:
[0067] Step 101 : Acquire a target color image and a target depth image obtained by photographing a target scene.
[0068] In this embodiment, the executor of the method for filling the hole area of the depth image (such as a server, a terminal device, an image processing unit with an image processing function, etc.) can obtain the target color image and the target depth image obtained by shooting the target scene from other electronic devices or locally through a wired connection or a wireless connection.
[0069] In this embodiment, the above-mentioned execution entity can first obtain the target color image and then obtain the target depth image, or first obtain the target depth image and then obtain the target color image, or simultaneously obtain the target color image first and then obtain the target depth image. The embodiment of the present application does not limit the order of obtaining the target color image and the target depth image.
[0070] The target scene may be any scene to be photographed. For example, the target scene may be an indoor scene of a house or an outdoor scene of a house. The target color image may be at least one color image obtained by photographing the target scene. The target depth image may be at least one depth image obtained by photographing the target scene. The pixel value of a pixel in the target depth image represents the distance between the shooting position of the target depth image and the photographed position corresponding to the pixel in the target scene.
[0071] In practice, the target color image can be obtained by capturing a color image capturing device. For example, the color image capturing device can be a color camera. The target depth image can be obtained by capturing a depth image capturing device. For example, the depth image capturing device can be a structured-light camera, a stereo camera, a time of flight (TOF) camera, and the like.
[0072] Here, usually there is a difference in at least one of the shooting angle and the shooting position (for example, the position of the color image shooting device or the position of the depth image shooting device) of the target color image and the target depth image. Therefore, there may be pixels in the target color image that do not correspond to any pixels in the target depth image. In other words, the picture of the target scene in the target color image may not be completely the same as the picture of the target scene in the target depth image (that is, part is the same and the other part is different).
[0073] It can be understood that the relative position between the color image capturing device and the depth image capturing device can be randomly determined during the capturing process, or can be predetermined and fixed.
[0074] Step 102 : In response to the existence of a hole area in the target depth image, pixel values of pixels in the hole area are determined based on the target color image.
[0075] In this embodiment, when there is a hole area in the target depth image, the execution subject may determine the pixel values of the pixels in the hole area based on the target color image.
[0076] It is understandable that there may often be a hole area in the depth image (including the target depth image) captured. For example, in the process of capturing a depth image using a structured light camera, when there are weak textures and / or repeated textures in the scene, there are usually hole areas corresponding to the weak textures and / or repeated textures in the depth image captured.
[0077] In practice, it is possible to determine whether there is a hole area in the target depth image and determine the position of the hole area in the target depth image by detecting whether there are pixels whose pixel values are less than or equal to a preset value (such as 0) or whose pixel values are empty in the target depth image. Optionally, the image area of the target depth image corresponding to the weak texture image area and / or the repeated texture image area in the target color image can also be determined as a hole area. Thus, it is possible to determine whether there is a hole area in the target depth image and determine the position of the hole area in the target depth image by determining whether there are weak texture image areas and / or repeated texture image areas in the target color image.
[0078] In some optional implementations of this embodiment, the execution subject may determine the pixel values of the pixels in the hole area based on the target color image by the following steps:
[0079] Based on the pixel values of the pixels in the target color image, the pixel values of the pixels in the hole region are determined.
[0080] It can be understood that the pixel values of the pixels in the hole area are determined based on the pixel values of the pixels in the target color image, and the hole area can be filled according to the color information (such as color difference), thereby improving the accuracy of filling the hole area of the depth image.
[0081] In the above optional implementation, the above execution subject may determine the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image by taking the following steps:
[0082] For a pixel in the hole area (i.e., a pixel to be filled), a pixel value of the pixel is determined based on the pixel values of pixels in a first neighborhood range of a pixel in the target color image corresponding to the pixel in the hole area. Here, the pixel in the target color image corresponding to the pixel in the hole area may indicate the same position in the target scene as the pixel in the hole area.
[0083] The first neighborhood range may be an image region of the target color image that includes the pixel in the hole region, or an image region of the target color image that is adjacent to the pixel in the hole region. As an example, the first neighborhood range may be an image region consisting of a preset number of pixels (e.g., 25, 49, etc.) centered on the pixel in the hole region.
[0084] Here, the pixel value of the pixel (i.e., the pixel to be filled) may be the average value of the pixel values of the pixels within the first neighborhood or the product of the weighted summation and the first preset coefficient. The first preset coefficient may represent the relationship (e.g., multiple relationship) between the pixel value of a specific pixel in the target color image and the pixel value of the pixel corresponding to the specific pixel in the target depth image.
[0085] It can be understood that pixels within the first neighborhood are usually closer to the pixel (i.e., the pixel to be filled). Therefore, using the pixel values of the pixels within the first neighborhood to determine the pixel value of the pixel can further improve the accuracy of filling the hole area of the depth image.
[0086] In some optional implementations of this embodiment, the above-mentioned execution subject may also adopt the following steps to determine the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image:
[0087] For the pixels in the hole area (i.e., the pixels to be filled), select at least one pixel in the first neighborhood range of the pixels corresponding to the pixel from the target color image, randomly select a pixel from the at least one pixel, and use the product of the selected pixel value and the first preset coefficient as the pixel value of the pixel (i.e., the pixel to be filled). Optionally, the pixel value of the pixel (i.e., the pixel to be filled) may also be the average value of the pixel values of each pixel in the at least one pixel or the product of the weighted summation and the first preset coefficient. The first preset coefficient may characterize the relationship between the pixel value of a specific pixel in the target color image and the pixel value of the pixel corresponding to the specific pixel in the target depth image (e.g., a multiple relationship).
[0088] In some optional implementations of this embodiment, the above execution subject may also adopt the following steps to determine the pixel values of the pixels in the hole area based on the target color image:
[0089] For a pixel in the hole area (i.e., a pixel to be filled), a pixel value of the pixel (i.e., a pixel to be filled) is determined based on the smoothness of an image in a second neighborhood range of pixels in the target color image corresponding to the pixel in the hole area (i.e., a pixel to be filled).
[0090] Among them, the smoothness value of the image (including the target color image) can be characterized by the gradient of the image, etc. The pixel value of the pixel (i.e., the pixel to be filled) can be the product of the average value of the smoothness value of the pixels of the image in the second neighborhood range and the second preset coefficient, or the pixel value of the pixel (i.e., the pixel to be filled) can also be the product of the weighted sum of the smoothness value of the pixels of the image in the second neighborhood range and the second preset coefficient. The second neighborhood range can be an image area in the target color image that contains the pixel in the hole area, or it can be an image area adjacent to the pixel in the hole area in the target color image. As an example, the above-mentioned second neighborhood range can be an image area composed of a preset number of pixels (e.g., 25, 49, etc.) centered on the pixel in the hole area. The above-mentioned second preset coefficient can represent the relationship (e.g., a multiple relationship) between the smoothness of a specific pixel in the target color image and the pixel value of the pixel corresponding to the above-mentioned specific pixel in the target depth image.
[0091] It can be understood that by using the smoothness of the image within the second neighborhood to determine the pixel value of the pixel, the accuracy of filling the hole area of the depth image can be improved.
[0092] In some optional implementations of the present embodiment, the degree of difference between the pixel values of the pixels in the hole area in the above step 102 is negatively correlated with at least one of the following: the degree of similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the degree of smoothness of the pixels in the target color image corresponding to the pixel in the hole area.
[0093] The similarity between the pixel values can be represented by a difference, etc. For example, the similarity can be represented by a difference between at least one of the R (red) value, G (green) value, B (blue) value, H (hue) value, S (saturation) value, and V (value) value of the pixel.
[0094] In practice, generally speaking, the more similar and smoother the pixel values of the neighboring pixels of the pixel corresponding to the pixel to be filled in the color image (including the target color image), the smaller the depth difference between the pixel to be filled and the neighboring pixels to be filled. Therefore, the above optional implementation method can fill the hole area according to the color and texture difference, thereby further improving the accuracy of filling the hole area of the depth image.
[0095] In some optional implementations of this embodiment, the hole region in the above step 102 corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
[0096] It can be understood that the weak texture area in the target color image has fewer feature points and the feature points in the repeated texture area overlap, which may cause the acquired target depth image to have a hole area. Therefore, the above optional implementation method can determine the position of the hole area in the target depth image through the weak texture area and the repeated texture area in the target color image, thereby improving the positioning accuracy of the hole area, and then improving the accuracy of filling the hole area of the depth image.
[0097] Step 103: Fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0098] In this embodiment, the execution subject may fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0099] It can be understood that since the pixel value of a pixel in the depth image represents the distance between the shooting position of the depth image and the shooting position corresponding to the pixel in the target scene, the hole area can be filled based on the pixel value of the pixel in the hole area to obtain a filled depth image.
[0100] The method for filling the hole area of the depth image provided by the above-mentioned embodiment of the present invention can obtain the target color image and the target depth image obtained by shooting the target scene, and then, in the case where there is a hole area in the target depth image, the pixel values of the pixels in the hole area are determined based on the target color image, and finally, the hole area is filled based on the pixel values of the pixels in the hole area to obtain the filled depth image. In the embodiment of the present disclosure, the hole area of the target depth image of the target scene can be filled based on the target color image of the target scene, thereby enriching the filling method of the hole area of the depth image, and can improve the accuracy of filling the hole area of the depth image, which is helpful to generate a color panoramic image containing richer depth information.
[0101] Further references Figure 2 , Figure 2 FIG. 2 is a flowchart of another embodiment of filling a hole region of a depth image of the present invention. The hole region filling process 200 of the depth image includes the following steps:
[0102] Step 201 : acquiring a target color image obtained by photographing a target scene with a preset number of color image photographing devices, and a target depth image obtained by photographing a target scene with a preset number of depth image photographing devices.
[0103] In this embodiment, the execution subject of the method for filling the hole area of the depth image (such as a server, a terminal device, an image processing unit with an image processing function, etc.) can obtain the target color image obtained by shooting the target scene by a preset number of color image shooting devices from other electronic devices or locally through a wired connection or a wireless connection, and obtain the target depth image obtained by shooting the target scene by a preset number of depth image shooting devices. The relative positions between the preset number of color image shooting devices and the preset number of depth image shooting devices are predetermined.
[0104] In this embodiment, the above-mentioned execution entity can first obtain the target color image and then obtain the target depth image, or first obtain the target depth image and then obtain the target color image, or obtain the target color image and the target depth image at the same time. The embodiment of the present application does not limit the order of obtaining the above-mentioned target color image and target depth image.
[0105] Exemplarily, the color image capturing device may be a color camera. The depth image capturing device may be a structured light camera, a binocular vision camera, a time-of-flight camera, etc. Here, each color image capturing device may capture one or more color images, and each depth image capturing device may capture one or more depth images. The color image capturing device and the depth image capturing device may be disposed on the execution subject, or may be disposed independently of the execution subject. The color image capturing device and the depth image capturing device may be directly or indirectly connected to the execution subject for communication.
[0106] In some optional implementations of this embodiment, during the shooting process, a preset number of color image shooting devices and a preset number of depth image shooting devices all rotate around the target rotation axis; and
[0107] The target color image and target depth image are obtained by taking the following steps:
[0108] First, when the rotation angles of a preset number of color image capturing devices and a preset number of depth image capturing devices are preset angles, the preset number of color image capturing devices and the preset number of depth image capturing devices can respectively capture the scene areas corresponding to the rotation angles in the target scene to obtain color images and depth images.
[0109] It should be understood that during the shooting process, a preset number of color image shooting devices and a preset number of depth image shooting devices all rotate around the target rotation axis. Whenever the preset number of color image shooting devices and the preset number of depth image shooting devices rotate by a preset angle, the preset number of color image shooting devices and the preset number of depth image shooting devices can respectively shoot the scene area corresponding to the rotation angle in the target scene.
[0110] The preset angle may be one angle or multiple angles.
[0111] For example, when the preset angle is an angle, it can be represented that: each time a preset number of color image capturing devices and a preset number of depth image capturing devices rotate the preset angle once, the preset number of color image capturing devices and the preset number of depth image capturing devices can respectively shoot the scene area corresponding to the rotation angle in the target scene once. For example, the preset angle can be 90 degrees, which represents that each time a preset number of color image capturing devices and a preset number of depth image capturing devices rotate 90 degrees, the preset number of color image capturing devices and the preset number of depth image capturing devices can respectively shoot the scene area corresponding to the rotation angle in the target scene once. That is, the preset number of color image capturing devices and the preset number of depth image capturing devices can respectively shoot the target scene when they rotate 1 90 degrees, 2 90 degrees (i.e., 180 degrees), 3 90 degrees (i.e., 270 degrees), and 4 90 degrees (i.e., 360 degrees).
[0112] For another example, when the preset angle includes multiple angles, it can be represented as follows: whenever a preset number of color image capturing devices and a preset number of depth image capturing devices rotate any one of the preset angles, the preset number of color image capturing devices and the preset number of depth image capturing devices can respectively take a picture of the scene area corresponding to the rotation angle in the target scene. For example, the preset angles can include 90 degrees, 180 degrees, 270 degrees, and 360 degrees, representing that whenever a preset number of color image capturing devices and a preset number of depth image capturing devices rotate 90 degrees, 180 degrees, 270 degrees, and 360 degrees, the scene area corresponding to each of the above rotation angles in the target scene is photographed once.
[0113] Here, the number of color images captured by the preset number of color image capturing devices is generally equal to the number of depth images captured by the preset number of depth image capturing devices.
[0114] Then, when the number of captured color images and depth images reaches preset values respectively, the captured color images of the preset value are used as target color images, and the captured depth images of the preset value are used as target depth images.
[0115] Here, the above optional implementation is illustrated by taking the preset number as 2, the rotation angle as 120 degrees, 240 degrees, 360 degrees, and the preset value as 6 as an example. Among them, the straight line where the two color image shooting devices are located can be perpendicular to the ground, and the straight line where the two depth image shooting devices are located can be perpendicular to the ground.
[0116] During the shooting process, the two color image shooting devices and the two depth image shooting devices rotate around the target rotation axis. Every time they rotate 120 degrees, the two color image shooting devices and the two depth image shooting devices can shoot once respectively. Thus, when rotating 120 degrees, the two color image shooting devices and the two depth image shooting devices can shoot 2 color images and 2 depth images. Afterwards, when rotating 120 degrees again (i.e., rotating 240 degrees relative to the initial posture), the two color image shooting devices and the two depth image shooting devices can shoot again to obtain 2 color images and 2 depth images. Then, when rotating 120 degrees again (i.e., rotating 360 degrees relative to the initial posture), the two color image shooting devices and the two depth image shooting devices can shoot again to obtain 2 color images and 2 depth images. So far, a total of 6 color images and 6 depth images are obtained, and the number of color images and depth images captured reaches the preset value. Thus, the six captured color images may be used as target color images, and the six captured depth images may be used as target depth images.
[0117] It can be understood that in the above optional implementation, a preset number of color image shooting devices and a preset number of depth image shooting devices can rotate around the target rotation axis, so as to capture multiple color images and depth images with close shooting positions and predetermined posture relationships therebetween. This helps to more quickly and accurately determine the relevant information of the hole area (such as the position of the hole area in each depth image, the image area in the color image corresponding to the hole area, etc.), thereby achieving faster and more accurate filling of the hole area.
[0118] Optionally, the target color image and the target depth image can also be obtained by shooting the target scene at multiple angles and / or multiple directions using a preset number of color image shooting devices and a preset number of depth image shooting devices. The multiple angles and directions can be randomly determined or preset.
[0119] Step 202 : In response to the existence of a hole area in the target depth image, pixel values of pixels in the hole area are determined based on the target color image.
[0120] In this embodiment, step 202 and Figure 1 Step 102 in the corresponding embodiment is basically the same and will not be described again here.
[0121] Step 203: Fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0122] In this embodiment, step 203 and Figure 1 Step 103 in the corresponding embodiment is basically the same and will not be described again here.
[0123] It should be noted that, in addition to the above contents, the embodiments of the present application may also include Figure 1 The same or similar features and effects as those in the corresponding embodiments will not be described in detail here.
[0124] from Figure 2 It can be seen that the process 200 of the method for filling the hole area of the depth image in this embodiment can obtain a target color image obtained by photographing the target scene through a preset number of color image shooting devices, and a target depth image obtained by photographing the target scene through a preset number of depth image shooting devices, wherein the relative positions between the preset number of color image shooting devices and the preset number of depth image shooting devices are predetermined, and then, in the case where there is a hole area in the target depth image, the pixel values of the pixels in the hole area are determined based on the target color image, and finally, based on the pixel values of the pixels in the hole area, the hole area is filled to obtain a filled depth image. The relative positions between the preset number of color image shooting devices and the preset number of depth image shooting devices can be predetermined, thereby, the relative position relationship between each color image shooting device and each depth image shooting device can be predetermined, and then the relative posture relationship between each color image and each depth image shot by each color image shooting device and each depth image shooting device can be obtained, so that through the subsequent steps, the pixel values of the pixels in the hole area can be determined more quickly, thereby improving the speed of filling the hole area.
[0125] In some optional implementations of this embodiment, the preset value is the quotient of 360 degrees and the preset angle.
[0126] It can be understood that the above optional implementation method can capture multiple color images and depth images with close shooting positions and predetermined posture relationships therebetween, and each color image and each depth image can reflect the 360-degree scene information (including color information and depth information) of the above-mentioned target scene, thereby achieving more comprehensive acquisition of scene information of the target scene.
[0127] In some optional implementations of this embodiment, the above-mentioned execution subject may also synthesize a preset number of depth images into a depth panoramic image based on the posture relationship between each depth image in the preset number of depth images.
[0128] Here, the above-mentioned posture relationship can be represented in the form of a matrix, etc., and the posture relationship can be determined based on the position, shooting angle, rotation angle, etc. of the color image shooting device and the depth image shooting device. No further details will be given here.
[0129] Specifically, for each pixel in each depth image, the image coordinate system of the pixel can be projected into the three-dimensional space of the camera coordinate system to obtain a point cloud composed of [x, y, z]. Then, according to the posture relationship of each depth image, the point clouds in different camera coordinate systems can be converted to the same coordinate system to obtain a complete point cloud, and then a preset number of depth images can be synthesized into a depth panoramic image.
[0130] It can be understood that the above optional implementation manner can realize the synthesis of the depth panoramic image, so as to obtain the depth panoramic image that does not include the hole area.
[0131] In some optional implementations of this embodiment, the above execution subject may further perform the following steps:
[0132] Step 1: for a pixel in the depth panoramic image, determine the color information of a pixel corresponding to the pixel in the captured color image.
[0133] Step 2: Generate a color panoramic image containing depth information based on the obtained color information and the pixel values of the pixels in the depth panoramic image.
[0134] Exemplarily, the following steps may be repeatedly performed for each of the captured color images, thereby generating a color panoramic image containing depth information:
[0135] Based on the depth panoramic image, traverse each pixel of the depth panoramic image (for example, the first pixel [x0, y0]), convert each pixel into a three-dimensional coordinate point p [x, y, z], and based on the external parameter matrix between the color image and the depth panoramic image (which can characterize the correspondence between the pixels in the color image and the pixels in the depth panoramic image), the above three-dimensional coordinate point p can be projected into the camera coordinate system of the color image. Then, the above three-dimensional coordinate point p can be projected into the color image, and the color pixel value can be assigned to the pixel position (for example, the first pixel [x0, y0]). After traversing all pixels, the local image of the color image in the panoramic image can be obtained (that is, the image area corresponding to the color image in the color panoramic image to be generated and containing depth information).
[0136] It can be understood that the panoramic view in the prior art Figure 1 Generally, the panoramic image is taken by a panoramic camera, which usually only contains two-dimensional image information of the scene. However, the above optional implementation method can generate a color panoramic image containing depth information.
[0137] Please continue to refer to Figure 3 , Figure 3 FIG. 3 is a flow chart of another embodiment of filling a hole region of a depth image of the present invention. The process 300 of filling a hole region of a depth image includes the following steps:
[0138] Step 301: Acquire a target color image and a target depth image obtained by photographing a target scene.
[0139] In this embodiment, step 301 and Figure 1 Step 101 in the corresponding embodiment is basically the same and will not be described again here.
[0140] Step 302, in response to the presence of a hole area in the target depth image, for a pixel in the hole area, based on the similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area, determines the pixel value of the pixel.
[0141] In this embodiment, when there is a hole area in the target depth image, for the pixels in the hole area, the executor of the hole area filling method of the depth image (such as a server, a terminal device, an image processing unit with an image processing function, etc.) can determine the pixel value of the pixel based on the similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area.
[0142] Exemplarily, the pixel value of each pixel in the hole area may be the result of the pixel values (characterizing depth) of each pixel in the image area containing the pixel (e.g., an image area composed of a preset number (25, 49, etc.) of pixels centered on the pixel) in the target hole image. The weight of the pixel value of each pixel may be negatively correlated with at least one of the following: the degree of similarity between the pixel values (characterizing color information) of the pixel corresponding to the pixel in the hole area in the target color image, and the degree of smoothness of the pixel (characterizing color information) corresponding to the pixel in the hole area in the target color image. The pixel corresponding to the pixel in the hole area in the target color image indicates the same position in the target scene.
[0143] Optionally, when calculating the pixel value of each pixel in the hole area, if the target hole image includes an image area of the pixel and there are pixels located in the hole area other than the pixel, the weights of the pixels located in the hole area other than the pixel can be set to 0.
[0144] Step 303: Fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0145] In this embodiment, step 303 and Figure 1 Step 103 in the corresponding embodiment is basically the same and will not be described again here.
[0146] It should be noted that, in addition to the above contents, the embodiments of the present application may also include Figure 1 The same or similar features and effects as those in the corresponding embodiments will not be described in detail here.
[0147] from Figure 3 It can be seen that in the process 300 of the method for filling the hole area of the depth image in this embodiment, first, the target color image and the target depth image obtained by shooting the target scene are obtained, and then, in the case where there is a hole area in the target depth image, for the pixel in the hole area, the pixel value of the pixel is determined based on the pixel value of the pixel in the first neighborhood range of the pixel corresponding to the pixel in the hole area in the target color image and the smoothness of the image in the second neighborhood range. Finally, based on the pixel value of the pixel in the hole area, the hole area is filled to obtain the filled depth image. Since the pixel values of the pixels adjacent to the pixels to be filled are more similar and smoother, the depth difference between the pixel to be filled and the adjacent pixels to be filled is often smaller. Therefore, the above optional implementation method can fill the hole area according to the color and texture differences, thereby further improving the accuracy of filling the hole area of the depth image.
[0148] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a hole area filling device for a depth image. Figure 1 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 1 The same or corresponding features of the method embodiment shown, and the generation of Figure 1 The method embodiment shown has the same or corresponding effects. The device can be specifically applied to various electronic devices.
[0149] like Figure 4As shown, the hole area filling device 400 of the depth image of this embodiment includes: an acquisition unit 401, a first determination unit 402 and a filling unit 403. Among them, the acquisition unit 401 is configured to acquire a target color image and a target depth image obtained by shooting a target scene; the first determination unit 402 is configured to determine the pixel values of the pixels in the hole area based on the target color image in response to the existence of a hole area in the target depth image; the filling unit 403 is configured to fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0150] In this embodiment, the acquisition unit 401 of the hole region filling device 400 of the depth image can acquire a target color image and a target depth image obtained by photographing a target scene.
[0151] In this embodiment, when there is a hole area in the target depth image, the first determining unit 402 may determine the pixel values of the pixels in the hole area based on the target color image.
[0152] In this embodiment, the filling unit 403 may fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0153] In some optional implementations of this embodiment, the acquiring unit 401 includes:
[0154] An acquisition subunit (not shown in the figure) is configured to acquire a target color image obtained by photographing the target scene via a preset number of color image capturing devices, and a target depth image obtained by photographing the target scene via a preset number of depth image capturing devices, wherein the relative positions between the preset number of color image capturing devices and the preset number of depth image capturing devices are predetermined.
[0155] In some optional implementations of this embodiment, during the shooting process, a preset number of color image shooting devices and a preset number of depth image shooting devices all rotate around the target rotation axis; and
[0156] The target color image and target depth image are obtained by taking the following steps:
[0157] In response to the rotation angles of a preset number of color image capturing devices and a preset number of depth image capturing devices being preset angles, capturing a scene area corresponding to the rotation angle in a target scene to obtain a color image and a depth image;
[0158] In response to the number of the captured color images and depth images reaching preset values, the captured color images of the preset value are used as target color images, and the captured depth images of the preset value are used as target depth images.
[0159] In some optional implementations of this embodiment, the preset value is the quotient of 360 degrees and the preset angle.
[0160] In some optional implementations of this embodiment, the apparatus 400 further includes:
[0161] The synthesis unit (not shown in the figure) is configured to synthesize a preset number of depth images into a depth panoramic image based on the posture relationship between each depth image in the preset number of depth images.
[0162] In some optional implementations of this embodiment, the apparatus 400 further includes:
[0163] A second determining unit (not shown in the figure) is configured to determine, for a pixel in the depth panoramic image, color information of a pixel corresponding to the pixel in the captured color image;
[0164] The generating unit (not shown in the figure) is configured to generate a color panoramic image containing depth information based on the obtained color information and pixel values of pixels in the depth panoramic image.
[0165] In some optional implementations of this embodiment, the first determining unit 402 includes:
[0166] The first determining subunit (not shown in the figure) is configured to determine the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image.
[0167] In some optional implementations of this embodiment, the first determining subunit includes:
[0168] The determination module (not shown in the figure) is configured to determine the pixel value of the pixel in the hole area based on the pixel values of the pixels in the first neighborhood range of the pixel in the hole area in the target color image.
[0169] In some optional implementations of this embodiment, the first determining unit 402 includes:
[0170] The second determination subunit (not shown in the figure) is configured to determine the pixel value of the pixel in the hole area based on the smoothness of the image in the second neighborhood range of the pixel in the hole area in the target color image.
[0171] In some optional implementations of the present embodiment, the pixel value of a pixel in the filled depth image is positively correlated with the pixel value of a pixel in a first neighborhood range of a pixel in the target color image corresponding to the pixel in the hole area, and the pixel value of a pixel in the filled depth image is negatively correlated with at least one of the following smoothness levels of an image in a second neighborhood range of a pixel in the target color image corresponding to the pixel in the hole area.
[0172] In some optional implementations of this embodiment, the hole region corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
[0173] In the device for filling hole areas of a depth image provided by the above-mentioned embodiment of the present disclosure, an acquisition unit 401 acquires a target color image and a target depth image obtained by photographing a target scene. Then, in the case where a hole area exists in the target depth image, a first determination unit 402 determines the pixel values of the pixels in the hole area based on the target color image. Finally, a filling unit 403 fills the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image. Thus, the hole area of the target depth image of the target scene can be filled based on the target color image of the target scene, thereby enriching the filling method of the hole area of the depth image, improving the accuracy of filling the hole area of the depth image, and facilitating the generation of a color panoramic image containing richer depth information.
[0174] Please refer to the following Figure 5 , Figure 5 The present invention is a schematic diagram of the structure of an embodiment of a system for filling hole areas of a depth image. The system for filling hole areas of a depth image includes an image processing unit, a color image shooting device and a depth image shooting device that are communicatively connected to the image processing unit. Among them: the color image shooting device is configured to: shoot the target scene to obtain a target color image; the depth image shooting device is configured to: shoot the target scene to obtain a target depth image; the image processing unit is configured to: obtain a target color image and a target depth image; in response to the existence of a hole area in the target depth image, determine the pixel values of the pixels in the hole area based on the target color image; based on the pixel values of the pixels in the hole area, fill the hole area to obtain a filled depth image.
[0175] like Figure 5 As shown, in step 501, a color image shooting device shoots a target scene to obtain a target color image.
[0176] In this embodiment, the color image shooting device can shoot the target scene to obtain a target color image.
[0177] In step 502, a depth image capturing device captures a target scene to obtain a target depth image.
[0178] In this embodiment, the depth image shooting device can shoot the target scene to obtain a target depth image.
[0179] In step 503, the color image capturing device sends the target color image to the image processing unit.
[0180] In this embodiment, the color image capturing device may send the target color image to the image processing unit.
[0181] In step 504, the depth capturing device sends the target depth to the image processing unit.
[0182] In this embodiment, the depth photographing device may send the target depth to the image processing unit.
[0183] In step 505 , in response to the presence of a hole region in the target depth image, the image processing unit determines pixel values of pixels in the hole region based on the target color image.
[0184] In this embodiment, when there is a hole area in the target depth image, the image processing unit may determine the pixel values of the pixels in the hole area based on the target color image.
[0185] In step 506 , the image processing unit fills the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0186] In this embodiment, the image processing unit may fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
[0187] In some optional implementations of this embodiment, the number of color image capturing devices and the number of depth image capturing devices are both preset numbers, and the relative positions between the preset number of color image capturing devices and the preset number of depth image capturing devices are predetermined.
[0188] In some optional implementations of this embodiment, the system further includes a rotation axis, the color image capturing device and the depth image capturing device are respectively connected to the rotation axis, and during the shooting process, the color image capturing device and the depth image capturing device both rotate around the rotation axis; and
[0189] The color image shooting device shoots a color image every time it rotates a preset angle, and uses the multiple color images shot as target color images;
[0190] The depth image capturing device captures a depth image each time it rotates a preset angle, and uses the captured multiple depth images as target depth images;
[0191] The number of captured color images and the number of target depth images are both the quotient of 360 degrees and a preset angle.
[0192] In some optional implementations of this embodiment, the image processing unit is further configured to:
[0193] Based on the position and posture relationship between each of the multiple depth images obtained by shooting, the multiple depth images are synthesized into a depth panoramic image.
[0194] In some optional implementations of this embodiment, the image processing unit is further configured to:
[0195] For a pixel in the depth panoramic image, determine color information of a pixel corresponding to the pixel in the plurality of captured color images;
[0196] Based on the obtained color information and the pixel values of the pixels in the depth panoramic image, a color panoramic image including the depth information is generated.
[0197] In some optional implementations of the present embodiment, the pixel value of a pixel in the filled depth image is positively correlated with the pixel value of a pixel in a first neighborhood range of a pixel in the target color image corresponding to the pixel in the hole area, and the pixel value of a pixel in the filled depth image is negatively correlated with at least one of the following smoothness levels of an image in a second neighborhood range of a pixel in the target color image corresponding to the pixel in the hole area.
[0198] In some optional implementations of this embodiment, the hole region corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
[0199] It should be noted that, in addition to the above contents, the embodiments of the present application may also include Figure 1 The same or similar features and effects as those of the corresponding embodiments. For example, the image processing unit in the hole region filling system of the depth image of the present invention can also be configured to perform Figure 1 For another example, the color image capturing device (or depth image capturing device) in the hole region filling system of the depth image of the present invention may include: Figure 1 The color image capturing device (or depth image capturing device) in the corresponding embodiment has the same or similar features. In addition, Figure 1The color image capturing device (or depth image capturing device) in the corresponding embodiment may also include the same or similar features as the color image capturing device (or depth image capturing device) in the hole area filling system of the depth image of the present invention described above, which will not be described in detail here.
[0200] The hole area filling system of the depth image provided by the above-mentioned embodiment of the present disclosure includes an image processing unit, a color image shooting device and a depth image shooting device which are communicatively connected to the image processing unit, wherein: the color image shooting device is configured to: shoot the target scene to obtain the target color image; the depth image shooting device is configured to: shoot the target scene to obtain the target depth image; the image processing unit is configured to: obtain the target color image and the target depth image; in response to the existence of a hole area in the target depth image, the pixel value of the pixel in the hole area is determined based on the target color image; based on the pixel value of the pixel in the hole area, the hole area is filled to obtain the filled depth image. Thus, the hole area of the target depth image of the target scene can be filled based on the target color image of the target scene, thereby enriching the filling method of the hole area of the depth image, improving the accuracy of filling the hole area of the depth image, and facilitating the generation of a color panoramic image containing richer depth information.
[0201] Below, reference Figure 6 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.
[0202] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.
[0203] like Figure 6 As shown, the electronic device 6 includes one or more processors 601 and a memory 602 .
[0204] The processor 601 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0205] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may run the program instructions to implement the hole area filling method of the depth image of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0206] In one example, the electronic device may further include: an input device 603 and an output device 604, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0207] For example, when the electronic device is the first device or the second device, the input device 603 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 603 may be a communication network connector, for receiving collected input signals from the first device and the second device.
[0208] In addition, the input device 603 may also include, for example, a keyboard, a mouse, etc. The output device 604 may output various information to the outside, including determined distance information, direction information, etc. The output device 604 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.
[0209] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0210] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for filling hole areas of a depth image according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.
[0211] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0212] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method for filling hole areas of a depth image according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.
[0213] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0214] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0215] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0216] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.
[0217] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0218] The method and apparatus of the present invention may be implemented in many ways. For example, the method and apparatus of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.
[0219] The description of the present invention is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for particular uses.
Claims
1. A method for filling a hole area in a depth image, characterized in that: include: Acquire a target color image and a target depth image obtained by photographing the target scene; In response to the presence of a hole area in the target depth image, determining pixel values of pixels in the hole area based on the target color image includes: For the pixel in the hole area, the pixel value of the pixel is determined based on the smoothness of the image in the second neighborhood range of the pixel in the target color image corresponding to the pixel in the hole area, wherein the smoothness value is represented by the gradient of the image; the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following: the similarity between the pixel values of the pixels in the target color image corresponding to the pixel in the hole area, and the smoothness of the pixels in the target color image corresponding to the pixel in the hole area; the pixel value is the product of the average value or weighted summation of the smoothness values of the pixels in the image in the second neighborhood range and a second preset coefficient; the second preset coefficient represents the relationship between the smoothness of the pixel in the target color image and the pixel value of the pixel corresponding to this pixel in the target depth image; The hole area is filled based on the pixel values of the pixels in the hole area to obtain a filled depth image.
2. The method according to claim 1, characterized in that The step of determining the pixel values of pixels in the hole area based on the target color image includes: Based on the pixel values of the pixels in the target color image, the pixel values of the pixels in the hole area are determined.
3. The method according to claim 2, characterized in that The step of determining the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image comprises: For a pixel in the hole area, a pixel value of the pixel is determined based on pixel values of pixels in a first neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
4. The method according to claim 1, characterized in that: The hole region corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
5. A device for filling a hole region in a depth image, characterized in that: include: An acquisition unit is configured to acquire a target color image and a target depth image obtained by photographing a target scene; A first determining unit is configured to determine, in response to the presence of a hole region in the target depth image, pixel values of pixels in the hole region based on the target color image; Wherein, the first determination unit includes: a second determination subunit, configured to determine the pixel value of the pixel in the hole area based on the smoothness of the image of the second neighborhood range of the pixel corresponding to the pixel in the hole area in the target color image, wherein the smoothness value is represented by the gradient of the image; the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following: the similarity between the pixel values of the pixels corresponding to the pixel in the hole area in the target color image, and the smoothness of the pixels corresponding to the pixel in the hole area in the target color image; the pixel value is the product of the average value or weighted summation of the smoothness values of the pixels in the image of the second neighborhood range and a second preset coefficient; the second preset coefficient represents the relationship between the smoothness of the pixel in the target color image and the pixel value of the pixel corresponding to this pixel in the target depth image; The filling unit is configured to fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image.
6. The device according to claim 5, characterized in that The first determining unit includes: The first determining subunit is configured to determine the pixel values of the pixels in the hole area based on the pixel values of the pixels in the target color image.
7. The device according to claim 6, characterized in that The first determining subunit includes: The determination module is configured to determine, for a pixel in the hole area, a pixel value of the pixel based on pixel values of pixels in a first neighborhood range of pixels in the target color image corresponding to the pixel in the hole area.
8. The device according to claim 5, characterized in that The hole region corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
9. A system for filling holes in depth images, characterized in that: The system comprises an image processing unit, a color image capturing device and a depth image capturing device which are communicatively connected to the image processing unit, wherein: The color image shooting device is configured to: shoot a target scene to obtain a target color image; The depth image shooting device is configured to: shoot the target scene to obtain a target depth image; The image processing unit is configured to: acquire the target color image and the target depth image; in response to the existence of a hole area in the target depth image, determine the pixel values of the pixels in the hole area based on the target color image; fill the hole area based on the pixel values of the pixels in the hole area to obtain a filled depth image; Wherein, determining the pixel value of the pixel in the hole area based on the target color image includes: for the pixel in the hole area, determining the pixel value of the pixel based on the smoothness of the image in the second neighborhood range of the pixel in the target color image corresponding to the pixel in the hole area, wherein the smoothness value is represented by the gradient of the image; the degree of difference between the pixel values of the pixels in the hole area is negatively correlated with at least one of the following: the similarity between the pixel values of the pixels corresponding to the pixel in the hole area in the target color image, and the smoothness of the pixels corresponding to the pixel in the hole area in the target color image; the pixel value is the product of the average value or weighted summation of the smoothness values of the pixels in the image in the second neighborhood range and a second preset coefficient; the second preset coefficient represents the relationship between the smoothness of the pixel in the target color image and the pixel value of the pixel corresponding to this pixel in the target depth image.
10. The system according to claim 9, characterized in that The hole region corresponds to at least one of a weak texture region and a repeated texture region in the target color image.
11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of claims 1 to 4 above.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 4 is implemented.
