Method, accelerated method, device and electronic equipment for multi-window depth image processing

Abstract

The application provides a method and an acceleration method for multi-window depth image processing, a device and an electronic device. A reference image containing a speckle pattern is obtained, the speckle pattern including a plurality of feature points; a center point position of the reference image is obtained; a size of a matching window is set according to a distance of a point on the reference image from the center point; a scene image containing the speckle pattern is obtained; and the reference image and the scene image are matched according to the set matching window to obtain depth information. Different sizes of windows are used for matching in different regions of the image according to distortion of a projector, so that the integrity of the depth is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing, and specifically to a method, acceleration method, apparatus, and electronic device for multi-window depth image processing. Background Technology

[0002] Binocular stereo vision-based depth cameras are sensitive to ambient light intensity and rely heavily on the features of the image itself. Therefore, it is difficult to extract effective and robust features in cases of insufficient lighting or lack of texture, which leads to increased matching errors or even matching failures.

[0003] The structured light method-based depth camera was proposed to address the complexity and robustness issues of the aforementioned binocular matching algorithm. Other processing steps are similar to those of binocular stereo vision. The structured light method does not rely on the object's inherent color and texture; instead, it employs active projection of known patterns to achieve fast and robust feature point matching, resulting in high accuracy and significantly expanding its applicability.

[0004] With the continuous development of depth cameras and the explosive growth of the machine vision industry, the requirements and standards for depth cameras are gradually increasing. High-precision depth information has a positive effect on the entire industry and related industries. However, current depth cameras suffer from problems such as decreased accuracy with increasing distance, blurred boundaries, and inaccurate data. Summary of the Invention

[0005] This application aims to provide a multi-window depth image processing method, acceleration method, apparatus, and electronic device. By adjusting the matching window, the probability of mismatch can be reduced to a certain extent, and by changing the filtering window, the problems of poor boundary effect, excessive smoothing, and decreased accuracy at long distances can be solved.

[0006] Ensure that the depth camera has good depth accuracy at different distances.

[0007] According to one aspect of this application, a multi-window depth image processing method is proposed, comprising:

[0008] Obtain a reference image containing a speckle pattern;

[0009] Obtain the center point position of the reference image;

[0010] The size of the matching window is set according to the distance of a point on the reference image from the center point;

[0011] Obtain a scene image containing the speckle pattern;

[0012] The scene image is matched according to the set matching window to obtain depth information.

[0013] According to some embodiments, the method further includes: obtaining the center point position of the reference image, including:

[0014] Set a matching template;

[0015] The reference image is matched according to the matching template, and the image block with the highest matching degree is used as the center image block;

[0016] The center point position of the reference image is obtained based on the center image block.

[0017] According to some embodiments, the method further includes setting the matching window to be larger the farther away from the center point.

[0018] According to some embodiments, the method further includes: the matching window contains more than 5 speckle patterns.

[0019] According to some embodiments, the method further includes: after obtaining depth information, filtering the obtained depth information using filtering windows of different sizes according to different depths.

[0020] According to some embodiments, the method further includes: the filtering window decreasing as the depth distance increases.

[0021] According to some embodiments, the method further includes:

[0022] Different post-processing parameters are set according to different depths, and these post-processing parameters are used to post-process the depth information.

[0023] According to some embodiments, the method further includes:

[0024] Obtain the center of the scene image;

[0025] Based on the obtained center point position of the reference image and the center of the scene image, parallax information is obtained through pre-calculation.

[0026] Matching is performed based on the pre-calculated disparity information.

[0027] According to another aspect of this application, a method for accelerating multi-window depth image processing is proposed, comprising:

[0028] Obtain the center of the scene image;

[0029] Based on the obtained center point position of the reference image and the center of the scene image, parallax information is obtained through pre-calculation.

[0030] Matching is performed based on the pre-calculated disparity information.

[0031] According to another aspect of this application, a multi-window depth image processing apparatus is provided, comprising:

[0032] The image acquisition module is used to acquire reference images and scene images containing speckle patterns;

[0033] The center point calculation module is used to calculate the center point position of the reference image;

[0034] The matching module is used to set the size of the matching window and match the scene image according to the set matching window to obtain depth information;

[0035] The post-processing module is used to perform post-processing on the depth information.

[0036] According to another aspect of this application, an electronic device is provided, comprising:

[0037] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in any one of the above methods.

[0038] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the aforementioned method.

[0039] According to the example embodiment of this application, by matching different sized windows for different regions in the image based on the distortion of the projector, the integrity of the depth is ensured.

[0040] According to the example embodiments of this application, by using different mean and median filter window sizes and post-processing parameters for different depth distances and based on the actual situation of the scene, the accuracy of the depth is ensured without causing excessive post-processing, thus avoiding excessive smoothing of the depth at long distances.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0043] Figure 1 A schematic diagram illustrating the principle of multi-window-based depth calculation according to an example embodiment of this application is shown.

[0044] Figure 2 A flowchart illustrating a multi-window depth image processing method according to an example embodiment of this application is shown.

[0045] Figure 3 A schematic diagram of a structured light-based acquisition module according to an example embodiment of this application is shown.

[0046] Figure 4 A reference image rendering with speckle pattern is shown according to an example embodiment of this application.

[0047] Figure 5 A schematic diagram of the center image block of a reference image according to an example embodiment of this application is shown.

[0048] Figure 6 A schematic diagram of a matching window according to an example embodiment of this application is shown.

[0049] Figure 7 The illustration shows scene images and processing effects according to an example embodiment of this application.

[0050] Figure 8 A flowchart illustrating the post-processing of a multi-window depth image according to an example embodiment of this application is shown.

[0051] Figure 9 A flowchart illustrating a method for accelerating depth images in a multi-window manner according to an example embodiment of this application is shown.

[0052] Figure 10 A block diagram of a multi-window depth image processing apparatus according to an example embodiment of this application is shown.

[0053] Figure 11 A block diagram of an electronic device according to an exemplary embodiment is shown. Detailed Implementation

[0054] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0055] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0056] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0057] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0058] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0059] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0060] In 3D depth acquisition, the most common method is similar to the method of stereo matching. Stereo matching is based entirely on image processing technology. It obtains matching points by finding the same feature points in two images, thereby obtaining the depth value. However, the method based entirely on image matching has great difficulties, and it is hard to guarantee the accuracy and correctness of the matching. Therefore, structured light technology was developed to solve the matching problem.

[0061] In depth ranging based on binocular matching, the light source is uncoded light such as ambient light or white light, and image recognition depends entirely on the feature points of the object being photographed. Therefore, matching has always been a challenge for binocular systems.

[0062] The difference with structured light-based depth ranging lies in the encoding or characterization of the projected light source. This results in an image captured by the encoded light source projected onto the object, modulated by the object's surface depth.

[0063] Because structured light sources carry many feature points or codes, they provide numerous matching corner points or direct codewords, facilitating feature point matching. In other words, structured light actively provides many feature points for matching or directly provides codewords, eliminating the need to use feature points inherent in the subject itself, thus providing better matching results.

[0064] Binocular stereo vision-based depth cameras are sensitive to ambient light intensity and rely heavily on the features of the image itself. Therefore, it is difficult to extract effective and robust features in cases of insufficient lighting or lack of texture, which leads to increased matching errors or even matching failures.

[0065] The structured light method-based depth camera was proposed to address the complexity and robustness issues of the aforementioned binocular matching algorithm. Other processing steps are similar to those of binocular stereo vision. The structured light method does not rely on the object's inherent color and texture; instead, it employs active projection of known patterns to achieve fast and robust feature point matching, resulting in high accuracy and significantly expanding its applicability.

[0066] The difference between the two in matching is that binocular vision uses the feature points of the object itself, while structured light uses feature points actively provided by the light source or direct character encoding. Another difference is that because the objects being photographed are diverse, each binocular matching faces a different image, requiring the re-extraction of feature points; while structured light projects the same pattern, and the feature points are fixed, not changing according to scene changes, thus reducing the difficulty of matching.

[0067] Structured light matching first projects structured light onto the surface of an object, then a camera receives the reflected structured light pattern. Since the received pattern will inevitably be deformed due to the three-dimensional shape of the object, spatial information about the object's surface can be calculated by observing the position and degree of deformation of the pattern on the camera. Conventional structured light methods still partially employ depth calculation based on the principle of triangulation.

[0068] Advantages of structured light-based depth cameras:

[0069] 1) Because structured light actively projects coded light, it is very suitable for use in scenes with insufficient lighting (or even no light) and lack of texture. 2) Structured light projection patterns are generally carefully designed, so high measurement accuracy can be achieved within a certain range. 3) The technology is mature, and depth images can achieve relatively high resolution.

[0070] Structured light is light with a specific pattern, such as points, lines, or surfaces. The principle of depth image acquisition based on structured light is as follows: structured light is projected onto the scene, and an image sensor captures the corresponding pattern. Since the structured light pattern deforms due to the shape of objects, the depth information of each point in the scene can be calculated using triangulation by analyzing the position of the pattern image in the captured image and the degree of deformation. Structured light measurement technology provides high-precision and fast 3D information and has been widely used in automotive, gaming, and medical fields.

[0071] A depth image, also known as a range image, is an image that uses the distance (depth) from the image acquisition device to various points in the scene as pixel values. It directly reflects the geometry of the visible surfaces of objects. A depth image can be converted into point cloud data through coordinate transformation, and point cloud data with regularity and necessary information can also be converted back into depth image data.

[0072] With the continuous development of depth cameras and the explosive growth of the machine vision industry, the standards and requirements for depth cameras are gradually increasing. High-precision depth information has a positive impact on the entire industry and related sectors. However, current depth cameras suffer from problems such as decreased accuracy with increasing distance, blurred boundaries, and inaccurate data.

[0073] To ensure good depth performance, this application proposes a multi-window depth image processing method to guarantee that the depth camera has good depth at different distances.

[0074] This application addresses the computational principles of structured light depth cameras and speckle feature points. Based on the projector distortion, it uses windows of different sizes to match different regions in the image, ensuring the integrity of the depth.

[0075] This application uses different mean and median filter window sizes and post-processing parameters for different depth distances, based on the actual scene conditions. This ensures the accuracy of the depth without causing excessive post-processing and avoids overly smoothing the depth at long distances.

[0076] The technical solution and beneficial effects of this application will be described in detail below with reference to the accompanying drawings.

[0077] Figure 1 A schematic diagram illustrating the principle of depth calculation based on structured light matching according to an example embodiment of this application is shown.

[0078] A basic structured light measurement system consists of a camera and a projector. The projector projects an encoded speckle pattern onto the scene surface, and the camera captures an image of the scene with the speckle pattern. By calculating the offset between the projected speckle and the captured speckle, the depth information of the scene can be obtained.

[0079] See Figure 1Let C be a camera and P be a projector, b be the distance between the camera and the projector, and f be the focal length of the camera. The projector P projects a speckle pattern onto a reference plane and the surface of the scene objects. From the speckles projected onto the scene surface by the projector, any point O is selected, and its corresponding projection point on the reference plane is R. The point O corresponds to point K1 in the scene image captured by the camera, and point R corresponds to point R1 in the reference image captured by the camera. Z0 is the shooting distance of the reference plane, d is the parallax between the corresponding points R1 and K1 found in the matching, and Z is the depth corresponding to point O.

[0080] According to the principle of triangle similarity, ΔRKC is similar to ΔR1K1C, therefore:

[0081]

[0082] ΔOKR is similar to ΔOCP, so we get:

[0083]

[0084] R1K1 is the disparity d, therefore the formula for calculating depth Z is as follows:

[0085]

[0086] Matching algorithms include, but are not limited to, the following:

[0087] a) Feature-based matching, such as feature point extraction matching methods (SIFT, FAST, etc.);

[0088] b) Matching based on grayscale information and local features, such as (NCC, ZNCC);

[0089] c) Matching based on global information, such as (SGBM, SGM);

[0090] Taking the NCC matching algorithm based on grayscale information and local features as an example, the depth calculation process is as follows: within a certain window, the similarity between the pixels in the scene image and the pixels in the reference image is calculated. The image is searched row by row to obtain the corresponding point with the highest similarity, thereby obtaining the disparity, which is the offset between the current pixel and the same point in the reference image. This offset and the parameters of the camera device are used to calculate the depth and obtain the specific depth value.

[0091] A reference image with a speckle pattern is formed on a plane (which can be made of projection cloth, a flat plate, etc.) on which a laser beam (including infrared, visible light, ultraviolet light, invisible light, etc.) with a fixed pattern is projected by the projector. The laser beam is perpendicular to the central axis (Z-axis) of the projector and the vertical distance is known. The speckle pattern is formed by irregularly and randomly distributed speckles.

[0092] Figure 2A flowchart illustrating a multi-window depth image processing method according to an example embodiment of this application is shown.

[0093] See Figure 2 In S201, a reference image containing the projected speckle pattern is acquired.

[0094] According to some embodiments, acquiring an image of the projected pattern requires a projector and a camera, such as an infrared projector and an infrared camera, which can be referred to as a module. Figure 3 A schematic diagram of a structured light-based acquisition module according to an example embodiment of this application is shown.

[0095] Figure 4 The illustration shows a reference image with speckle pattern according to an example embodiment of this application. The speckle pattern includes multiple speckle feature points, the distribution of which has a certain degree of randomness and is unique within a certain range. The distribution of speckles in a region forms a graphic structure, which can be called a speckle pattern. The entire speckle image is generated by projecting the speckle pattern copied from this region.

[0096] The corresponding central area pattern blocks obtained by each module are different. Figure 5 The diagram shows a schematic of the central image block of the reference image, where the area within the red box represents the central image block. The size of the central image block can be 120*100 pixels, while the reference image size is 800*1280 pixels. The size of the central image block can vary depending on the reference image size. For different modules, the distribution pattern of the speckle pattern within the central image block can differ.

[0097] In S203, the center point position of the reference image is obtained.

[0098] According to some embodiments, matching is first performed using a pre-defined template, such as... Figure 5 The central image block shown is used as a template. This template contains the projected speckle pattern and is set to a certain size according to the projector. This template is used to match the entire reference image. Only the image block without distortion can match the designed template and the matching degree is the highest. The position information of the central image block is obtained. The center point of the reference image is obtained with the center point of the image block as the center.

[0099] In S205, the size of the matching window is set according to the distance of a point on the reference image from the center point.

[0100] According to some embodiments, the overall matching window is set with the center point of the image block as the center and according to the distortion degree of different projectors to ensure that there are enough speckles in different areas. Generally, we believe that there should be at least 5 speckles (e.g., 8-10) in the matching window to achieve a good matching effect.

[0101] According to the rules of distortion, the further away from the center the area, the greater the distortion, and the greater the distance between speckles. Therefore, by judging the distance of each pixel from the center, the size of the window is determined, and the farther away the distance, the larger the window.

[0102] According to some embodiments, after initially determining the size, the speckle feature points within the window of the current pixel are counted. It is more appropriate for there to be 8-10 speckle feature points within the window. If there are fewer than 8 speckle points within the window, the window is enlarged. Conversely, if there are more than 10 speckle points within the window, the window needs to be reduced. The adjustment of the window size can be achieved through an algorithm.

[0103] For example, if the image resolution is 800*1280, the window size can be set to 13*13, 17*17, 21*21, or 25*25 pixels. Figure 6 The diagram shown is a matching window illustration according to an example embodiment of this application. It contains multiple windows of different sizes.

[0104] In S207, acquire a scene image containing the projected speckle pattern.

[0105] According to some embodiments, a scene image containing a projected speckle pattern is obtained using the same camera and projector as the one used to acquire the reference image. For example... Figure 7 The scene images shown are from an example embodiment of this application. The first image from left to right is a real scene image, and the second image is a scene image containing a projected speckle pattern.

[0106] In S209, the scene image is matched according to the set matching window to obtain depth information.

[0107] According to some embodiments, as the projector angle increases, the distortion becomes more severe. Taking the center block of the projected pattern as a reference, we use different depth algorithm matching windows on the image, processing them from small to large, depending on the different distortion conditions.

[0108] Taking the NCC (Normalization Cross-Correlation) matching algorithm, which is based on grayscale information and local features, as an example, based on the similarity between pixels in the scene image and pixels in the reference image, a matching search is performed using a search pattern that expands outward from the image center or a peer search pattern to obtain the corresponding point with the highest similarity. This yields the disparity, which is the offset between the current pixel in the scene image and the same point in the reference image. This offset, along with camera parameters (such as focal length), is used to calculate the depth value of the current point. Figure 7The rightmost image is a schematic diagram of the processing effect according to an example embodiment of this application.

[0109] According to some embodiments, during matching, feature-based matching, such as feature point extraction matching methods (SIFT, FAST, etc.), matching based on grayscale information and local features, such as (NCC, ZNCC), and matching based on global information, such as (SGBM, SGM) can be used. All three matching methods are window-based matching methods.

[0110] According to some embodiments, after obtaining depth information, the obtained depth information is filtered using a filter window of different size according to different depths. As the distance increases, the filter window decreases, with larger median and mean filter windows used at close range and smaller filter windows used at long range.

[0111] According to some embodiments, different post-processing parameters can be set according to different depths. These post-processing parameters are used to post-process the depth information. For example, in flying point filtering and hole filling algorithms, the parameters are adjusted to ensure that dense and accurate point clouds can be obtained at all distances, including both long and short distances.

[0112] Figure 8 A flowchart illustrating the post-processing of a multi-window depth image according to an example embodiment of this application is shown.

[0113] According to some embodiments, post-processing may include flying point filtering, hole filling, median filtering, mean filtering, Gaussian filtering, etc.

[0114] Taking the flying point filtering algorithm as an example, see Figure 8 In S801, the depth map input uses the depth information calculated from the depth as the input information.

[0115] In S803, the region is segmented. The depth image is traversed, and a deviation threshold is set. If the value between two adjacent pixels is greater than the deviation threshold, then the two points do not belong to the same region. For example, if the calculated depths of two adjacent pixels are 800 and 850, the set deviation value can be 20. Since the depth difference between the two pixels is 50, it is determined that the difference is greater than 20, so they are determined to be points in different regions.

[0116] In S805, region size statistics are performed. After performing a full map search in step S803 and segmenting regions with similar depths, the number of valid values ​​in each region is counted. Here, the number of valid values ​​refers to the number of points with similar depths that are not zero. Regions with fewer than 200 valid values ​​are removed, and regions with more than 200 valid values ​​are retained.

[0117] Hole filling works similarly; for example, regions with fewer than the threshold number of valid values ​​are filled, while regions with more than the threshold are left unprocessed.

[0118] Depending on the testing distance, taking a face depth camera as an example (maximum distance 1200mm):

[0119] When the depth is 300mm-500mm, the window can be set to 9*9 or 11*11, and the size of flying point filtering and hole filling can be set to 500 pixels;

[0120] When the depth is 500mm-800mm, the window can be appropriately reduced to 7*7 or 5*5, and the size of flying point filtering and hole filling can be set to 300 pixels;

[0121] When the depth is 800mm-1200mm, the window can be appropriately reduced to 5*5 or 3*3, and the size of the flying point filtering and hole filling can be set to 200 pixels.

[0122] According to some implementations, depth calculations often result in low frame rates due to their large computational load, meaning that the speed of calculating depth maps is low. Therefore, it is necessary to design a way to accelerate the process and reduce the computational load.

[0123] Figure 9 A flowchart illustrating a method for accelerating depth images in a multi-window manner according to an example embodiment of this application is shown.

[0124] When using a matching algorithm, the center of the projector pattern on the reference image can be found first by matching. The center of the pattern is then pre-calculated with the center of the scene image to obtain the disparity information of that area. Using this as reference data, matching and searching can be performed within a small range of disparities, thereby reducing the amount of computation.

[0125] In S901, the center of the reference image and the scene image is obtained.

[0126] Obtaining the center of the reference image is the same as obtaining the center point position of the reference image in step S203, and will not be repeated here. The center of the scene image can be obtained by using the geometric center of the scene image as the center, for example, by dividing the length and width in pixels by 2 to obtain the center point.

[0127] In S903, the disparity is pre-calculated.

[0128] According to some embodiments, by calculating the center point of the reference image and the scene image, the depth disparity corresponding to the center point can be obtained, which can be used as the reference disparity to reduce the disparity calculation range of the search.

[0129] In S905, the search range is set based on the pre-calculated disparity.

[0130] According to some embodiments, after obtaining the depth disparity corresponding to the center point, a search pattern is adopted that expands outward from the image center. Using the disparity corresponding to the center point as a basis, the search range of the surrounding adjacent areas is set. For example, if the disparity of the center point is 10, then the search range of the adjacent areas expanding outward from this center can be referenced to 10. The search can be conducted within a certain range. After calculating the disparity data within this range, a newly calculated disparity within that range (which can be the average of the disparities within that range) is used as a reference for the search range of the surrounding adjacent areas, and so on.

[0131] According to some embodiments, if the total parallax search range is 100, then after obtaining the reference distance, the parallax search range can be reduced to 10, reducing the overall computational load by 10 times.

[0132] It should be clearly understood that this application describes how specific examples are formed and used, but this application is not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0133] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the program that performs the functions defined by the methods provided in this application can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0134] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0135] Through the description of the exemplary embodiments, those skilled in the art will readily understand that the multi-window depth image processing method according to the embodiments of this application has at least one or more of the following advantages.

[0136] According to the example embodiment, based on the calculation principle of the structured light depth camera and speckle characteristics, different sized windows are used to match different regions in the image according to the distortion of the projector, so as to ensure the integrity of the depth. Even without post-processing, the original data has good depth information and almost no holes.

[0137] According to the example implementation, post-processing is performed on the depth at different distances, which ensures the accuracy of the depth without causing excessive post-processing and avoids making the depth at a long distance too smooth.

[0138] According to the example implementation, an acceleration strategy can be used to increase the frame rate to a high level while greatly reducing the amount of computation.

[0139] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0140] Figure 10 A block diagram of a multi-window depth image processing apparatus according to an exemplary embodiment is shown.

[0141] like Figure 10 As shown, the multi-window depth image processing apparatus may include: an image acquisition module 1010, a center point calculation module 1020, a matching module 1030, and a post-processing module 1040.

[0142] See Figure 10 Referring to the preceding description, the image acquisition module 1010 is used to acquire a reference image and a scene image containing a projected speckle pattern;

[0143] The center point calculation module 1020 is used to calculate the center point position of the reference image;

[0144] The matching module 1030 is used to set the size of the matching window and match the scene image according to the set matching window to obtain depth information;

[0145] The post-processing module 1040 is used to filter the depth information, remove flying points, and fill holes.

[0146] The device performs functions similar to those described above; other functions are described in the preceding text and will not be repeated here.

[0147] Figure 11 A block diagram of an electronic device according to an exemplary embodiment is shown.

[0148] The following reference Figure 11 To describe an electronic device 200 according to this embodiment of the present application. Figure 11 The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0149] like Figure 11 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0150] The storage unit stores program code, which can be executed by the processing unit 210, causing the processing unit 210 to perform the methods described in this specification according to various exemplary embodiments of this application.

[0151] Storage unit 220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 2201 and / or cache memory 2202, and may further include read-only memory (ROM) 2203.

[0152] Storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0153] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0154] Electronic device 200 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0155] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this application.

[0156] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0158] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0159] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0160] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for multi-window depth image processing, characterized in that, include: Obtain a reference image containing a speckle pattern, wherein the speckle pattern includes multiple feature points; Obtain the center point position of the reference image; The size of the matching window is set according to the distance of a point on the reference image from the center point, and the matching window is set to be larger the farther away from the center point; Obtain a scene image containing the speckle pattern; The reference image is matched with the scene image according to the set matching window to obtain depth information; After obtaining the depth information, filter the obtained depth information using filter windows of different sizes according to different depths.

2. The method according to claim 1, characterized in that, Obtaining the center point position of the reference image includes: Set a matching template; The reference image is matched according to the matching template, and the image block with the highest matching degree is used as the center image block; The center point position of the reference image is obtained based on the center image block.

3. The method according to claim 1, characterized in that, The matching window contains more than 5 speckles.

4. The method according to claim 1, characterized in that, The filtering window decreases as the depth increases.

5. The method according to claim 1, characterized in that, Also includes: Different post-processing parameters are set according to different depths to perform post-processing on the depth information.

6. The method according to claim 1, characterized in that, Also includes: Obtain the center of the scene image; Based on the obtained center point position of the reference image and the center of the scene image, parallax information is obtained through pre-calculation. Matching is performed based on the pre-calculated disparity information.

7. A multi-window depth image processing apparatus, characterized in that, include: The image acquisition module is used to acquire reference images and scene images containing speckle patterns; The center point calculation module is used to calculate the center point position of the reference image; The matching module is used to set the size of the matching window, which is set to be larger the farther away from the center point, and to match the scene image according to the set matching window to obtain depth information; The post-processing module is used to post-process the depth information. After obtaining the depth information, it uses a filter window of different size to filter the obtained depth information according to different depths.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method of any one of claims 1-6.

Images