Infrared speckle depth camera and intelligent device

Abstract

The invention discloses an infrared speckle depth camera and intelligent equipment, and the camera comprises a speckle projector which is used for projecting infrared speckles; the infrared receiver is used for receiving reflection signals of the infrared speckles; and a processor which generates an infrared speckle image and a dense depth map according to the reflection signal, judges the motion state of the infrared speckle depth camera according to the dense depth map at different moments, and judges whether shielding exists or not according to the difference value between the speckle center on the infrared speckle image and surrounding pixels during motion. And obtaining a shielding area through the infrared speckle images at different moments and the overlapping range.

Description

Technical Field

[0001] The present invention relates to the technical field of depth cameras, and specifically, to an infrared speckle depth camera and an intelligent device. Background Art

[0002] In the field of modern visual perception and measurement, depth cameras play a crucial role. Traditional depth camera technologies such as structured light and lidar, although meeting the requirements of some scenarios to a certain extent, also have their respective limitations. For example, structured light technology is vulnerable to ambient light interference in complex environments, resulting in a decrease in measurement accuracy; lidar is costly and large in size, restricting its application in some miniaturized and portable devices.

[0003] As an emerging depth measurement technology, infrared speckle depth cameras have gradually attracted attention. It projects infrared speckles using a speckle projector and obtains depth information by receiving the reflected signals with an infrared receiver. However, existing infrared speckle depth cameras have deficiencies in motion state monitoring and occlusion detection. In practical applications, depth cameras often need to work in dynamic environments. For example, in mobile robot navigation and augmented reality (AR) / virtual reality (VR) devices, accurately judging the motion state of the camera is crucial for real-time data processing and scene understanding. At the same time, when there are occlusions in the camera's field of view, how to accurately detect and identify the occluded areas to avoid misleading subsequent data processing and analysis is also an urgent problem to be solved.

[0004] The disclosure of the above background art content is only for assisting in understanding the inventive concept and technical solution of the present invention, and it does not necessarily belong to the prior art of this patent application. Without clear evidence indicating that the above content was publicly available on the filing date of this patent application, the above background art should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0005] Therefore, the present invention provides an infrared speckle depth camera that can effectively identify dirt conditions, accurately eliminate incorrect depth data caused by dirt, provide more reliable and accurate depth information for mobile scenarios, and assist it in operating safely and efficiently in complex environments.

[0006] In a first aspect, the present invention provides an infrared speckle depth camera, which is characterized by comprising:

[0007] A speckle projector for projecting infrared speckles;

[0008] An infrared receiver for receiving the reflected signals of the infrared speckles;

[0009] A processor generates an infrared speckle image and a dense depth map based on the reflected signal, determines the motion state of the infrared speckle depth camera according to the dense depth maps at different times, and determines whether there is occlusion during motion based on the difference between the speckle center and surrounding pixels on the infrared speckle image, and obtains the occlusion area through the overlapping range of the infrared speckle images at different times.

[0010] Optionally, in the infrared speckle depth camera, the processor further generates a sparse depth map based on the reflected signal and fills the sparse depth value in the occlusion area.

[0011] Optionally, in the infrared speckle depth camera, the processing by the processor includes:

[0012] Step S1: Obtain a first infrared speckle image and a first dense depth map at a first time; wherein, the first dense depth map and the first infrared speckle image are pixel-aligned.

[0013] Step S2: Extract a first speckle center on the first infrared speckle image and calculate a first difference between the first speckle center and surrounding pixels; when the first difference is less than a first depth threshold, mark the speckle area where the first speckle center is located as a first alternative area.

[0014] Step S3: Obtain a second infrared speckle image and a second dense depth map at a second time; wherein, the second dense depth map and the second infrared speckle image are pixel-aligned.

[0015] Step S4: Subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points with depth values exceeding a second depth threshold in the third dense depth map exceeds a first quantity threshold, it is considered that the infrared speckle depth camera is in a motion state, and execute Step S5.

[0016] Step S5: Extract a second speckle center on the second infrared speckle image and calculate a second difference between the second speckle center and surrounding pixels; when the second difference is less than the first depth threshold, mark the speckle area where the second speckle center is located as a second alternative area.

[0017] Step S6: If the number of overlapping pixel points in the first alternative area and the second alternative area exceeds a second quantity threshold, mark the first alternative area and the second alternative area as occlusion areas.

[0018] Optionally, in the infrared speckle depth camera, Step S2 includes:

[0019] Step S21: extracting a first speckle center from the first infrared speckle image;

[0020] Step S22: taking each of the first speckle centers as the center of the circle and a circular area with a radius r as the neighborhood, calculating the grayscale value difference between the first speckle center and other pixels in the neighborhood to obtain a first difference;

[0021] Step S23: when the first difference is less than a first depth threshold, marking the speckle region where the first speckle center is located as a first candidate region; wherein the first depth threshold is dynamically adjusted according to average depth information and speckle density of the first infrared speckle image.

[0022] Optionally, the infrared speckle depth camera is characterized in that step S4 comprises:

[0023] Step S41: subtracting the first dense depth map from the second dense depth map pixel by pixel to obtain a third dense depth map, and filtering to eliminate jumping noise;

[0024] Step S42: Counting the number of pixels in the third dense depth map whose depth values ​​exceed the second depth threshold, that is, the excess number;

[0025] Step S43: if the excess quantity is greater than a first quantity threshold, it is considered that the infrared speckle depth camera is in motion, and step S5 is executed; wherein the first quantity threshold is adjusted according to the size of the first infrared speckle pattern and the speckle spacing.

[0026] Optionally, the infrared speckle depth camera is characterized in that, in step S4, motion analysis is performed on the second infrared speckle images at multiple different second moments as motion states of the multiple second infrared speckle patterns.

[0027] Optionally, the infrared speckle depth camera is characterized in that step S6 comprises:

[0028] Step S61: Calculating the number of overlapping pixels in the first candidate area and the second candidate area using a fast overlap calculation method based on an image mask;

[0029] Step S62: if the number of overlapping pixels exceeds a second number threshold, marking the first candidate area and the second candidate area as occlusion areas; the second number threshold is adjusted according to the occlusion probability and the average size of the speckle area;

[0030] Step S63: Eliminate the non-corresponding closed areas in the first candidate area and the second candidate area to obtain a final occluded area.

[0031] In a second aspect, the present invention provides an infrared speckle depth camera, which is characterized by comprising:

[0032] A dense speckle projector for projecting dense infrared speckles;

[0033] A sparse speckle projector for projecting sparse infrared speckles;

[0034] An infrared receiver for receiving the reflection signal of the dense infrared speckles to generate a dense depth map, and receiving the reflection signal of the sparse infrared speckles to generate a sparse depth map;

[0035] A processor for judging the motion state of the infrared speckle depth camera according to the dense depth maps at different times, and judging whether there is occlusion according to the difference between the speckle center and the surrounding pixels on the infrared speckle image during motion, and obtaining the occlusion area through the overlapping range of the infrared speckle images at different times; removing the dense depth values of the occlusion area and filling in the sparse depth values.

[0036] Optionally, in the infrared speckle depth camera, the processor includes the following during processing:

[0037] Step S1: Obtain a first infrared speckle image and a first dense depth map at a first time; wherein, the first dense depth map and the first infrared speckle image are pixel-aligned;

[0038] Step S2: Extract a first speckle center on the first infrared speckle image, and calculate a first difference between the first speckle center and the surrounding pixels; when the first difference is less than a first depth threshold, mark the speckle area where the first speckle center is located as a first alternative area;

[0039] Step S3: Obtain a second infrared speckle image and a second dense depth map at a second time; wherein, the second dense depth map and the second infrared speckle image are pixel-aligned;

[0040] Step S4: Subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points with depth values exceeding a second depth threshold in the third dense depth map exceeds a first quantity threshold, it is considered that the infrared speckle depth camera is in a motion state, and execute Step S5;

[0041] Step S5: Extract a second speckle center on the second infrared speckle image, and calculate a second difference between the second speckle center and the surrounding pixels; when the second difference is less than the first depth threshold, mark the speckle area where the second speckle center is located as a second alternative area;

[0042] Step S6: If the number of overlapping pixel points in the first alternative region and the second alternative region exceeds the second quantity threshold, mark the first alternative region and the second alternative region as occluded regions;

[0043] Step S7: Obtain a first sparse depth map of the first infrared speckle pattern and a second sparse depth map of the second infrared speckle pattern; wherein, the first sparse depth map is pixel-aligned with the first infrared speckle image, and the second sparse depth map is pixel-aligned with the second infrared speckle image;

[0044] Step S8: Remove the depth values of the occluded regions in the first dense depth map, and fill the corresponding depth values in the first sparse depth map into the first dense depth map; remove the depth values of the occluded regions in the second dense depth map, and fill the corresponding depth values in the second sparse depth map into the second dense depth map.

[0045] In a third aspect, the present invention provides an intelligent device, characterized by including the infrared speckle depth camera according to any one of the foregoing.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The processor in the present invention can determine the motion state of the infrared speckle depth camera based on the dense depth maps at different times. This analysis method based on the change of the depth map has higher accuracy and real-time performance compared with the traditional camera that simply relies on external sensors (such as accelerometers, gyroscopes, etc.) to determine the motion state. It starts directly from the image data obtained by the camera. The depth map can intuitively reflect the distance change of the objects within the camera's field of view. Even in a complex dynamic environment, it can accurately capture the movements such as the camera's translation and rotation, providing a reliable basis for subsequent data processing and analysis.

[0048] The present invention determines whether there is occlusion by analyzing the difference between the speckle center and the surrounding pixels on the infrared speckle image, making full use of the characteristics of the speckle image. The change in the difference between the speckle center and the surrounding pixels can sensitively reflect the appearance of the occluding object. Compared with other occlusion detection methods, it can quickly and effectively detect the occlusion situation without complex algorithms and a large amount of computing resources, greatly improving the adaptability of the camera in complex scenarios.

[0049] The present invention obtains the occluded area through the overlapping range of infrared speckle images at different times, which ingeniously utilizes the image information in the time series. As the camera moves, the images at different times contain the dynamic changes of the scene. By comparing the overlapping parts of these images, the position and range of the occluded area can be accurately determined. This positioning method not only improves the accuracy of occlusion detection, but also provides detailed and accurate information for subsequent image restoration, data compensation and other operations, which helps to improve the performance of the entire vision system.

[0050] The present invention can achieve effective measurement of occlusion with only less hardware. On the one hand, it reduces the additional occlusion possibility caused by the increase of hardware, and on the other hand, it also reduces the cost, which is beneficial to the popularization and application of the present invention. Brief Description of the Drawings

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0052] Figure 1 It is a schematic structural diagram of an infrared speckle depth camera in an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of steps when a processor processes in an embodiment of the present invention;

[0054] Figure 3 It is a flowchart of steps for marking the first alternative area in an embodiment of the present invention;

[0055] Figure 4 It is a flowchart of steps for judging the motion state in an embodiment of the present invention;

[0056] Figure 5 It is a flowchart of steps for marking the occluded area in an embodiment of the present invention;

[0057] Figure 6 It is a schematic structural diagram of another infrared speckle depth camera in an embodiment of the present invention;

[0058] Figure 7 It is a flowchart of steps when another processor processes in an embodiment of the present invention. Detailed Embodiments

[0059] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0061] An infrared speckle depth camera provided by an embodiment of the present invention aims to solve the problems existing in the prior art.

[0062] The technical solution of the present invention and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the drawings.

[0063] The present invention integrates a speckle projector, an infrared receiver and a processor, can accurately judge the motion state according to the dense depth map at different times, efficiently detect occlusion through the speckle image features and accurately determine the occlusion area, and has the advantages of high integration, stability and cost effectiveness.

[0064] Figure 1 It is a schematic structural diagram of an infrared speckle depth camera in an embodiment of the present invention. As Figure 1 shown, an infrared speckle depth camera in an embodiment of the present invention includes:

[0065] A speckle projector for projecting infrared speckles.

[0066] Specifically, the speckle projector is one of the key components in an infrared speckle depth camera, and its main function is to project infrared speckles. It uses specific optical principles to convert infrared light into a speckle pattern with random distribution characteristics or a coded speckle pattern. These speckle patterns are projected onto the surface of the target object and serve as the basis for subsequent depth information acquisition. When the speckles are projected onto the object surface, due to factors such as the shape and distance of the object surface, the reflection of the speckles will also vary. This variation contains rich information about the object surface and provides a key basis for the infrared receiver to receive the reflected signal and subsequent depth calculation. For example, in an indoor environment, when the speckles are projected onto objects such as walls and furniture, different distances and surface materials will cause the reflected speckles to exhibit different distributions and characteristics.

[0067] An infrared receiver for receiving the reflected signal of the infrared speckles.

[0068] Specifically, the infrared receiver is responsible for receiving the infrared speckle signal reflected from the object surface after being projected by the speckle projector. It is equipped with highly sensitive infrared sensing elements that can accurately capture weak reflected signals. By collecting and preliminarily processing these reflected signals, the optical signals are converted into electrical signals, providing raw data for the subsequent depth calculation and image generation by the processor. In practical applications, the performance of the infrared receiver directly affects the camera's adaptability to the environment and the accuracy of depth measurement. For example, in a relatively dark environment, a highly sensitive infrared receiver can still effectively receive the reflected signal to ensure the normal operation of the camera.

[0069] A processor that generates an infrared speckle image and a dense depth map based on the reflected signal, determines the motion state of the infrared speckle depth camera based on the dense depth maps at different times, determines whether there is occlusion based on the difference between the speckle center and the surrounding pixels on the infrared speckle image during motion, and obtains the occlusion area through the overlapping range of the infrared speckle images at different times.

[0070] Specifically, the processor is the core control and data processing unit of the entire infrared speckle depth camera. It undertakes multiple important tasks. First, based on the reflection signals received by the infrared receiver, it generates infrared speckle images and dense depth maps through complex algorithms. When generating the depth map, the processor accurately calculates the distances between the camera and each point on the object surface according to the principle of triangulation and the reflection characteristics of the speckles, thereby constructing a dense depth map that reflects the three-dimensional information of the object surface. Second, the processor can determine the motion state of the camera based on the dense depth maps at different times. By comparing the position and distance changes of the objects in the depth maps at adjacent times, it can accurately identify whether the camera is in a translational, rotational, or stationary state. When the camera is in a motion state, the processor also determines whether there is occlusion based on the difference between the speckle center and the surrounding pixels on the infrared speckle image. Because when an occluder appears, the distribution and intensity of the speckles change, and the difference between the speckle center and the surrounding pixels also changes accordingly. In addition, the processor obtains the occlusion area by analyzing the overlapping range of the infrared speckle images at different times. As the camera moves, the images at different times record the dynamic changes of the scene. By comparing the overlapping parts of these images, the processor can accurately determine the position and range of the occlusion area, providing key information for subsequent data processing and analysis.

[0071] In some embodiments, the processor also generates a sparse depth map based on the reflection signal and fills the occlusion area with sparse depth values. In the infrared speckle depth camera, the sparse depth map generated by the processor is an important data form. It is obtained by processing the reflection signals received by the infrared receiver in a more concise and efficient manner. Compared with the dense depth map that needs to accurately calculate the distances of each point on the object surface, the sparse depth map focuses more on obtaining the depth information of the key positions on the object surface. During its generation process, the processor will select points that are of great significance for describing the object structure and spatial position according to a specific algorithm, calculate the depth values of these points, and thus construct a sparse depth map. This processing method greatly reduces the data volume and has a very high credibility. When the camera identifies the occlusion area, the processor will select appropriate depth values from the sparse depth map to fill the occlusion area, making the data of the entire depth map more complete and providing strong support for subsequent analysis.

[0072] Figure 2 The following is a flowchart of the steps during the processing of the processor in the embodiments of the present invention. As Figure 2 shown, the steps during the processing of the processor in the embodiments of the present invention include:

[0073] Step S1: Obtain the first infrared speckle image and the first dense depth map at the first moment; wherein, the first dense depth map and the first infrared speckle image are pixel-aligned.

[0074] In this step, the processor first obtains the reflection signal at the first moment from the infrared receiver. Based on these signals, through complex algorithms, a first infrared speckle image and a first dense depth map are respectively generated. Here, "pixel alignment" means that the pixel points at the corresponding positions in the two images can accurately reflect the information of the same physical position, which is crucial for subsequent depth analysis and feature extraction based on the images. Through this pixel alignment, the accuracy of information correlation and comparison between different types of images can be ensured, providing a reliable data basis for subsequent processing.

[0075] Step S2: Extract the first speckle center on the first infrared speckle image, and calculate the first difference between the first speckle center and the surrounding pixels; when the first difference is less than the first depth threshold, mark the speckle area where the first speckle center is located as the first alternative area.

[0076] In this step, after obtaining the first infrared speckle image, the processor uses a specific image recognition algorithm to accurately locate the speckle centers. These speckle centers are the key feature points in the speckle pattern. Then, calculate the difference, that is, the first difference, between the gray value or other relevant attributes of each first speckle center and its surrounding pixels. This difference can reflect the feature difference between the speckle center and the surrounding area. When the first difference is less than the first depth threshold, it indicates that the feature change in this speckle area is relatively small, and there may be a special situation, such as being blocked, etc. Therefore, mark the speckle area where the first speckle center is located as the first alternative area for further analysis and judgment of whether it is an occlusion area in the future.

[0077] Step S3: Obtain a second infrared speckle image and a second dense depth map at the second moment; wherein, the second dense depth map and the second infrared speckle image are pixel-aligned.

[0078] In this step, similar to step S1, at the second moment, the processor obtains the reflection signal from the infrared receiver again, and then generates a second infrared speckle image and a second dense depth map, also ensuring their pixel alignment. Obtaining images and depth maps at different moments is to judge the motion state of the camera and the dynamic changes in the scene by comparing the data changes at different moments, which is very crucial for accurately understanding and analyzing the scene information in a complex environment.

[0079] Step S4: Subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points with depth values exceeding the second depth threshold in the third dense depth map exceeds the first quantity threshold, it is considered that the infrared speckle depth camera is in a motion state, and step S5 is executed.

[0080] In this step, by subtracting the dense depth maps at two different times, a third dense depth map is generated, which mainly reflects the change in the depth of objects in the scene between these two times. If the number of pixel points with depth values exceeding the second depth threshold in the third dense depth map exceeds the first quantity threshold, it indicates that the depth of a large number of objects in the scene has changed significantly between these two times. This change is usually caused by the movement of the camera itself. When the camera moves, its relative position to the objects in the scene changes, resulting in an obvious change in the depth map. Therefore, when this condition is met, the infrared speckle depth camera is considered to be in a moving state, and the next step is executed for further analysis.

[0081] In some embodiments, in step S4, motion analysis is performed on the second infrared speckle images at multiple different second times, as the motion state of the multiple second infrared speckle maps. Based on the original method of judging the camera motion state by two sets of depth maps, adding the operation of performing motion analysis on the second infrared speckle images at multiple different second times significantly improves the accuracy and stability of the judgment. In terms of accuracy, a single comparison is accidental. For example, a slight movement of an object in the scene at a certain moment may be misjudged as camera motion. However, when comparing multiple times, the probability of misjudgment caused by the movement of the object itself is greatly reduced because it is unlikely that the movement of the object itself at multiple times creates an illusion of depth change similar to camera motion. In terms of stability, the analysis of multiple sets of data makes the judgment result more persuasive. Even if some data is disturbed, other data can support the judgment. Moreover, the multi-time analysis can also obtain more detailed information about the camera motion, such as the motion direction and speed trend, providing a more comprehensive basis for the subsequent steps and making the entire processing flow more suitable for complex and changing actual application scenarios.

[0082] Step S5: Extract the second speckle center on the second infrared speckle image, and calculate the second difference between the second speckle center and the surrounding pixels; when the second difference is less than the first depth threshold, mark the speckle area where the second speckle center is located as the second alternative area.

[0083] In this step, after determining that the camera is in a moving state, the operation of step S2 is repeated on the second infrared speckle image. Extracting the second speckle center and calculating the second difference between it and the surrounding pixels aims to also find the areas where occlusion may exist in the image after the camera moves. When the second difference is less than the first depth threshold, mark the speckle area where the second speckle center is located as the second alternative area for subsequent comparison with the first alternative area to further determine the occlusion area.

[0084] Step S6: If the number of overlapping pixel points in the first alternative region and the second alternative region exceeds the second quantity threshold, mark the first alternative region and the second alternative region as occluded regions.

[0085] In this step, through the previous steps, the alternative regions that may be occluded at the first moment and the second moment are obtained respectively. By comparing these two alternative regions, if the number of their overlapping pixel points exceeds the second quantity threshold, it indicates that there is a continuous situation where the feature changes are not obvious in these two regions before and after the camera movement. And such regions that are relatively stable and have small feature changes at different moments are very likely to be occluded regions. Therefore, mark these two alternative regions as occluded regions, so as to perform special processing on the occluded regions subsequently, such as filling sparse depth values, etc., to ensure the integrity and accuracy of the depth map data, and provide more reliable information for subsequent data analysis and applications.

[0086] Figure 3 It is a flowchart of the steps for marking the first alternative region in an embodiment of the present invention. As Figure 3 shown, the steps for marking the first alternative region in an embodiment of the present invention include:

[0087] Step S21: Extract the first speckle center on the first infrared speckle image.

[0088] In this step, the processor uses a specific image recognition algorithm to accurately locate the speckle center in the first infrared speckle image. These speckle centers are the key feature points of the speckle pattern, and their positions and distributions contain rich scene information. For example, based on algorithms such as edge detection and corner detection, the core positions of the speckles can be accurately identified, providing a basis for subsequent analysis.

[0089] Step S22: Taking each of the first speckle centers as the center of a circle, using a circular region with a radius of r as the neighborhood, calculate the gray value difference between the first speckle center and other pixels in the neighborhood to obtain the first difference.

[0090] In this step, after determining the first speckle center, a circular neighborhood with a radius of r is constructed with it as the center. The setting of this neighborhood range is carefully considered, which not only ensures that enough surrounding pixels can be covered to obtain sufficient feature information, but also avoids introducing too much irrelevant information to interfere with the analysis due to an overly large range. In this neighborhood, the processor calculates the gray value difference between the first speckle center and other pixels. As a basic attribute of the image, the gray value difference can intuitively reflect the feature differences between the speckle center and the surrounding area. For example, if the area around the speckle center is a uniform background, the gray value difference is small; if there are obvious object edges or other structures around, the gray value difference will be large.

[0091] Step S23: When the first difference is less than the first depth threshold, mark the speckle region where the first speckle center is located as the first alternative region; wherein, the first depth threshold is dynamically adjusted according to the average depth information and speckle density of the first infrared speckle image.

[0092] In this step, after calculating the first difference, it is compared with the first depth threshold. When the first difference is less than the threshold, it indicates that the feature change of this speckle region is relatively small, and there may be special situations such as occlusion. Therefore, this speckle region is marked as the first alternative region for further determination of whether it is an occlusion region in the follow-up. The first depth threshold here is not a fixed value, but is dynamically adjusted according to the average depth information and speckle density of the first infrared speckle image. The average depth information reflects the approximate distance range of the objects in the scene. If the average depth is large, it means the scene is relatively empty, and the threshold can be appropriately relaxed; on the contrary, if the average depth is small and the scene is more compact, the threshold needs to be tightened. The speckle density also affects the threshold setting. A large speckle density means a large number of speckles per unit area and rich image details, and the threshold should be adjusted accordingly to adapt to this complex situation; the opposite is true for a small speckle density. Through this dynamic adjustment, the threshold is more adapted to the actual scene, improving the accuracy of occlusion region judgment.

[0093] Figure 4 This is a flowchart of the steps for judging the motion state in an embodiment of the present invention. As Figure 4 shown, the steps for judging the motion state in an embodiment of the present invention include:

[0094] Step S41: Subtract the first dense depth map and the second dense depth map pixel by pixel to obtain a third dense depth map, and filter to eliminate the jump noise.

[0095] In this step, the processor performs a pixel-by-pixel subtraction operation on the first dense depth map and the second dense depth map. Since the two depth maps respectively correspond to the scene depth information acquired by the camera at different times, the pixel-by-pixel subtraction can intuitively reflect the change in the depth of the objects in the scene between these two times. The obtained third dense depth map contains rich dynamic information, but at the same time, it may also introduce some jump noises due to measurement errors or environmental interference. To ensure the accuracy of subsequent analysis, it is necessary to filter the third dense depth map. Common filtering algorithms such as Gaussian filtering and median filtering can effectively smooth the image and eliminate these jump noises, enabling the third dense depth map to more accurately reflect the real depth change.

[0096] Step S42: Count the number of pixel points in the third dense depth map whose depth values exceed the second depth threshold, that is, the excess number.

[0097] In this step, after filtering the third dense depth map, the processor starts to count the number of pixel points whose depth values exceed the second depth threshold. This second depth threshold is a key indicator for judging whether the depth change is significant. If the depth value of a certain pixel point exceeds this threshold after subtracting the two depth maps, it indicates that the object at this position has undergone an obvious depth change between these two moments. By counting the number of pixel points exceeding this threshold, the degree of depth change in the entire scene can be quantified.

[0098] Step S43: If the exceeded quantity is greater than the first quantity threshold, it is considered that the infrared speckle depth camera is in a motion state, and step S5 is executed; wherein, the first quantity threshold is adjusted according to the size and speckle spacing of the first infrared speckle map.

[0099] In this step, the counted exceeded quantity is compared with the first quantity threshold. If the exceeded quantity is greater than the first quantity threshold, it indicates that between these two moments, the depths of a large number of objects in the scene have changed significantly. This large-scale depth change is usually caused by the movement of the camera itself. When the camera moves, its relative position to the objects in the scene changes, thus causing an obvious change in the depth map. Therefore, when this condition is met, it is considered that the infrared speckle depth camera is in a motion state, and the next step S5 is executed for further analysis. Here, the first quantity threshold is not fixed, but is adjusted according to the size and speckle spacing of the first infrared speckle map. The size of the first infrared speckle map reflects the field of view of the camera. The larger the size, the wider the camera's field of view, and the more objects may be included in the scene. Correspondingly, the first quantity threshold needs to be appropriately increased; otherwise, it is decreased. The speckle spacing affects the accuracy and resolution of depth measurement. A smaller speckle spacing results in more accurate depth measurement and can detect more subtle depth changes. At this time, the first quantity threshold can be appropriately reduced; the opposite is true for a larger speckle spacing. Through this dynamic adjustment based on image features, the first quantity threshold better meets the actual scene requirements and improves the accuracy of camera motion state judgment.

[0100] Figure 5 This is a flowchart of the steps for marking an occlusion area in an embodiment of the present invention. As Figure 5 shown, the steps for marking an occlusion area in an embodiment of the present invention include:

[0101] Step S61: Calculate the number of overlapping pixel points in the first alternative area and the second alternative area based on the fast overlapping calculation method of the image mask.

[0102] In this step, the image mask technology is used to quickly and accurately process the first candidate area and the second candidate area. The image mask can be understood as a binary image of the same size as the image, in which the parts that need attention are marked with specific values ​​(such as 1 for the area of ​​interest and 0 for other areas). In this step, the first candidate area and the second candidate area are respectively converted into corresponding image masks, and then the number of pixels in the overlapping part of the two areas is efficiently obtained through a pre-set fast overlap calculation algorithm. Compared with the traditional pixel-by-pixel comparison method, this method greatly improves the calculation efficiency, and while ensuring accuracy, it meets the application scenarios with high real-time requirements.

[0103] Step S62: if the number of overlapping pixels exceeds a second number threshold, marking the first candidate area and the second candidate area as occlusion areas; the second number threshold is adjusted according to the occlusion probability and the average size of the speckle area.

[0104] In this step, when the calculated number of overlapping pixels exceeds the second quantity threshold, it indicates that there is a high possibility that the two areas are occluded. This second quantity threshold is not a fixed value, it will be dynamically adjusted according to the occlusion probability and the average size of the speckle area. The occlusion probability is obtained based on a comprehensive evaluation of historical data, scene analysis and other factors. If the probability of occlusion in a certain type of scene is high, the second quantity threshold will be reduced accordingly to detect the occluded area more sensitively; otherwise it will be increased. The average size of the speckle area also affects the threshold. If the speckle area is large on average, it means that the single speckle covers a wide range. At this time, the second quantity threshold can be appropriately increased to avoid misjudgment due to the characteristics of the speckle itself; if the speckle area is small on average, the threshold will be reduced accordingly. Through this dynamic adjustment, the judgment of the occluded area is more in line with the actual situation.

[0105] Step S63: Eliminate the non-corresponding closed areas in the first candidate area and the second candidate area to obtain a final occluded area.

[0106] In this step, there may be some non-corresponding closed areas in the first candidate area and the second candidate area preliminarily marked as occluded areas. These areas may be caused by noise interference, local feature abnormalities, etc., and are not real occluded areas. Through specific image analysis algorithms, such as morphological operations (erosion, expansion, etc.), contour detection, etc., these non-corresponding closed areas can be identified and removed from the preliminarily determined occluded areas. After this step, the final accurate occluded area is obtained, which provides a reliable data basis for subsequent data processing, such as filling sparse depth values ​​and other operations.

[0107] Figure 6This is a schematic diagram of another infrared speckle depth camera in an embodiment of the present invention. As Figure 6 shown, another infrared speckle depth camera in an embodiment of the present invention includes:

[0108] A dense speckle projector for projecting dense infrared speckles.

[0109] Specifically, the dense speckle projector is responsible for projecting dense infrared speckles onto the target scene. These densely distributed speckles provide the basic pattern for subsequent acquisition of depth information. When the speckles are projected onto the object surface, the reflected light carries the depth information of the object surface. By analyzing the changes in these reflected lights, the depth values of different positions of the object can be calculated.

[0110] A sparse speckle projector for projecting sparse infrared speckles.

[0111] Specifically, the sparse speckle projector projects sparse infrared speckles onto the target scene. Different from the dense speckles, the distribution of the sparse speckles is relatively scattered. During the operation of the camera, the sparse depth map generated by the sparse speckles can be used as supplementary information for the dense depth map in some cases. Especially when dealing with occluded areas, the sparse depth map can provide additional depth data for filling.

[0112] An infrared receiver for receiving the reflection signals of the dense infrared speckles to generate a dense depth map and receiving the reflection signals of the sparse infrared speckles to generate a sparse depth map.

[0113] Specifically, the infrared receiver is a key component in the camera responsible for receiving infrared signals. It can respectively receive the dense infrared speckle signals reflected from the object surface after being projected by the dense speckle projector and the sparse infrared speckle signals reflected after being projected by the sparse speckle projector. By processing and analyzing these reflection signals, the infrared receiver can generate the corresponding dense depth map and sparse depth map. Specifically, it will calculate the distance between each point on the object surface and the camera using a specific algorithm based on information such as the intensity and phase of the reflection signal, thereby forming a depth map.

[0114] A processor for judging the motion state of the infrared speckle depth camera according to the dense depth map at different times, and judging whether there is occlusion during motion according to the difference between the speckle center and the surrounding pixels on the infrared speckle image, and obtaining the occluded area through the overlapping range of the infrared speckle images at different times; removing the dense depth values of the occluded area and filling in the sparse depth values.

[0115] Specifically, as the core processing unit of the camera, the processor undertakes a variety of important tasks.

[0116] By analyzing the dense depth maps obtained at different times, the processor can determine the motion state of the infrared speckle depth camera. For example, it can detect changes in features such as the position and shape of objects in the depth map. If there are obvious translations or rotations in the positions of objects in the depth maps at adjacent times, then it can be determined that the camera has moved.

[0117] When the camera is moving, the processor determines whether there is an occlusion situation based on the difference between the speckle center and surrounding pixels on the infrared speckle image. When there is an occlusion, the difference between the speckle center and surrounding pixels will show abnormal changes. At the same time, by comparing the infrared speckle images at different times and using the information of the image overlap range, the processor can accurately determine the position and range of the occlusion area.

[0118] After determining the occlusion area, the processor removes the depth values of this area in the dense depth map and then fills them with the depth values at the corresponding positions in the sparse depth map. The purpose of doing this is to still be able to obtain relatively accurate depth information in the presence of occlusions, and improve the measurement accuracy and reliability of the depth camera in complex scenarios.

[0119] Figure 7 It is a flowchart of steps during the processing of another processor in the embodiments of the present invention. As Figure 7 shown, the steps during the processing of another processor in the embodiments of the present invention include:

[0120] Step S1: Obtain a first infrared speckle image and a first dense depth map at a first time; wherein, the first dense depth map and the first infrared speckle image are pixel-aligned;

[0121] Step S2: Extract a first speckle center on the first infrared speckle image and calculate a first difference between the first speckle center and surrounding pixels; when the first difference is less than a first depth threshold, mark the speckle area where the first speckle center is located as a first alternative area;

[0122] Step S3: Obtain a second infrared speckle image and a second dense depth map at a second time; wherein, the second dense depth map and the second infrared speckle image are pixel-aligned;

[0123] Step S4: Subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points with depth values exceeding a second depth threshold in the third dense depth map exceeds a first quantity threshold, it is considered that the infrared speckle depth camera is in a motion state, and execute Step S5;

[0124] Step S5: Extract the second speckle center from the second infrared speckle image, and calculate the second difference between the second speckle center and the surrounding pixels; when the second difference is less than the first depth threshold, mark the speckle area where the second speckle center is located as the second alternative area;

[0125] Step S6: If the number of overlapping pixel points in the first alternative area and the second alternative area exceeds the second quantity threshold, mark the first alternative area and the second alternative area as occluded areas.

[0126] The above steps are the same as those in the previous embodiment and will not be elaborated here.

[0127] Step S7: Obtain the first sparse depth map of the first infrared speckle map and the second sparse depth map of the second infrared speckle map; wherein, the first sparse depth map is pixel-aligned with the first infrared speckle image, and the second sparse depth map is pixel-aligned with the second infrared speckle image.

[0128] In this step, obtain the first sparse depth map corresponding to the first infrared speckle map and the second sparse depth map corresponding to the second infrared speckle map. Also ensure that the first sparse depth map is pixel-aligned with the first infrared speckle image, and the second sparse depth map is pixel-aligned with the second infrared speckle image. The sparse depth map serves as supplementary data and plays a role in subsequent processing of the depth values in occluded areas.

[0129] Step S8: Remove the depth values of the occluded areas from the first dense depth map, and fill the corresponding depth values from the first sparse depth map into the first dense depth map; remove the depth values of the occluded areas from the second dense depth map, and fill the corresponding depth values from the second sparse depth map into the second dense depth map.

[0130] In this step, delete the depth values marked as occluded areas from the first dense depth map, and obtain the depth values at the corresponding positions from the first sparse depth map and fill them into the corresponding positions of the first dense depth map. Perform the same operation on the second dense depth map, that is, remove the depth values of the occluded areas and fill the depth values of the corresponding second sparse depth map. The purpose of this is to utilize the relatively reliable depth information of the sparse depth map to correct the inaccuracy of the dense depth map caused by occlusion, and improve the overall accuracy and reliability of the depth map.

[0131] This specification also provides an intelligent device, including the infrared speckle depth camera in any of the foregoing embodiments. This specification exemplarily describes the intelligent device. Those skilled in the art can understand that the description of the intelligent device in this specification is only for the description of the intelligent device to facilitate those skilled in the art to better understand the role of the infrared speckle depth camera in the intelligent device, and should not constitute a limitation on the protection scope.

[0132] The intelligent device includes an infrared speckle depth camera, a moving component, a control system, a communication module, and a data storage module.

[0133] As the core visual perception component of the intelligent device, the infrared speckle depth camera is composed of a speckle projector, an infrared receiver, and a processor. The speckle projector is responsible for projecting infrared speckles, which are projected onto the surfaces of surrounding environmental objects; the infrared receiver receives the infrared speckle signals reflected from the object surfaces and converts them into electrical signals and transmits them to the processor; the processor generates an infrared speckle image and a dense depth map based on the reflected signals, and can also judge the motion state of the camera according to the dense depth maps at different times. When in motion, it judges whether there is occlusion by analyzing the difference between the speckle center and the surrounding pixels on the infrared speckle image, obtains the occlusion area through the overlapping range of the infrared speckle images at different times, and at the same time generates a sparse depth map and fills the sparse depth value in the occlusion area.

[0134] The moving component is a key part for the intelligent device to achieve spatial position movement. It can be a wheeled structure, such as the rubber tires used in common mobile robots, and realizes linear movement, steering and other actions through motor drive; it can also be a crawler structure, which is suitable for complex terrains and can provide better grip and stability; it may also be a multi-legged structure, such as some bionic robots, which imitate the leg movement mode of animals and have better adaptability in narrow spaces or special terrains. The moving component is connected to the control system of the device and receives control instructions to adjust the motion state.

[0135] The control system is responsible for coordinating the work of each component of the device. It receives the environmental perception data from the infrared speckle depth camera and the instructions issued by the user or the preset program. After analysis and processing, it sends motion control signals to the moving component to determine parameters such as the moving direction and speed. At the same time, the control system will also interact with other possible functional modules, such as the communication module, the data storage module, etc.

[0136] The communication module realizes the information interaction between the intelligent device and external devices or systems. Common communication methods include wireless communication technologies such as Wi-Fi, Bluetooth, 4G / 5G, etc. Through the communication module, the intelligent device can transmit information such as the environmental data and motion state collected by itself to a remote server or other devices, and can also receive external control instructions and data updates.

[0137] The data storage module is used to store the data generated during the operation of the device, such as the images and depth data collected by the infrared speckle depth camera, the motion trajectory record of the device, historical operation instructions, etc. The data storage module can be an internal flash chip or a removable memory card, such as an SD card, etc., which facilitates the management and subsequent analysis of a large amount of data.

[0138] Functions of the infrared speckle depth camera

[0139] Environmental perception and navigation assistance: The generated dense depth map and infrared speckle images provide the intelligent device with the ability to perceive the surrounding environment in three dimensions. By analyzing this data, the device can identify information such as obstacles, terrain undulations, and object positions in the environment. During movement, based on this environmental perception data and combined with preset navigation algorithms, the intelligent device can plan a reasonable movement path, avoid obstacles, and achieve autonomous navigation. For example, in an indoor environment, the camera can detect obstacles such as desks, chairs, and walls, and guide the moving parts to bypass them to ensure the safe movement of the device.

[0140] Motion state monitoring and feedback: Judging its own motion state according to the dense depth maps at different times, this function is crucial for the motion control of intelligent devices. When the device is moving, the camera monitors the changes in the motion state in real time and feeds the information back to the control system. If it is detected that the motion of the device deviates, such as deviating from the predetermined path or experiencing unnecessary shaking, the control system can timely adjust the operating parameters of the moving parts to make the device return to the correct motion trajectory and ensure the stability and accuracy of the motion.

[0141] Occlusion detection and response: It can detect the occluded area, which is of great significance for the operation of intelligent devices in complex environments. When occlusion is detected, the device can take corresponding countermeasures. For example, during data acquisition, if part of the area is occluded, the device can pause the current task, try to adjust the position or angle to obtain complete data; during navigation, occlusion may mean that there are unknown obstacles ahead, and the device can slow down, re-plan the path, and avoid collisions. By filling the occluded area with sparse depth values, the integrity of the depth data can also be ensured, providing a more reliable basis for subsequent analysis and decision-making.

[0142] Improving the level of intelligence: The rich environmental information provided by the infrared speckle depth camera lays the foundation for intelligent devices to achieve more advanced intelligent functions. Based on this data, the device can perform operations such as object recognition and scene understanding, further enhancing its cognitive ability of the environment. For example, by learning and analyzing the depth features and speckle image features of different objects, the device can identify specific objects, achieve more precise interaction and task execution, thus meeting the application requirements in various complex scenarios and improving the intelligence level and practicality of the device.

[0143] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0144] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific implementation manners. Those skilled in the art can make various deformations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An infrared speckle depth camera, characterized in that: include: A speckle projector, used for projecting infrared speckles; An infrared receiver, used for receiving a reflection signal of the infrared speckle; a processor, generating an infrared speckle image and a dense depth map according to the reflection signal, judging the motion state of the infrared speckle depth camera according to the dense depth map at different times, judging whether there is occlusion according to the difference between the speckle center and surrounding pixels on the infrared speckle image during motion, and obtaining the occlusion area through the overlapping range of the infrared speckle images at different times.

2. The infrared speckle depth camera according to claim 1, characterized in that: The processor also generates a sparse depth map according to the reflection signal, and fills the occluded area with sparse depth values.

3. The infrared speckle depth camera according to claim 1, characterized in that: The processor includes: Step S1: acquiring a first infrared speckle image and a first dense depth map at a first moment; wherein the first dense depth map and the first infrared speckle image are pixel-aligned; Step S2: extracting a first speckle center from the first infrared speckle image, and calculating a first difference between the first speckle center and surrounding pixels; when the first difference is less than a first depth threshold, marking a speckle region where the first speckle center is located as a first candidate region; Step S3: Acquire a second infrared speckle image and a second dense depth map at a second moment; wherein the pixels of the second dense depth map and the second infrared speckle image are aligned; Step S4: subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points whose depth values ​​exceed the second depth threshold in the third dense depth map exceeds the first number threshold, it is considered that the infrared speckle depth camera is in motion, and step S5 is executed; Step S5: extracting a second speckle center from the second infrared speckle image, and calculating a second difference between the second speckle center and surrounding pixels; when the second difference is less than a first depth threshold, marking a speckle region where the second speckle center is located as a second candidate region; Step S6: if the number of overlapping pixels in the first candidate area and the second candidate area exceeds a second number threshold, marking the first candidate area and the second candidate area as occluded areas.

4. The infrared speckle depth camera according to claim 3, characterized in that: Step S2 includes: Step S21: extracting a first speckle center from the first infrared speckle image; Step S22: taking each of the first speckle centers as the center of the circle and a circular area with a radius r as the neighborhood, calculating the grayscale value difference between the first speckle center and other pixels in the neighborhood to obtain a first difference; Step S23: when the first difference is less than a first depth threshold, marking the speckle region where the first speckle center is located as a first candidate region; wherein the first depth threshold is dynamically adjusted according to average depth information and speckle density of the first infrared speckle image.

5. The infrared speckle depth camera according to claim 3, characterized in that: Step S4 includes: Step S41: subtracting the first dense depth map from the second dense depth map pixel by pixel to obtain a third dense depth map, and filtering to eliminate jumping noise; Step S42: Counting the number of pixels in the third dense depth map whose depth values ​​exceed the second depth threshold, that is, the excess number; Step S43: if the excess quantity is greater than a first quantity threshold, it is considered that the infrared speckle depth camera is in motion, and step S5 is executed; wherein the first quantity threshold is adjusted according to the size of the first infrared speckle pattern and the speckle spacing.

6. The infrared speckle depth camera according to claim 3, characterized in that: In step S4, motion analysis is performed on a plurality of the second infrared speckle images at different second moments as motion states of a plurality of the second infrared speckle patterns.

7. The infrared speckle depth camera according to claim 3, characterized in that: Step S6 includes: Step S61: Calculating the number of overlapping pixels in the first candidate area and the second candidate area using a fast overlap calculation method based on an image mask; Step S62: if the number of overlapping pixels exceeds a second number threshold, marking the first candidate area and the second candidate area as occlusion areas; the second number threshold is adjusted according to the occlusion probability and the average size of the speckle area; Step S63: Eliminate the non-corresponding closed areas in the first candidate area and the second candidate area to obtain a final occluded area.

8. An infrared speckle depth camera, characterized in that: include: A dense speckle projector, used for projecting dense infrared speckles; A sparse speckle projector, used for projecting sparse infrared speckles; An infrared receiver, configured to receive the reflection signal of the dense infrared speckle to generate a dense depth map, and receive the reflection signal of the sparse infrared speckle to generate a sparse depth map; A processor is provided for judging the motion state of the infrared speckle depth camera according to the dense depth images at different times, and judging whether there is occlusion according to the difference between the speckle center and the surrounding pixels on the infrared speckle image during motion, and obtaining the occlusion area through the overlapping range of the infrared speckle images at different times; removing the dense depth value of the occlusion area, and filling the sparse depth value.

9. The infrared speckle depth camera according to claim 8, characterized in that: The processor includes: Step S1: acquiring a first infrared speckle image and a first dense depth map at a first moment; wherein the first dense depth map and the first infrared speckle image are pixel-aligned; Step S2: extracting a first speckle center from the first infrared speckle image, and calculating a first difference between the first speckle center and surrounding pixels; when the first difference is less than a first depth threshold, marking a speckle region where the first speckle center is located as a first candidate region; Step S3: Acquire a second infrared speckle image and a second dense depth map at a second moment; wherein the pixels of the second dense depth map and the second infrared speckle image are aligned; Step S4: subtract the first dense depth map from the second dense depth map to obtain a third dense depth map. If the number of pixel points whose depth values ​​exceed the second depth threshold in the third dense depth map exceeds the first number threshold, it is considered that the infrared speckle depth camera is in motion, and step S5 is executed; Step S5: extracting a second speckle center from the second infrared speckle image, and calculating a second difference between the second speckle center and surrounding pixels; when the second difference is less than a first depth threshold, marking a speckle region where the second speckle center is located as a second candidate region; Step S6: if the number of overlapping pixels in the first candidate area and the second candidate area exceeds a second number threshold, marking the first candidate area and the second candidate area as occluded areas; Step S7: acquiring a first sparse depth map of the first infrared speckle image and a second sparse depth map of the second infrared speckle image; wherein the first sparse depth map and the first infrared speckle image are pixel-aligned, and the second sparse depth map and the second infrared speckle image are pixel-aligned; Step S8: removing the depth value of the occluded area in the first dense depth map, and filling the corresponding depth value in the first sparse depth map into the first dense depth map; removing the depth value of the occluded area in the second dense depth map, and filling the corresponding depth value in the second sparse depth map into the second dense depth map.

10. A smart device, characterized in that: An infrared speckle depth camera comprising any one of claims 1 to 9.

Images