Depth camera alignment performance measurement method and device and related equipment

By acquiring the depth and color images of the three-dimensional unit captured by the depth camera, calculating the coordinate offset and comparing it with the preset threshold, the inefficiency and accuracy of the alignment performance evaluation of the depth camera is solved, and fast and accurate alignment performance is achieved Measurement.

CN119071472APending Publication Date: 2024-12-03Hefei Xinming Intelligent Technology Co., Ltd.
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411114483.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to quantify the alignment effect of depth images and color images of depth cameras, resulting in inefficient evaluation and susceptible to human subjective factors, and the results are inaccurate.

Method used

By obtaining the depth images and color images of multiple stereo units captured by the depth camera, the coordinate information of each stereo unit in the image is determined, the coordinate offset is calculated, and it is compared with the preset offset threshold to generate the depth camera. Alignment performance measurement results.

Benefits of technology

Fast and accurate quantitative evaluation of depth camera alignment performance is achieved, reducing the influence of human subjective factors and improving the accuracy of evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071472A_ABST
    Figure CN119071472A_ABST
Patent Text Reader

Abstract

The invention provides a depth camera alignment performance measurement method and device and related equipment, and relates to the technical field of camera testing. The method comprises the steps that a depth image and a color image obtained by shooting a shot object through a depth camera are obtained, and the shot object comprises a plurality of three-dimensional units located at different positions; determining corresponding coordinate information of each three-dimensional unit in the depth image and the color image; and generating an alignment performance measurement result of the depth camera according to the corresponding coordinate information of each three-dimensional unit in the depth image and the color image. According to the invention, the alignment performance of the depth camera can be rapidly and accurately subjected to quantitative evaluation.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Depth cameras, also known as 3D (Three Dimensions) cameras, can obtain three-dimensional geometric information of objects in the scene, including the position and depth information of the objects, through specific technical means. These technical means include but are not limited to structured light solutions, binocular solutions, time-of-flight (TOF) solutions, etc. The core of these solutions is to collect depth images and color images. Depth images can provide three-dimensional position and size information of the photographed object, while color images can show the color and texture information of the photographed object. Combining these two images can provide rich visual information.

[0003] Since different depth cameras have differences in optical devices, production assembly, depth calibration and alignment algorithms, the alignment effect of depth images and color images of depth cameras also varies greatly. Before the depth camera leaves the factory, the alignment effect of the depth image and color image of the depth camera becomes an important indicator to measure the performance of the depth camera.

[0004] At present, the related technologies mainly rely on experienced engineers to evaluate the alignment effect of depth images and color images. This method is not only inefficient, but also easily affected by human subjective factors, resulting in inaccurate evaluation results. Therefore, how to provide a method for quantitatively evaluating the alignment effect of depth cameras is an urgent technical problem in this field.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a method, an apparatus and related equipment for measuring the performance of a depth camera alignment, which are at least used to solve the technical problem in the related art that the performance of a depth camera alignment cannot be quantitatively evaluated.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a method for measuring the alignment performance of a depth camera is provided, comprising: obtaining a depth image and a color image obtained by photographing a photographed object with a depth camera, wherein the photographed object includes a plurality of stereo units located at different positions; determining coordinate information corresponding to each stereo unit in the depth image and the color image; and generating an alignment performance measurement result of the depth camera according to the coordinate information corresponding to each stereo unit in the depth image and the color image.

[0009] In some embodiments, the alignment performance measurement result of the depth camera is generated according to the coordinate information corresponding to each stereo unit in the depth image and the color image, including: calculating the coordinate offset of each stereo unit according to the coordinate information corresponding to each stereo unit in the depth image and the color image; comparing the coordinate offset of each stereo unit with a preset offset threshold; and generating the alignment performance measurement result of the depth camera according to the comparison result.

[0010] In some embodiments, the coordinate offset of each stereo unit is compared with a preset offset threshold, including: determining the field of view angle range of each stereo unit, wherein different field of view angle ranges correspond to different offset thresholds; and comparing the coordinate offset of each stereo unit with the offset threshold of the corresponding field of view angle range.

[0011] In some embodiments, determining the coordinate information corresponding to each stereo unit in the depth image and the color image includes: determining the deformation information of each stereo unit in the depth image and the color image; determining whether the depth image and the color image have preset deformation conditions based on the deformation information of each stereo unit in the depth image and the color image; if both the depth image and the color image meet the preset deformation conditions, obtaining the coordinate information of the preset position on each stereo unit in the depth image and the color image as the coordinate information corresponding to each stereo unit in the depth image and the color image.

[0012] In some embodiments, the three-dimensional unit is a geometric body with a regular shape.

[0013] In some embodiments, the three-dimensional unit includes at least one of the following: a cylinder, a prism, a cone, and a sphere.

[0014] According to another aspect of the present disclosure, a depth camera alignment performance measurement device is also provided, including: an image acquisition module, used to obtain a depth image and a color image obtained by a depth camera photographing a photographed object, wherein the photographed object includes a plurality of stereo units located at different positions; an image processing module, used to determine the coordinate information corresponding to each stereo unit in the depth image and the color image; and a performance analysis module, used to generate an alignment performance measurement result of the depth camera according to the coordinate information corresponding to each stereo unit in the depth image and the color image.

[0015] According to another aspect of the present disclosure, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned depth camera alignment performance measurement methods by executing the executable instructions.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the depth camera alignment performance measurement method described in any one of the above is implemented.

[0017] According to another aspect of the present disclosure, a computer program product is further provided, including: a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the depth camera alignment performance measurement method described in any one of the above is implemented.

[0018] The depth camera alignment performance measurement method, device and related equipment provided in the embodiments of the present disclosure, through the depth camera to shoot stereo units located at multiple different positions on the photographed object, obtain corresponding depth images and color images, determine the corresponding coordinate information of each stereo unit in the depth image and color image, and then generate the alignment performance measurement result of the depth camera according to the corresponding coordinate information of each stereo unit in the depth image and color image. Through the embodiments of the present disclosure, the alignment performance of the depth camera can be quickly and accurately quantitatively evaluated.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 A schematic diagram of a depth camera measurement system in an embodiment of the present disclosure is shown;

[0022] Figure 2 A flow chart of a method for measuring the performance of a depth camera alignment according to an embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic diagram of a test chart including a plurality of cylinders in an embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram of a test chart including a plurality of triangular prisms in an embodiment of the present disclosure is shown;

[0025] Figure 5 A schematic diagram of a test chart including a plurality of quadrangular prisms in an embodiment of the present disclosure is shown;

[0026] Figure 6A schematic diagram of another test chart including a plurality of quadrangular prisms in an embodiment of the present disclosure is shown;

[0027] Figure 7 A flow chart of another method for measuring the performance of a depth camera alignment according to an embodiment of the present disclosure is shown;

[0028] Figure 8 A flow chart of another method for measuring the performance of depth camera alignment according to an embodiment of the present disclosure is shown;

[0029] Fig. 9 A schematic diagram showing the result of numbering each stereoscopic unit on a test chart in an embodiment of the present disclosure is shown;

[0030] Fig.10 Schematic diagrams showing two distortion phenomena of a depth camera in an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram showing a de-distortion effect of a depth camera in an embodiment of the present disclosure is shown;

[0032] Fig.12 A schematic diagram showing a de-distortion detection result of a depth camera in an embodiment of the present disclosure is shown;

[0033] Fig.13 A schematic diagram showing the alignment effect of a depth camera in an embodiment of the present disclosure is shown;

[0034] Fig.14 A schematic diagram showing an analysis of alignment measurement results of a depth camera in an embodiment of the present disclosure is shown;

[0035] Fig.15 A flowchart of the alignment performance analysis of a depth camera in an embodiment of the present disclosure is shown;

[0036] Fig.16 A schematic diagram of a depth camera alignment performance measurement device according to an embodiment of the present disclosure is shown;

[0037] Fig.17 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0039] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0040] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0041] Figure 1 FIG. 2 is a schematic diagram of a depth camera measurement system in an embodiment of the present disclosure. Figure 1 As shown, the measurement system includes a depth camera 101, a test slide rail 102 and a test chart 103. The measurement distance 104 is the distance from the depth camera 101 to the test chart 103, which usually needs to be within the measurement range of the depth camera.

[0042] The depth camera 101 may be any camera capable of acquiring depth information and color information of the photographed object. In some embodiments, the scheme adopted includes but is not limited to any of the following: structured light scheme, binocular scheme, and light flight time scheme. The test rail 102 is used to install the depth camera 101, and the depth camera 101 can move on the test rail 102. The test chart 103 may be a customized chart, on which a plurality of stereo units with different spatial positions are arranged, and the stereo units may include but are not limited to at least one of the following: a cylinder, a prism, a cone, and a sphere.

[0043] Those skilled in the art will know that Figure 1 The number of depth cameras, test rails, and test charts in the embodiment is only for illustration, and any number of depth cameras, test rails, and test charts may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0044] Under the above system architecture, a method for measuring the performance of a depth camera alignment is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities.

[0045] In some embodiments, the depth camera alignment performance measurement method provided in the embodiments of the present disclosure may be executed by the depth camera of the above-mentioned system architecture; in other embodiments, the depth camera alignment performance measurement method provided in the embodiments of the present disclosure may be executed by an image processing device (the image processing device may be a terminal or a server) that communicates with the depth camera in the above-mentioned system architecture; in other embodiments, the depth camera alignment performance measurement method provided in the embodiments of the present disclosure may be implemented by the image processing device and the depth camera in the above-mentioned system architecture through interaction.

[0046] Figure 2 A flow chart of a method for measuring the performance of a depth camera alignment according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the depth camera alignment performance measurement method provided in the embodiment of the present disclosure includes the following steps:

[0047] S202, obtaining a depth image and a color image obtained by photographing a photographed object with a depth camera, wherein the photographed object includes a plurality of stereo units located at different positions.

[0048] It should be noted that the photographed object in the embodiment of the present disclosure can be any object to be photographed, and can be various test charts used to test camera performance, or various objects photographed during the actual use of the camera. In the embodiment of the present disclosure, the object to be photographed by the depth camera includes multiple stereo units located at different positions, and the alignment performance of the depth camera on the depth image and the color image can be quantitatively evaluated. The alignment performance of the depth camera on the depth image and the color image is evaluated based on the coordinate offset of each stereo unit on the depth image and the color image.

[0049] To facilitate subsequent data analysis, in some embodiments, the plurality of three-dimensional units on the object under test may be geometric bodies with regular shapes.

[0050] Further, in some embodiments, the three-dimensional unit in the embodiments of the present disclosure may include but is not limited to at least one of the following: a cylinder, a prism, a cone, and a sphere. When the three-dimensional unit is a prism, it may be a triangular prism, a quadrangular prism, a hexagonal prism, or other multi-prisms. Figure 3 A test chart including a plurality of cylinders in an embodiment of the present disclosure is shown; Figure 4 A test chart including a plurality of triangular prisms in an embodiment of the present disclosure is shown; Figure 5 A test chart including a plurality of quadrangular prisms in an embodiment of the present disclosure is shown.

[0051] It should be noted that the arrangement of the multiple three-dimensional units can be arbitrarily arranged according to actual needs, and the embodiments of the present disclosure do not specifically limit this. Figure 6A schematic diagram of another test chart including a plurality of quadrangular prisms in an embodiment of the present disclosure is shown. Figure 6 As shown, in order to more accurately understand the alignment effect of the depth image and the color image, the entire test chart can be filled with stereo units. The alignment effect of the depth image and the color image at each stereo unit position can be understood through the coordinate offset of the corresponding coordinates of the stereo unit.

[0052] The depth camera in the embodiments of the present disclosure may be any camera that can obtain a depth image and a color image by photographing the photographed object; the depth image is used to reflect the distance from each point on the photographed object to the depth camera; the color image is used to reflect the color information of each point on the photographed object.

[0053] In most cases, the depth image is a grayscale image, and the grayscale value of each pixel in the grayscale image represents the distance from each point on the photographed object to the depth camera; however, in some embodiments, a color image may be used to reflect the distance from each point on the photographed object to the depth camera, and thus, the depth image may also be a color image.

[0054] In the disclosed embodiment, the color image obtained by the depth camera shooting the photographed object can be any one of a black and white image, a grayscale image or a color image. Each pixel in a black and white image has only two possible color values; each pixel in a grayscale image represents a grayscale level, which is an intermediate value between black and white, representing different degrees of gray; each pixel in a color image has different color information, and the types of color images are different due to different color models. The color image in the disclosed embodiment can be but is not limited to: RGB images, YUV images, CMYK images, etc.; black and white images are the simplest, with the smallest amount of data and fast processing speed; grayscale images provide visual information of brightness levels; and color images provide the richest color information. For different application scenarios, the depth camera may collect different images.

[0055] S204: Determine the coordinate information corresponding to each stereo unit in the depth image and the color image.

[0056] It should be noted that, for stereo units of different shapes, different position points may be selected on the stereo units to obtain corresponding coordinate information of different stereo units in the depth image and the color image.

[0057] In some embodiments, when the stereo unit is a cylinder, the coordinate information of the center (or center of mass) of the bottom surface of the cylinder in the depth image and the color image can be obtained; in other embodiments, when the stereo unit is a triangular prism, the coordinate information of the corner points of the bottom surface of the triangular prism in the depth image and the color image can be obtained.

[0058] S206 , generating an alignment performance measurement result of the depth camera according to the corresponding coordinate information of each stereo unit in the depth image and the color image.

[0059] In a specific implementation, each stereo unit on the photographed object can be numbered, and for the stereo units with the same number, their first coordinate information in the depth image and the second coordinate information in the color image are calculated respectively, and the offset between the first coordinate information and the second coordinate information is calculated. The larger the offset, the worse the alignment effect of the depth camera; the smaller the offset, the better the alignment effect of the depth camera.

[0060] When evaluating the alignment effect of the depth camera, it can be determined based on the coordinate offset of a single stereo unit in the depth image and the color image, or it can be determined based on the coordinate offset of multiple stereo units in the depth image and the color image. For example, the number of stereo units on the photographed object that does not exceed a preset offset threshold or the ratio of the number to the total number (referred to as the number ratio) is counted, and the alignment effect of the depth camera is determined based on the number or the number ratio.

[0061] Considering that the depth camera's own problems may cause the depth image or color image captured by the depth camera to have a large deformation (such as barrel distortion or pincushion distortion), the image is corrected through certain technical means to remove the distortion.

[0062] In order to avoid excessive coordinate offset of the same stereo unit in the depth image and color image due to image distortion, deformation detection may be performed on the depth image and color image taken by the depth camera before determining the corresponding coordinate information of each stereo unit in the depth image and color image, and when the depth image and color image meet the preset deformation conditions, the corresponding coordinate information of each stereo unit in the depth image and color image may be determined.

[0063] Thus, in some embodiments, if Figure 7 As shown, the depth camera alignment performance measurement method provided in the embodiment of the present disclosure can determine the corresponding coordinate information of each stereo unit in the depth image and the color image through the following steps:

[0064] S2042, determining deformation information of each stereo unit in the depth image and the color image;

[0065] S2044, determining whether the depth image and the color image have preset deformation conditions according to the deformation information of each stereo unit in the depth image and the color image;

[0066] S2046: If both the depth image and the color image satisfy the preset deformation condition, obtain coordinate information of a preset position on each stereo unit in the depth image and the color image as corresponding coordinate information of each stereo unit in the depth image and the color image.

[0067] It should be noted that the shape of the stereo unit is different, and the method of determining its deformation information is also different. In some embodiments, when the stereo unit is a cylinder, the ellipticity of the bottom surface of the cylinder can be calculated, and the ellipticity can be used as deformation information to characterize its deformation; if the deformation of each stereo unit on the photographed object is too serious (such as the ellipticity exceeds the preset threshold), it is determined that the depth image and the color image are seriously deformed. In this case, the calculated coordinate offset information is not suitable for evaluating the alignment effect of the depth camera. Therefore, through the above embodiment, only for the depth image and color image that meet the preset deformation conditions, the corresponding coordinate information of each stereo unit in the depth image and the color image is obtained, and the coordinate offset is calculated, which can more accurately reflect the alignment effect of the depth camera.

[0068] It should also be noted that the above-mentioned preset position can be a point on the three-dimensional unit that can represent its position, and the corresponding preset position may be different depending on the shape of the three-dimensional unit. If the three-dimensional unit is a cylinder, the preset position can be the center (or centroid) of the bottom surface of the cylinder; if the three-dimensional unit is a triangular prism, the preset position can be the corner point of the bottom surface of the triangular prism.

[0069] In some embodiments, if the depth image and the color image do not meet the preset deformation conditions, the depth image and the color image captured by the depth camera can meet the preset deformation conditions by adjusting the internal parameters and / or external parameters of the depth camera, or replacing certain parts of the depth camera.

[0070] In some embodiments, Figure 8 As shown, the depth camera alignment performance measurement method provided in the embodiment of the present disclosure can generate the depth camera alignment performance measurement result according to the coordinate information of each stereo unit in the depth image and the color image through the following steps:

[0071] S2062, calculating a coordinate offset of each stereo unit according to coordinate information of each stereo unit in the depth image and the color image;

[0072] S2064, comparing the coordinate offset of each stereo unit with a preset offset threshold;

[0073] S2066: Generate an alignment performance measurement result of the depth camera according to the comparison result.

[0074] In a specific implementation, the above-mentioned preset offset threshold is a pre-configured coordinate offset threshold. After calculating the coordinate offset of each stereo unit, the coordinate offset of each stereo unit is compared with the preset offset threshold to determine the alignment effect (alignment performance) of the depth camera, and the alignment performance of the depth camera can be evaluated from multiple dimensions.

[0075] For a depth camera, different FOV ranges have different corresponding distortion conditions. Generally, light will be distorted to a greater extent when it is far away from the center of the lens, resulting in the captured image not being consistent with the geometric shape of the actual object. Therefore, in order to more accurately evaluate the alignment performance of the depth camera, in some embodiments, the depth camera alignment performance measurement method provided in the embodiments of the present disclosure can configure different offset thresholds for different FOV ranges. Figure 8 As shown, the above S2064 can be implemented by the following steps:

[0076] S2064a, determining the field of view angle range of each stereo unit, wherein different field of view angle ranges correspond to different offset thresholds;

[0077] S2064b, comparing the coordinate offset of each stereo unit with the offset threshold of the corresponding field of view angle range.

[0078] It should be noted that the field of view angle range refers to the field of view that an observer can see. The specific numerical range of the field of view angle will vary in different application fields.

[0079] by Figure 3 Taking the test chart shown in the figure as an example, a specific example is listed below to explain in detail the depth camera alignment performance measurement method provided in the embodiment of the present disclosure.

[0080] In specific implementation, the test chart used can meet the following conditions:

[0081] 1) The dimensions of the test chart are 500×400×10mm in length, width and thickness. It can be made of white acrylic or wood, and has the characteristics of flatness and diffuse reflection.

[0082] 2) Multiple cylinders with a height of 5mm and a diameter of 5mm are set on the diagonal line. Dark acrylic or wood can be used. They are flat and diffusely reflective. The number of cylinders is 13. Figure 3 The stereo units on the test chart shown are numbered to obtain Fig. 9 Test chart shown.

[0083] It should be noted that in the disclosed embodiment, the actual size of the test chart, the height of the cylinders, and the number are not specifically limited. In actual tests, the size information of the test chart can be determined based on the detection distance and FOV (Field of View) of the depth camera to be tested.

[0084] The measurement process of the depth camera alignment effect is as follows:

[0085] 1) Test environment installation: The 3D depth camera 101 is fixed on the test guide rail 102 and parallel to the test chart 103, and is slid to the center position just above the alignment chart. The distance to the background surface of the alignment chart is measured with a laser rangefinder, which is defined as Z back. The distance from the 3D depth camera to the upper surface of the cylinder of the alignment chart is defined as Z circle.

[0086] 2) Collect image data: After the depth camera starts the alignment algorithm, shoot the alignment card module within the depth camera FOV. The outermost cylinder of the alignment card must be located at the four corners of the image. Then output a depth image and an RGB color image.

[0087] 3) Image data analysis: Select the depth of the Z circle of interest in the depth image for segmentation analysis (background Z back, cylinder Z circle) to ensure the depth information of the cylinder, then number each circle and calculate the ellipticity and center of mass coordinates of each circle;

[0088] Select the gray value of interest for segmentation analysis in the RGB color image (white background, dark cylinder). Similarly, number each circle, which corresponds to the number of the circle in the depth image, and calculate the ellipticity and centroid coordinates of each circle.

[0089] 4) Analysis of the de-distortion effect: The optical characteristics of the camera lens and the lens, that is, the lens is thick in the middle and thin at the edges, are the source of distortion. Usually, light will be distorted to a greater extent away from the center of the lens, resulting in the image being captured not matching the geometry of the actual object. Distortion generally occurs Fig.10 Barrel and pincushion distortion shown.

[0090] Usually, both infrared IR cameras and color RGB cameras have distortion. In order to solve these distortion problems, distortion correction can be used. The basic principle is to obtain the camera's internal parameters (such as focal length, principal point coordinates, etc.) and distortion coefficients through camera calibration, and then use these parameters to correct the image to eliminate the distortion. This can significantly improve the image quality and accuracy, and provide a better foundation for subsequent alignment processing and analysis. However, whether the camera calibration method is effective in removing the distortion successfully requires testing its distortion removal effect.

[0091] For the depth image and color image obtained in S1006, when calculating the ellipticity of all circles, it is necessary to first obtain the minor axis and major axis of each circle. The calculation formula of the ellipticity is: ellipticity = minor axis / major axis. The de-distortion effect can be obtained from the ellipticity:

[0092] a. The closer the ellipticity is to 1, the more standard the circle is, that is, the less the image is distorted, which means the dedistortion effect is better;

[0093] b. A standard value is preset before detection. When the ellipticity of a circle in the identified depth image or color image is less than the preset threshold (such as 0.9), it can be judged that the de-distortion effect does not meet the standard;

[0094] c. According to the distortion principle, the four corners of the image are most severely deformed, such as Fig.11 The following figure shows the detection results of the de-distortion effect of the depth camera. The left side shows the ideal de-distortion effect, and the right side shows the unsatisfactory de-distortion effect. Fig.11 The 13 circles shown on the left and right have corresponding ellipticities as follows: Fig.12 As shown, it can be seen that the ellipticity of the circles numbered 1, 2, 12 and 13 is lower than the preset threshold (such as 0.9), and the de-distortion effect is not ideal.

[0095] 5) Analysis of alignment effect: First, understand the depth camera's depth image and RGB color image alignment algorithm implementation, and follow the industry's common steps:

[0096] ① First calibrate the camera, calibrate the infrared IR camera and the color RGB camera of the depth camera to determine the relationship between the two. This usually involves taking a set of images and using these images to calculate the internal and external parameters between the two cameras.

[0097] ② Processing of the depth image, restoring each pixel in the depth image to the depth coordinate system, and then converting it to the world coordinate system. Specifically, the 2D point on the depth image (such as point A) will be converted to a 3D point in the world coordinate system (set as point A), and then this 3D point will be projected onto the color image to obtain the coordinates (x, y) of point A on the RGB image.

[0098] ③ Processing of color images: extract the color information of point A on the RGB image, that is, the values ​​of the three RGB channels, and assign these values ​​to the coordinates of point A on the original depth map. In this way, the depth image can display color information.

[0099] ④ Alignment and registration: Since the spatial coordinate systems of color images and depth images are different, a registration operation is required so that the pixels on the color image and the pixels on the depth image can maintain a one-to-one correspondence. This is usually achieved through some image processing techniques, such as feature matching, 3D reconstruction, etc.

[0100] Due to factors such as the optical devices, production assembly, depth calibration and alignment processing algorithms of 3D depth cameras of different solutions, there will be great differences in the registration and alignment of depth maps and color maps. Even if the solutions are the same, the consistency of the optical devices or the stability of the production assembly will also cause great differences in the registration and alignment of depth maps and color maps. Therefore, it is very important to detect the alignment effect of depth maps and color maps to evaluate the accuracy of their alignment effect.

[0101] Number each circle in the depth image and the color image, and there is a one-to-one correspondence between the circles in the depth image and the circles in the color image. Calculate the centroid coordinates of each circle in the depth image and the centroid coordinates of each circle in the color image, and then subtract the centroid coordinates at the same position to get the alignment offset, in units of pixels. Judge the alignment effect:

[0102] a. The smaller the alignment offset pixel, the better the alignment effect;

[0103] b. Due to the influence of distortion, the alignment offset is larger at the edge, and the alignment effect is worse;

[0104] c. A specification value (coordinate offset threshold) is preset before detection. The coordinate offset threshold corresponding to the 0.5 FOV viewing angle of the preset specification center is 2 pixels (2 pixels); the coordinate offset threshold corresponding to the 0.5 FOV viewing angle of the preset specification center is 5 pixels (5 pixels); when the alignment offset pixel is greater than the preset specification, it can be judged that the alignment effect does not meet the standard.

[0105] Fig.13 The figure shows the alignment effect of the depth camera on the depth image and the color image. The left side shows the centroid analysis result of the depth image, and the right side shows the centroid analysis result of the color image. Fig.14 The figure shows the offset of the coordinate information of 13 cylinders on the depth image and color image. The inner circle of the 0.5 FOV of the depth camera is numbered as 5, 6, 7, 8, 9; the outer circle of the 0.5 FOV of the depth camera is numbered as 1, 2, 3, 4, 10, 11, 12, 13.

[0106] Fig.15 A flow chart of the alignment performance analysis of a depth camera in an embodiment of the present disclosure is shown as follows: Fig.15 As shown, including:

[0107] S1502, collect images, process the color image through S1504a to S1516a, and process the depth image through S1504b to S1516b.

[0108] The processing flow of color images is:

[0109] S1504a, obtaining the collected color image.

[0110] S1506a, performing segmentation processing on the color image.

[0111] S1508a: Number each circle segmented from the color image and find the ellipticity of each circle.

[0112] S1510a, determining whether the ellipticity of each circle in the color image is greater than a preset threshold.

[0113] S1512a, determining that the color image dedistortion effect is poor.

[0114] S1514a, determining that the color image dedistortion effect is good.

[0115] S1516a, find the centroid coordinates of each circle in the color image.

[0116] The processing flow of the depth image is:

[0117] S1504b, obtaining the collected depth image.

[0118] S1506b, performing segmentation processing on the depth image.

[0119] S1508b: Number each circle segmented from the depth image and calculate the ellipticity of each circle.

[0120] S1510b, determining whether the ellipticity of each circle in the depth image is greater than a preset threshold.

[0121] S1512b, determining that the depth image dedistortion effect is poor.

[0122] S1514b, determine that the depth image dedistortion effect is good.

[0123] S1516a, find the centroid coordinates of each circle in the depth image.

[0124] After obtaining the centroid coordinates of each circle in the color image and the centroid coordinates of each circle in the depth image, perform the following steps:

[0125] S1518, subtract the corresponding centroid coordinates in the color image and the depth image.

[0126] S1520, output alignment effect.

[0127] Based on the same inventive concept, the present disclosure also provides a depth camera alignment performance measurement device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0128] Fig.16 A schematic diagram of a depth camera alignment performance measurement device according to an embodiment of the present disclosure is shown. Fig.16 As shown, the device includes: an image acquisition module 161, an image processing module 162 and a performance analysis module 163.

[0129] Among them, the image acquisition module 161 is used to obtain the depth image and color image obtained by the depth camera shooting the photographed object, wherein the photographed object includes multiple stereo units located at different positions; the image processing module 162 is used to determine the corresponding coordinate information of each stereo unit in the depth image and the color image; the performance analysis module 163 is used to generate the alignment performance measurement result of the depth camera according to the coordinate information of each stereo unit in the depth image and the color image.

[0130] In some embodiments, the performance analysis module 163 may specifically include: an offset calculation module 1631, used to calculate the coordinate offset of each stereo unit according to the coordinate information of each stereo unit in the depth image and the color image; an offset comparison module 1632, used to compare the coordinate offset of each stereo unit with a preset offset threshold; an alignment measurement result generation module 1633, used to generate an alignment performance measurement result of the depth camera according to the comparison result.

[0131] In some embodiments, the offset comparison module 1632 is further used to: determine the field of view angle range of each stereo unit, where different field of view angle ranges correspond to different offset thresholds; and compare the coordinate offset of each stereo unit with the offset threshold of the corresponding field of view angle range.

[0132] In some embodiments, the above-mentioned image processing module 162 may specifically include: a deformation information determination module 1621, used to determine the deformation information of each stereo unit in the depth image and the color image; a deformation detection module 1622, used to determine whether the depth image and the color image have preset deformation conditions based on the deformation information of each stereo unit in the depth image and the color image; a coordinate information determination module 1623, used to obtain the coordinate information of the preset position on each stereo unit in the depth image and the color image if both the depth image and the color image meet the preset deformation conditions, as the corresponding coordinate information of each stereo unit in the depth image and the color image.

[0133] In some embodiments, the three-dimensional unit in the embodiments of the present disclosure is a geometric body with a regular shape.

[0134] Furthermore, in some embodiments, the three-dimensional unit in the embodiments of the present disclosure may include but is not limited to at least one of the following: a cylinder, a prism, a cone, and a sphere.

[0135] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0136] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0137] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned depth camera alignment performance measurement methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0138] Refer to the following Fig.17 1700 according to this embodiment of the present disclosure is described. Fig.17 The electronic device 1700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0139] like Fig.17 As shown, the electronic device 1700 is in the form of a general computing device. The components of the electronic device 1700 may include but are not limited to: at least one processing unit 1710, at least one storage unit 1720, and a bus 1730 connecting different system components (including the storage unit 1720 and the processing unit 1710).

[0140] The storage unit stores a program code, and the program code can be executed by the processing unit 1710, so that the processing unit 1710 executes the steps described in the "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1710 can execute the following steps of the above method embodiment: obtaining a depth image and a color image obtained by a depth camera photographing a photographed object, wherein the photographed object includes multiple stereo units located at different positions; determining the coordinate information corresponding to each stereo unit in the depth image and the color image; and generating a depth camera alignment performance measurement result based on the coordinate information corresponding to each stereo unit in the depth image and the color image.

[0141] The storage unit 1720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 17201 and / or a cache storage unit 17202 , and may further include a read-only storage unit (ROM) 17203 .

[0142] The storage unit 1720 may also include a program / utility 17204 having a set (at least one) of program modules 17205, such program modules 17205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0143] Bus 1730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0144] The electronic device 1700 may also communicate with one or more external devices 1740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1700, and / or communicate with any device that enables the electronic device 1700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1750. Furthermore, the electronic device 1700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1760. As shown, the network adapter 1760 communicates with other modules of the electronic device 1700 via a bus 1730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0145] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0146] Based on the same inventive concept, a computer-readable storage medium is also provided in an embodiment of the present disclosure, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned depth camera alignment performance measurement methods is implemented. Since the principle of solving the problem in the computer-readable storage medium embodiment is similar to that in the above-mentioned method embodiment, the implementation of the computer-readable storage medium embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0147] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0149] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0150] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0151] Based on the same inventive concept, a computer program product is also provided in an embodiment of the present disclosure, including a computer program product, including: a computer program or an instruction, which, when executed by a processor, implements any one of the depth camera alignment performance measurement methods in the above method embodiments. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0152] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0153] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for measuring the performance of a depth camera alignment, characterized in that: include: Acquire a depth image and a color image obtained by photographing a photographed object with a depth camera, wherein the photographed object includes a plurality of stereo units located at different positions; Determine the coordinate information corresponding to each stereo unit in the depth image and the color image; The alignment performance measurement result of the depth camera is generated according to the corresponding coordinate information of each stereo unit in the depth image and the color image.

2. The method for measuring the depth camera alignment performance according to claim 1, characterized in that: Generating an alignment performance measurement result of the depth camera according to corresponding coordinate information of each stereo unit in the depth image and the color image, including: Calculate the coordinate offset of each stereo unit according to the corresponding coordinate information of each stereo unit in the depth image and the color image; Comparing the coordinate offset of each stereo unit with a preset offset threshold; Based on the comparison result, an alignment performance measurement result of the depth camera is generated.

3. The method for measuring the depth camera alignment performance according to claim 2, characterized in that: Compare the coordinate offset of each stereo unit with a preset offset threshold, including: Determine the field of view angle range of each stereo unit, where different field of view angle ranges correspond to different offset thresholds; The coordinate offset of each stereo unit is compared with the offset threshold of the corresponding field of view angle range.

4. The method for measuring the depth camera alignment performance according to claim 1, characterized in that: Determine the coordinate information corresponding to each stereo unit in the depth image and the color image, including: Determine deformation information of each stereo unit in the depth image and the color image; Determining whether the depth image and the color image have preset deformation conditions according to deformation information of each stereo unit in the depth image and the color image; If both the depth image and the color image satisfy the preset deformation condition, the coordinate information of the preset position on each stereo unit in the depth image and the color image is obtained as the corresponding coordinate information of each stereo unit in the depth image and the color image.

5. The method for measuring the depth camera alignment performance according to any one of claims 1 to 4, characterized in that: The three-dimensional unit is a geometric body with a regular shape.

6. The method for measuring the depth camera alignment performance according to claim 5, characterized in that: The three-dimensional unit includes at least one of the following: a cylinder, a prism, a cone, and a sphere.

7. A depth camera alignment performance measurement device, characterized in that: include: An image acquisition module, used to obtain a depth image and a color image obtained by a depth camera photographing a photographed object, wherein the photographed object includes a plurality of stereo units located at different positions; An image processing module, used to determine the corresponding coordinate information of each stereo unit in the depth image and the color image; The performance analysis module is used to generate an alignment performance measurement result of the depth camera according to the corresponding coordinate information of each stereo unit in the depth image and the color image.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the depth camera alignment performance measurement method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the depth camera alignment performance measurement method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the depth camera alignment performance measurement method described in any one of claims 1 to 6 is implemented.