Depth information evaluation method, device, electronic device and storage medium
By registering the RGB diagrams and depth diagrams collected by the depth camera in the smart device, generating histograms to separate preset objects and backgrounds, the problem of low accuracy in depth information evaluation in the existing technology is solved, and the working effect of the smart device is improved and user experience.
Patent Information
- Application Number
- CN201910434971.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-05-23
AI Technical Summary
In the working modes of face or human body recognition, tracking, and leading, the registration error of RGB map and depth map and the area of interest are large or drifting, resulting in a decrease in the accuracy of depth information evaluation, affecting the work of the equipment. Effects and user experience.
By registering the RGB maps and depth maps collected by the depth camera, the effective area of interest of the preset object is obtained, and a histogram is generated to separate the preset object and background, and the depth information of the preset object is determined, thereby improving the evaluation accuracy .
It effectively avoids the impact of background on the evaluation of depth information of preset objects, improves the accuracy of evaluation of depth information, and improves the working effect and user experience of smart devices.
Smart Images

Figure CN111986245B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent devices, and in particular to a depth information evaluation method, device, electronic device and storage medium. Background Art
[0002] Current smart devices, such as robots, have working modes such as face (or body) recognition, tracking, and guidance, and the above working modes all require the evaluation of the depth information of the face (or body) first. The current depth information evaluation method mainly involves collecting RGB images and depth images from a depth camera, aligning the RGB images and depth images; shrinking the area of interest in the RGB image according to a preset ratio to exclude background information; and estimating the depth information of the face (or body) from the shrunk area.
[0003] However, in the above solution, due to the installation tolerance of the mass-produced robot, there is a certain error in the registration of the RGB image and the depth map, and the obtained region of interest is somewhat larger or drifts, resulting in a certain background information in the shrunk region of interest, which reduces the evaluation accuracy of the depth information and reduces the working effect and user experience of the smart device. Summary of the invention
[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.
[0005] To this end, the first object of the present invention is to propose a depth information evaluation method. In the method, the RGB image and the depth image captured by the depth camera are aligned, and after obtaining the effective region of interest corresponding to the preset object in the RGB image, a histogram is established, and the depth information of the preset object is determined according to the histogram, wherein the histogram can directly separate the preset object from the background, avoiding the influence of the background on the depth information evaluation of the preset object, thereby improving the evaluation accuracy of the depth information.
[0006] A second objective of the present invention is to provide a depth information evaluation device.
[0007] A third objective of the present invention is to provide an electronic device.
[0008] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium.
[0009] To achieve the above object, a first embodiment of the present invention provides a depth information evaluation method, comprising:
[0010] Aligning the RGB image and the depth image captured by the depth camera to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtaining the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object in the RGB image is located and the visual range of the depth image after the coordinate system conversion;
[0011] Generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval;
[0012] Determine the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram.
[0013] In the depth information evaluation method of the embodiment of the present invention, the histogram can directly separate the preset object and the background, avoiding the influence of the background on the depth information evaluation of the preset object, thereby improving the evaluation accuracy of the depth information and improving the working effect and user experience of the smart device.
[0014] In addition, the depth information evaluation method according to the above embodiment of the present invention may also have the following additional technical features:
[0015] In one embodiment of the present invention, generating a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object includes:
[0016] Determining each depth interval of the histogram according to a preset step size;
[0017] For each depth interval, subtract the preset depth value from the minimum depth of the depth interval to obtain the left boundary of the depth range corresponding to the depth interval; add the preset depth value to the maximum depth of the depth interval to obtain the right boundary of the depth range corresponding to the depth interval;
[0018] Determine the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object;
[0019] According to the depth range to which each pixel belongs, the number of pixels corresponding to each depth interval is determined.
[0020] In one embodiment of the present invention, before determining the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object, the method further includes:
[0021] Determine the sampling step length in the X-axis direction according to the total number of pixels in the effective region of interest corresponding to the preset object in the X-axis direction;
[0022] Determine the sampling step length in the Y-axis direction according to the total number of pixels in the Y-axis direction of the valid region of interest corresponding to the preset object;
[0023] Sampling pixels of the effective region of interest corresponding to the preset object according to the sampling step size in the X-axis direction and the sampling step size in the Y-axis direction in turn to obtain a sampled effective region of interest;
[0024] The determining, according to the depth information of each pixel in the effective region of interest corresponding to the preset object, a depth range to which each pixel belongs specifically includes:
[0025] According to the depth information of each pixel in the sampled effective region of interest corresponding to the preset object, the depth range to which each pixel belongs is determined.
[0026] In one embodiment of the present invention, before determining the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object, the method further includes:
[0027] A first pixel point in the effective region of interest corresponding to the preset object is removed, where the first pixel point is a pixel point whose corresponding depth information is outside a preset depth range.
[0028] In one embodiment of the present invention, determining the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram includes:
[0029] Traverse each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located;
[0030] If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as the preset object;
[0031] If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, the foreground is determined as the preset object;
[0032] Determine the depth information of the preset object according to the depth interval corresponding to the preset object.
[0033] In one embodiment of the present invention, before traversing each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background, the method further includes:
[0034] A first depth interval in the histogram is removed, where the first depth interval is a depth interval corresponding to a number of pixels less than a preset filtering threshold.
[0035] In one embodiment of the present invention, the method further comprises:
[0036] If the depth interval corresponding to the foreground and the depth interval corresponding to the background are different depth intervals, the ratio of the pixel ratio corresponding to the preset object to the preset full score threshold is determined as the confidence of the preset object;
[0037] If the depth interval corresponding to the foreground and the depth interval corresponding to the background are the same depth interval, a preset multiple of the ratio of the pixel proportion corresponding to the preset object to a preset full score threshold is determined as the confidence of the preset object.
[0038] In one embodiment of the present invention, the preset object is a human face or a human body.
[0039] To achieve the above object, a second embodiment of the present invention provides a depth information evaluation device, comprising:
[0040] An acquisition module is used to align the RGB image and the depth image captured by the depth camera to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtain the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object in the RGB image is located and the visual range of the depth image after the coordinate system conversion;
[0041] A generating module, configured to generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval;
[0042] The determination module is used to determine the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram.
[0043] In addition, the depth information evaluation device according to the above embodiment of the present invention may also have the following additional technical features:
[0044] In one embodiment of the present invention, the generating module is specifically used to determine each depth interval of the histogram according to a preset step size;
[0045] For each depth interval, subtract the preset depth value from the minimum depth of the depth interval to obtain the left boundary of the depth range corresponding to the depth interval; add the preset depth value to the maximum depth of the depth interval to obtain the right boundary of the depth range corresponding to the depth interval;
[0046] Determine the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object;
[0047] According to the depth range to which each pixel belongs, the number of pixels corresponding to each depth interval is determined.
[0048] In one embodiment of the present invention, the generating module is further configured to:
[0049] Determine the sampling step length in the X-axis direction according to the total number of pixels in the effective region of interest corresponding to the preset object in the X-axis direction;
[0050] Determine the sampling step length in the Y-axis direction according to the total number of pixels in the Y-axis direction of the valid region of interest corresponding to the preset object;
[0051] Sampling pixels of the effective region of interest corresponding to the preset object according to the sampling step size in the X-axis direction and the sampling step size in the Y-axis direction in turn to obtain a sampled effective region of interest;
[0052] The generating module is specifically used for:
[0053] According to the depth information of each pixel in the sampled effective region of interest corresponding to the preset object, the depth range to which each pixel belongs is determined.
[0054] In one embodiment of the present invention, the generating module is further configured to:
[0055] A first pixel point in the effective region of interest corresponding to the preset object is removed, where the first pixel point is a pixel point whose corresponding depth information is outside a preset depth range.
[0056] In one embodiment of the present invention, the determining module is specifically configured to:
[0057] Traverse each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located;
[0058] If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as the preset object;
[0059] If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, the foreground is determined as the preset object;
[0060] Determine the depth information of the preset object according to the depth interval corresponding to the preset object.
[0061] In one embodiment of the present invention, the determining module is further configured to:
[0062] A first depth interval in the histogram is removed, where the first depth interval is a depth interval corresponding to a number of pixels less than a preset filtering threshold.
[0063] In one embodiment of the present invention, the determining module is further configured to:
[0064] If the depth interval corresponding to the foreground and the depth interval corresponding to the background are different depth intervals, the ratio of the pixel ratio corresponding to the preset object to the preset full score threshold is determined as the confidence of the preset object;
[0065] If the depth interval corresponding to the foreground and the depth interval corresponding to the background are the same depth interval, a preset multiple of the ratio of the pixel proportion corresponding to the preset object to a preset full score threshold is determined as the confidence of the preset object.
[0066] In one embodiment of the present invention, the preset object is a human face or a human body.
[0067] To achieve the above-mentioned purpose, an embodiment of the third aspect of the present invention proposes an electronic device, characterized in that it includes: a shell, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the shell, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the depth information evaluation method described in the above-mentioned embodiment.
[0068] In order to achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present invention proposes a non-temporary computer-readable storage medium, on which a computer program is stored, characterized in that when the program is executed by a processor, the depth information evaluation method described in the above-mentioned embodiment is implemented.
[0069] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0071] Figure 1A schematic diagram of a flow chart of a depth information evaluation method provided by an embodiment of the present invention;
[0072] Figure 2 A schematic diagram of a flow chart of another depth information evaluation method provided by an embodiment of the present invention;
[0073] Figure 3 It is a schematic diagram of a histogram when the foreground is an occluder and the background is a preset object;
[0074] Figure 4 It is a schematic diagram of the histogram when the foreground is a preset object;
[0075] Figure 5 A schematic diagram of the structure of a depth information evaluation device provided by an embodiment of the present invention;
[0076] Figure 6 A block diagram of an exemplary electronic device provided in an embodiment of the present invention and suitable for implementing the implementation of the present invention is provided. DETAILED DESCRIPTION
[0077] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0078] The depth information evaluation method, device, electronic device, and storage medium according to embodiments of the present invention are described below with reference to the accompanying drawings.
[0079] The execution subject of the depth information evaluation method of the embodiment of the present invention is an electronic device, which may be an intelligent device that collects RGB images and depth images of a face or a body through a depth camera to evaluate the depth information of the face or the body, or may be an external device that communicates with the intelligent device, such as a server, etc. The intelligent device may be, for example, an intelligent robot, an intelligent home appliance, etc. The following description is made by taking the execution subject as an example of an intelligent device.
[0080] Figure 1 A schematic diagram of a flow chart of a depth information evaluation method provided by an embodiment of the present invention. Figure 1 As shown, the depth information evaluation method includes the following steps:
[0081] Step 101, align the RGB image and the depth image captured by the depth camera to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtain the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object is located in the RGB image and the visual range of the depth image after the coordinate system conversion.
[0082] In this embodiment, a depth camera is configured on the smart device, and the smart device can collect images within the monitoring range in real time through the depth camera, such as RGB images and depth images of faces or bodies.
[0083] In this embodiment, the depth camera may include two cameras, one camera is an RGBT camera for collecting depth maps; the other camera is an RGB camera for collecting RGB maps; the two cameras collect depth maps and RGB maps at the same time. The process of the smart device aligning the RGB map and the depth map collected by the depth camera can be specifically as follows: combining the intrinsic parameter matrix of the RGBT camera, that is, the relationship between the coordinate system of the RGBT camera and the coordinate system of the depth map, converting the coordinate system of the depth map to the coordinate system of the RGBT camera; then combining the extrinsic parameter matrix, that is, the relationship between the coordinate system of the RGBT camera and the coordinate system of the RGB camera, converting the coordinate system of the depth map to the coordinate system of the RGB camera; then combining the intrinsic parameter matrix of the RGB camera, that is, the relationship between the coordinate system of the RGB camera and the coordinate system of the RGB map, converting the coordinate system of the depth map to the coordinate system of the RGB map, thereby achieving consistency between the coordinate systems of the depth map and the RGB map, and for a certain pixel point, the RGB information and depth information of the pixel point can be obtained at the same time.
[0084] In this embodiment, the process of the smart device obtaining the effective region of interest corresponding to the preset object in the RGB image can be specifically as follows: (1) taking the preset object as the object of interest, performing interest recognition on the RGB image, and obtaining the region where the preset object is located in the RGB image; wherein the shape of the region can be any shape, and a rectangle can be used as an example in this embodiment. In addition, there can be multiple preset objects. For example, when the RGB image includes three faces, the preset objects can be three faces, and then three regions are obtained. (2) Obtaining the visual range of the depth map after coordinate system conversion. In this embodiment, the visual range of the depth map is generally smaller than the visual range of the RGB image. Therefore, the region determined by the visual range of the depth map in the RGB image can be determined as the effective region. (3) Intersecting the region where each preset object is located with the effective region, and then obtaining the effective region of interest corresponding to each preset object.
[0085] Step 102: Generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval.
[0086] In this embodiment, the process of the smart device executing step 102 may specifically include: determining each depth interval of the histogram according to a preset step size; for each depth interval, subtracting a preset depth value from the minimum depth of the depth interval to obtain the left boundary of the depth range corresponding to the depth interval; adding the preset depth value to the maximum depth of the depth interval to obtain the right boundary of the depth range corresponding to the depth interval; determining the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object; determining the number of pixels corresponding to each depth interval according to the depth range to which each pixel in the effective region of interest corresponding to the preset object belongs. The preset step size and the preset depth value are coefficients pre-set in the smart device.
[0087] In this embodiment, since the accuracy and precision in the histogram operation are a contradiction, reducing the step size can improve the precision, but the accuracy will decrease. Therefore, in order to ensure the accuracy of the histogram operation, the depth range corresponding to each depth interval can be expanded. For example, taking the depth interval as [A, B] as an example, the corresponding depth range can be [A-sigma, B+sigma]. Among them, A is the minimum depth of the depth interval; B is the maximum depth of the depth interval; sigma is the preset depth value; BA is the preset step size.
[0088] In this embodiment, the expansion of the depth range can further improve the accuracy of depth information assessment in the following scenarios: a person is facing the intelligent robot sideways, or a person is walking sideways.
[0089] As an example, in order to reduce the number of pixels that need to be processed in the effective region of interest corresponding to the preset object and reduce the amount of calculation, the smart device can also perform the following process before determining the depth range to which each pixel belongs based on the depth information of each pixel in the effective region of interest corresponding to the preset object: determine the sampling step in the X-axis direction based on the total number of pixels in the X-axis direction of the effective region of interest corresponding to the preset object; determine the sampling step in the Y-axis direction based on the total number of pixels in the Y-axis direction of the effective region of interest corresponding to the preset object; sample the pixels in the effective region of interest corresponding to the preset object according to the sampling step in the X-axis direction and the sampling step in the Y-axis direction in turn to obtain the sampled effective region of interest. Correspondingly, the smart device can specifically determine the depth range to which each pixel belongs based on the depth information of each pixel in the sampled effective region of interest corresponding to the preset object.
[0090] Among them, according to the total number of pixels in the X-axis direction in the effective region of interest corresponding to the preset object, the process of determining the sampling step in the X-axis direction can be specifically as follows: judging whether the total number of pixels in the X-axis direction in the effective region of interest exceeds the threshold VOXEL-X; if not, then no sampling is performed on the effective region of interest in the X-axis direction; if so, then the sampling step in the X-axis direction is calculated according to the following formula (1).
[0091] skip=(pixels_x%VOXEL_X)+1(1)
[0092] Where skip represents the sampling step size; pixels_x represents the total number of pixels in the X-axis direction in the effective region of interest.
[0093] Among them, according to the total number of pixels in the Y-axis direction in the effective region of interest corresponding to the preset object, the process of determining the sampling step in the Y-axis direction can be specifically as follows: judging whether the total number of pixels in the Y-axis direction in the effective region of interest exceeds the threshold value VOXEL-Y; if not, then no sampling is performed on the effective region of interest in the Y-axis direction; if so, then the sampling step in the Y-axis direction is calculated according to the following formula (2).
[0094] skip=(pixels_y%VOXEL_Y)+1(1)
[0095] Among them, pixels_y represents the total number of pixels in the Y-axis direction in the effective area of interest.
[0096] In this embodiment, the effective region of interest corresponding to the preset object is sampled in pixels according to the sampling step size in the X-axis direction and the sampling step size in the Y-axis direction. After obtaining the sampled effective region of interest, in order to ensure that the depth information of each pixel is within the preset depth range, the smart device can also perform the following process: remove the first pixel in the effective region of interest corresponding to the preset object, and the first pixel is a pixel whose corresponding depth information is outside the preset depth range. Among them, the preset depth range is a coefficient pre-set in the smart device. In this embodiment, the above coefficient can be set according to the usage scenario of the smart device, and the value of the coefficient can be different in different usage scenarios.
[0097] Step 103: Determine depth information of a preset object according to the number of pixels corresponding to each depth interval in the histogram.
[0098] In this embodiment, different objects in the depth map generally have different depth information. Therefore, the face, background, etc. can be separated through the histogram, so that the depth information of the preset object can be determined according to the histogram, avoiding background interference and improving the accuracy of the depth information evaluation of the preset object.
[0099] The depth information evaluation method of the embodiment of the present invention aligns the RGB image and the depth image collected by the depth camera to convert the coordinate system of the depth image into the coordinate system of the RGB image, and obtains the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object is located in the RGB image and the visual range of the depth image after the coordinate system conversion; a histogram corresponding to the preset object is generated according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval; the depth information of the preset object is determined according to the number of pixels corresponding to each depth interval in the histogram, wherein the histogram can directly separate the preset object from the background, avoid the influence of the background on the depth information evaluation of the preset object, thereby improving the accuracy of the depth information evaluation and improving the working effect and user experience of the smart device.
[0100] Figure 2 FIG. 1 is a flow chart of another depth information evaluation method provided by an embodiment of the present invention. Figure 2 As shown, in Figure 1 Based on the illustrated embodiment, step 103 may specifically include the following steps:
[0101] Step 1031, traverse each depth interval in the histogram, obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located.
[0102] In this embodiment, the pixel ratio corresponding to the foreground refers to the ratio of the number of pixels in the depth interval corresponding to the foreground to the total number of pixels in the effective region of interest. The pixel ratio corresponding to the background refers to the ratio of the number of pixels in the depth interval corresponding to the background to the total number of pixels in the effective region of interest.
[0103] Step 1032: If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as a preset object.
[0104] Step 1033: If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to a preset occlusion threshold, the foreground is determined as a preset object.
[0105] In this embodiment, when there is a certain amount of occlusion in the RGB image, for example, when part of a person's face is occluded by a book or other object, the occlusion area is generally not large and the number of pixels in the occlusion part is small. Therefore, when the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is less than the preset occlusion threshold, the foreground can be determined as occlusion and the background can be determined as a preset object such as a face or a human body. Figure 3 As shown in , it is a histogram diagram when the foreground is an occluder and the background is a preset object. When the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, it means that the number of pixels in the foreground is large and it is generally impossible to be an occluder. Therefore, the foreground is determined to be a preset object such as a face or a human body. Figure 4 The figure shows a histogram diagram when the foreground is a preset object.
[0106] Step 1034: Determine depth information of the preset object according to the depth interval corresponding to the preset object.
[0107] In this embodiment, the center value of the depth interval corresponding to the preset object may be determined as the depth information of the preset object.
[0108] Furthermore, before step 1031, the following steps may be included: removing a first depth interval in the histogram, where the first depth interval is a depth interval whose corresponding number of pixels is less than a preset filtering threshold.
[0109] In this embodiment, there are generally certain noise points in the effective area of interest corresponding to the preset object. The noise points have the same depth information and the number of noise points is small. Therefore, in order to remove the noise points in the effective area of interest, the first depth interval in the histogram can be filtered out and the number of pixels in the first depth interval can be set to 0.
[0110] Furthermore, after step 103, the smart device can also obtain the confidence level of the depth information of the preset object. The specific process may be: if the depth interval corresponding to the foreground and the depth interval corresponding to the background are different depth intervals, then the ratio of the pixel proportion corresponding to the preset object to the preset full score threshold is determined as the confidence level of the preset object; if the depth interval corresponding to the foreground and the depth interval corresponding to the background are the same depth interval, then the preset multiple of the ratio of the pixel proportion corresponding to the preset object to the preset full score threshold is determined as the confidence level of the preset object.
[0111] The depth information evaluation method of the embodiment of the present invention aligns the RGB image and the depth image collected by the depth camera to convert the coordinate system of the depth image into the coordinate system of the RGB image, and obtains the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object is located in the RGB image and the visual range of the depth image after the coordinate system conversion; a histogram corresponding to the preset object is generated according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval; the foreground and the background are determined according to the pixel proportion and position of each depth interval, and the depth interval corresponding to the preset object is determined according to the ratio of the pixel proportion of the foreground to the background, which can avoid the interference of obstructions and background, accurately locate the depth information of the preset object, thereby improving the accuracy of the depth information evaluation of the preset object, and improving the working effect and user experience of the smart device.
[0112] In order to implement the above embodiment, the embodiment of the present invention further proposes a depth information evaluation device. Figure 5 A schematic diagram of the structure of a depth information evaluation device provided by an embodiment of the present invention.
[0113] like Figure 5 As shown, the depth information evaluation device includes: an acquisition module 51, a generation module 52 and a determination module 53.
[0114] Among them, the acquisition module 51 is used to align the RGB image and the depth image collected by the depth camera to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtain the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object is located in the RGB image and the visual range of the depth image after the coordinate system conversion;
[0115] A generating module 52 is used to generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; the histogram includes: the number of pixels corresponding to each depth interval;
[0116] The determination module 53 is used to determine the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram.
[0117] In this embodiment, a depth camera is configured on the smart device, and the smart device can collect images within the monitoring range in real time through the depth camera, such as RGB images and depth images of faces or bodies.
[0118] In this embodiment, the depth camera may include two cameras, one camera is an RGBT camera for collecting depth maps; the other camera is an RGB camera for collecting RGB maps; the two cameras collect depth maps and RGB maps at the same time. The process of the acquisition module 51 registering the RGB map and the depth map collected by the depth camera may be specifically as follows: combining the intrinsic parameter matrix of the RGBT camera, that is, the relationship between the coordinate system of the RGBT camera and the coordinate system of the depth map, converting the coordinate system of the depth map to the coordinate system of the RGBT camera; then combining the extrinsic parameter matrix, that is, the relationship between the coordinate system of the RGBT camera and the coordinate system of the RGB camera, converting the coordinate system of the depth map to the coordinate system of the RGB camera; then combining the intrinsic parameter matrix of the RGB camera, that is, the relationship between the coordinate system of the RGB camera and the coordinate system of the RGB map, converting the coordinate system of the depth map to the coordinate system of the RGB map, thereby achieving consistency between the coordinate systems of the depth map and the RGB map, and for a certain pixel point, the RGB information and depth information of the pixel point can be obtained at the same time.
[0119] In this embodiment, the process of obtaining the effective region of interest corresponding to the preset object in the RGB image by the acquisition module 51 can be specifically as follows: (1) taking the preset object as the object of interest, performing interest recognition on the RGB image, and obtaining the region where the preset object is located in the RGB image; wherein the shape of the region can be any shape, and a rectangle can be used as an example in this embodiment. In addition, there can be multiple preset objects. For example, when the RGB image includes three faces, the preset objects can be three faces, and then three regions are obtained. (2) Obtaining the visual range of the depth image after coordinate system conversion. In this embodiment, the visual range of the depth image is generally smaller than the visual range of the RGB image. Therefore, the region determined by the visual range of the depth image in the RGB image can be determined as the effective region. (3) Intersecting the region where each preset object is located with the effective region, and then obtaining the effective region of interest corresponding to each preset object.
[0120] In this embodiment, as an example, the generation module 52 can be specifically used to determine each depth interval of the histogram according to a preset step size; for each depth interval, subtract the preset depth value from the minimum depth of the depth interval to obtain the left boundary of the depth range corresponding to the depth interval; add the preset depth value to the maximum depth of the depth interval to obtain the right boundary of the depth range corresponding to the depth interval; determine the depth range to which each pixel belongs according to the depth information of each pixel in the effective area of interest corresponding to the preset object; determine the number of pixels corresponding to each depth interval according to the depth range to which each pixel in the effective area of interest corresponding to the preset object belongs. Among them, the preset step size and the preset depth value are coefficients pre-set in the smart device.
[0121] In this embodiment, since the accuracy and precision in the histogram operation are a contradiction, reducing the step size can improve the precision, but the accuracy will decrease. Therefore, in order to ensure the accuracy of the histogram operation, the depth range corresponding to each depth interval can be expanded. For example, taking the depth interval as [A, B] as an example, the corresponding depth range can be [A-sigma, B+sigma]. Among them, A is the minimum depth of the depth interval; B is the maximum depth of the depth interval; sigma is the preset depth value; BA is the preset step size.
[0122] In this embodiment, the expansion of the depth range can further improve the accuracy of depth information assessment in the following scenarios: a person is facing the intelligent robot sideways, or a person is walking sideways.
[0123] In this embodiment, as an example, in order to reduce the number of pixels that need to be processed in the effective region of interest corresponding to the preset object and reduce the amount of calculation, the generation module 52 can also be used to determine the sampling step in the X-axis direction according to the total number of pixels in the X-axis direction of the effective region of interest corresponding to the preset object; determine the sampling step in the Y-axis direction according to the total number of pixels in the Y-axis direction of the effective region of interest corresponding to the preset object; and sample the pixels in the effective region of interest corresponding to the preset object according to the sampling step in the X-axis direction and the sampling step in the Y-axis direction to obtain the sampled effective region of interest. Correspondingly, the generation module 52 can be used to determine the depth range to which each pixel belongs according to the depth information of each pixel in the sampled effective region of interest corresponding to the preset object.
[0124] Among them, according to the total number of pixels in the X-axis direction in the effective region of interest corresponding to the preset object, the process of determining the sampling step in the X-axis direction can be specifically as follows: judging whether the total number of pixels in the X-axis direction in the effective region of interest exceeds the threshold VOXEL-X; if not, then no sampling is performed on the effective region of interest in the X-axis direction; if so, then the sampling step in the X-axis direction is calculated according to the following formula (1).
[0125] skip=(pixels_x%VOXEL_X)+1 (1)
[0126] Where skip represents the sampling step size; pixels_x represents the total number of pixels in the X-axis direction in the effective region of interest.
[0127] Among them, according to the total number of pixels in the Y-axis direction in the effective region of interest corresponding to the preset object, the process of determining the sampling step in the Y-axis direction can be specifically as follows: judging whether the total number of pixels in the Y-axis direction in the effective region of interest exceeds the threshold value VOXEL-Y; if not, then no sampling is performed on the effective region of interest in the Y-axis direction; if so, then the sampling step in the Y-axis direction is calculated according to the following formula (2).
[0128] skip=(pixels_y%VOXEL_Y)+1 (1)
[0129] Among them, pixels_y represents the total number of pixels in the Y-axis direction in the effective area of interest.
[0130] In this embodiment, the effective region of interest corresponding to the preset object is sampled in pixels according to the sampling step size in the X-axis direction and the sampling step size in the Y-axis direction. After obtaining the sampled effective region of interest, in order to ensure that the depth information of each pixel is within the preset depth range, the generation module 52 can also be used to remove the first pixel in the effective region of interest corresponding to the preset object, where the first pixel is a pixel whose corresponding depth information is outside the preset depth range. The preset depth range is a coefficient pre-set in the smart device. In this embodiment, the above coefficient can be set according to the usage scenario of the smart device, and the value of the coefficient can be different for different usage scenarios.
[0131] In a possible implementation of this embodiment, the determining module 53 is specifically configured to:
[0132] Traverse each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located;
[0133] If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as the preset object;
[0134] If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, the foreground is determined as the preset object;
[0135] Determine the depth information of the preset object according to the depth interval corresponding to the preset object.
[0136] Furthermore, the determination module 53 is specifically configured to remove a first depth interval in the histogram, where the first depth interval is a depth interval corresponding to a number of pixels that is less than a preset filtering threshold.
[0137] In this embodiment, there are generally certain noise points in the effective area of interest corresponding to the preset object. The noise points have the same depth information and the number of noise points is small. Therefore, in order to remove the noise points in the effective area of interest, the first depth interval in the histogram can be filtered out and the number of pixels in the first depth interval can be set to 0.
[0138] In this embodiment, the pixel ratio corresponding to the foreground refers to the ratio of the number of pixels in the depth interval corresponding to the foreground to the total number of pixels in the effective region of interest. The pixel ratio corresponding to the background refers to the ratio of the number of pixels in the depth interval corresponding to the background to the total number of pixels in the effective region of interest.
[0139] In this embodiment, when there is a certain amount of occlusion in the RGB image, for example, when part of a person's face is occluded by a book or other object, the occlusion area is generally not large and the number of pixels in the occlusion part is small. Therefore, when the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is less than the preset occlusion threshold, the foreground can be determined as occlusion and the background can be determined as a preset object such as a face or a human body. Figure 3 As shown in , it is a histogram diagram when the foreground is an occluder and the background is a preset object. When the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, it means that the number of pixels in the foreground is large and it is generally impossible to be an occluder. Therefore, the foreground is determined to be a preset object such as a face or a human body. Figure 4 The figure shows a histogram diagram when the foreground is a preset object.
[0140] Furthermore, the determination module 53 can also be used to, if the depth interval corresponding to the foreground and the depth interval corresponding to the background are different depth intervals, then determine the ratio of the pixel proportion corresponding to the preset object to the preset full score threshold as the confidence of the preset object; if the depth interval corresponding to the foreground and the depth interval corresponding to the background are the same depth interval, then determine the preset multiple of the ratio of the pixel proportion corresponding to the preset object to the preset full score threshold as the confidence of the preset object.
[0141] The depth information evaluation device of the embodiment of the present invention can improve the accuracy of depth information evaluation of a preset object and improve the working effect and user experience of a smart device.
[0142] In order to achieve the above objectives, an embodiment of the present invention further proposes a non-temporary computer-readable storage medium, on which a computer program is stored, characterized in that when the program is executed by a processor, the depth information evaluation method described in the above embodiment is implemented.
[0143] Figure 6 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application is shown. Figure 6As shown, the electronic device includes: a housing 310, a processor 320, a memory 330, a circuit board 340 and a power circuit 350, wherein the circuit board 340 is placed inside the space enclosed by the housing 310, and the processor 320 and the memory 330 are arranged on the circuit board 340; the power circuit 350 is used to supply power to various circuits or devices of the above electronic device; the memory 330 is used to store executable program codes; the processor 320 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 330, so as to execute the depth information evaluation method described in the above embodiment. The electronic device can be any depth information evaluation device, can be a smart device itself equipped with a depth camera, or can be an external device that communicates with the smart device, such as a server.
[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.
[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A depth information evaluation method, characterized in that: The following steps are involved: The RGB image and the depth image captured by the depth camera are registered to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtain the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object in the RGB image is located and the visual range of the depth image after the coordinate system conversion; the depth camera includes an RGBT camera and an RGB camera, the RGBT camera is used to collect the depth map, and the RGB camera is used to collect the RGB image; Generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; The histogram includes: the number of pixels corresponding to each depth interval; Determining the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram; Wherein, determining the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram includes: Traverse each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located; If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as the preset object; If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, the foreground is determined as the preset object; Determine the depth information of the preset object according to the depth interval corresponding to the preset object.
2. The method according to claim 1, characterized in that The step of generating a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object comprises: Determining each depth interval of the histogram according to a preset step size; For each depth interval, subtract the preset depth value from the minimum depth of the depth interval to obtain the left boundary of the depth range corresponding to the depth interval; add the preset depth value to the maximum depth of the depth interval to obtain the right boundary of the depth range corresponding to the depth interval; Determine the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object; According to the depth range to which each pixel belongs, the number of pixels corresponding to each depth interval is determined.
3. The method according to claim 2, characterized in that Before determining the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object, the method further includes: Determine the sampling step length in the X-axis direction according to the total number of pixels in the effective region of interest corresponding to the preset object in the X-axis direction; Determine the sampling step length in the Y-axis direction according to the total number of pixels in the Y-axis direction of the valid region of interest corresponding to the preset object; Sampling pixels of the effective region of interest corresponding to the preset object according to the sampling step size in the X-axis direction and the sampling step size in the Y-axis direction in turn to obtain a sampled effective region of interest; The determining, according to the depth information of each pixel in the effective region of interest corresponding to the preset object, a depth range to which each pixel belongs specifically includes: According to the depth information of each pixel in the sampled effective region of interest corresponding to the preset object, the depth range to which each pixel belongs is determined.
4. The method according to claim 2, characterized in that: Before determining the depth range to which each pixel belongs according to the depth information of each pixel in the effective region of interest corresponding to the preset object, the method further includes: A first pixel point in the effective region of interest corresponding to the preset object is removed, where the first pixel point is a pixel point whose corresponding depth information is outside a preset depth range.
5. The method according to claim 1, characterized in that Before traversing each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background, the method further includes: A first depth interval in the histogram is removed, where the first depth interval is a depth interval corresponding to a number of pixels less than a preset filtering threshold.
6. The method according to claim 1, characterized in that Also includes: If the depth interval corresponding to the foreground and the depth interval corresponding to the background are different depth intervals, the ratio of the pixel ratio corresponding to the preset object to the preset full score threshold is determined as the confidence of the preset object; If the depth interval corresponding to the foreground and the depth interval corresponding to the background are the same depth interval, a preset multiple of the ratio of the pixel proportion corresponding to the preset object to a preset full score threshold is determined as the confidence of the preset object.
7. A depth information evaluation device, characterized in that: include: An acquisition module, used to align the RGB image and the depth image collected by the depth camera, so as to convert the coordinate system of the depth image to the coordinate system of the RGB image, and obtain the effective region of interest corresponding to the preset object in the RGB image; the effective region of interest is determined according to the area where the preset object in the RGB image is located and the visual range of the depth image after the coordinate system conversion; the depth camera includes an RGBT camera and an RGB camera, the RGBT camera is used to collect the depth map, and the RGB camera is used to collect the RGB image; A generating module, configured to generate a histogram corresponding to the preset object according to the depth information of each pixel in the effective region of interest corresponding to the preset object; The histogram includes: the number of pixels corresponding to each depth interval; A determination module, used to determine the depth information of the preset object according to the number of pixels corresponding to each depth interval in the histogram; The determination module is specifically used to traverse each depth interval in the histogram to obtain the depth interval and pixel ratio corresponding to the foreground, and the depth interval and pixel ratio corresponding to the background; the depth interval corresponding to the foreground is the depth interval where the first peak in the histogram is located; the depth interval corresponding to the background is the depth interval where the highest peak in the histogram is located; If the ratio of the pixel ratio corresponding to the foreground to the pixel ratio corresponding to the background is less than a preset occlusion threshold, the background pair is determined as the preset object; If the ratio of the pixel proportion corresponding to the foreground to the pixel proportion corresponding to the background is greater than or equal to the preset occlusion threshold, the foreground is determined as the preset object; Determine the depth information of the preset object according to the depth interval corresponding to the preset object.
8. An electronic device, characterized in that: It includes a shell, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the shell, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the depth information evaluation method described in any one of claims 1-6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the depth information evaluation method as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Lens shielding detection method and device
CN104657993A
Depth and color image combined human face shielding detection early-warning device and implementation method
CN105354563A
Image area determination method and device
CN107564020A
Passenger flow volume statistics method and device
CN108932464A
An accelerated face detection method and a storage medium
CN109086724A
