A method for selective imaging of images using a depth color camera
Through the registration and selective exposure settings of depth cameras and color cameras, the problem of over-exposure or under-exposure in construction robots is solved, and the clear acquisition of window images and accurate detection of wall obstacles is achieved.
Patent Information
- Application Number
- CN202310061859.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-18
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2043-01-18
AI Technical Summary
During the construction of a construction robot, the RGB camera sees the wall in half of its field of view and the windows, which leads to overexposed or underexposed images and loses the ability to distinguish wall obstacles.
Through the appearance registration of the depth camera and color camera, the depth camera performs hollow detection, the color camera performs selective exposure settings, limit the field of view of the depth camera, discard invalid data, calculate and save valid RGB points, and generate using fixed exposure setting parameters Color image, filling in void areas to avoid overexposure or underexposure.
It realizes clear acquisition of window images, ensures accurate detection of wall obstacles, avoids overexposed or underexposed, and improves the ability to discern small objects.
Smart Images
Figure CN116055852B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of building construction image acquisition, and in particular to an image selective imaging method of a depth color camera. Background Art
[0002] During the construction process of the building robot, precise construction is required near the window. In the process of detecting wall obstacles and doors and windows, depth cameras and color cameras are used simultaneously. In this scenario, half of the RGB camera's field of view sees the wall, and the other half sees the space outside the window, which can easily cause underexposure of the wall and overexposure of the external space. The image forms two areas, one black and one white. It loses the ability to distinguish small objects such as wall obstacles. At this time, there are high requirements for the white balance of the camera, and it is necessary to avoid underexposure of part of the wall. Summary of the invention
[0003] The purpose of the present invention is to provide an image selective imaging method of a depth color camera to solve the technical problem in the prior art that overexposure and underexposure are prone to occur in window image acquisition by construction robots.
[0004] The present invention provides an image selective imaging method of a depth color camera, which is used for collecting window images of a building wall processing robot. The depth camera and the color camera are registered, the depth camera performs hole detection, and the color camera performs selective exposure setting. The imaging method comprises the following steps:
[0005] Sp1: Get the depth image of the depth camera;
[0006] Sp2: Get the image from the RGB camera;
[0007] Sp3: Attach the corresponding RGB color to each pixel in the depth image, such as attaching the corresponding color to the depth image pixel with coordinates (x, y);
[0008] Sp4: Selection based on the data of the depth camera: Combined with the logic of equipment construction, the field of view of the depth camera is limited to a certain range d through hardware or software. It can be considered that objects outside d have nothing to do with the current construction of the wall construction equipment, and both depth data and RGB data can be discarded; areas without depth data due to metal reflections or beyond the field of view are regarded as invalid data areas, and their depth and RGB data are discarded; the depth data within the distance d from the camera is retained, and the corresponding RGB point coordinates are calculated, and the calculated RGB points are saved as a set.
[0009] Further, each pixel in the depth image is assigned a corresponding RGB color method including:
[0010] Sp3-1: Construct a three-dimensional vector p_i r = (x, y, z), where x, y are the pixel coordinates of the point, and z is the depth value of the pixel;
[0011] Sp3-2: Use the inverse of the intrinsic parameter matrix Hi r, multiply it by p r to get the corresponding spatial point coordinates P r, P r = (x1, y1, z1), which means that in the depth coordinate system, the xyz three-axis distances from the origin of the depth camera are (x1, y1, z1), respectively. The intrinsic parameter matrix Hi r can be obtained through calibration by the camera manufacturer or calibration board;
[0012] Sp3-3: Since P_i r is the coordinate of the point in the depth coordinate system, convert it to the coordinate system of the RGB camera, multiply P_i r by the rotation matrix R, and add a translation vector T to obtain P_rgb, which is the coordinate of the point in the RGB coordinate system. The positional relationship between the depth camera and the RGB camera is represented by the rotation matrix R and the translation vector T, which are provided by the manufacturer or calibrated by the calibration plate;
[0013] Sp3-4: Multiply the intrinsic parameter matrix H_rgb of the RGB camera by P_rgb to get p_rgb, which is also a three-dimensional vector. Its x and y coordinates are the pixel coordinates of the point in the RGB image.
[0014] Sp3-5: Perform the above operation on each pixel in the depth image to obtain the registered depth map. Each point in the depth map corresponds to a pixel in the RGB image.
[0015] Furthermore, in this method, based on the image generated by the ISP hardware, points with matching depth data are selected, and exposure statistics are used to evaluate the exposure quality of the current image.
[0016] Furthermore, in the present method, for the selected RGB points, a color image is generated using fixed exposure setting parameters according to the intensity of the photoelectric signal transmitted back by the camera's photosensitive chip.
[0017] Furthermore, in this method, the generated color image will be incomplete at the hole position. The three RGB color channels are assigned a value of 255 and filled with color. The non-hole area will be displayed normally without underexposure or overexposure, which will not affect the obstacle detection of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 is a flow chart of the imaging method in the present invention. DETAILED DESCRIPTION
[0020] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.
[0021] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention. Specific embodiment:
[0023] Combine the following Figure 1 As shown, an embodiment of the present invention provides an image selective imaging method of a depth color camera, which is used for window edge image acquisition of a building wall processing robot. The depth camera and the color camera are registered, the depth camera performs hole detection, and the color camera performs selective exposure setting. The imaging method includes the following steps:
[0024] Sp1: Get the depth image of the depth camera;
[0025] Sp2: Get the image from the RGB camera;
[0026] Sp3: Attach the corresponding RGB color to each pixel in the depth image, such as attaching the corresponding color to the depth image pixel with coordinates (x, y);
[0027] The method of attaching a corresponding RGB color to each pixel in the depth image includes:
[0028] Sp3-1: Construct a three-dimensional vector p_i r = (x, y, z), where x, y are the pixel coordinates of the point, and z is the depth value of the pixel;
[0029] Sp3-2: Use the inverse of the intrinsic parameter matrix Hi r, multiply it by p r to get the corresponding spatial point coordinates P r, P r = (x1, y1, z1), which means that in the depth coordinate system, the xyz three-axis distances from the origin of the depth camera are (x1, y1, z1), respectively. The intrinsic parameter matrix Hi r can be obtained through calibration by the camera manufacturer or calibration board;
[0030] Sp3-3: Since P_i r is the coordinate of the point in the depth coordinate system, convert it to the coordinate system of the RGB camera, multiply P_i r by the rotation matrix R, and add a translation vector T to obtain P_rgb, which is the coordinate of the point in the RGB coordinate system. The positional relationship between the depth camera and the RGB camera is represented by the rotation matrix R and the translation vector T, which are provided by the manufacturer or calibrated by the calibration plate;
[0031] Sp3-4: Multiply the intrinsic parameter matrix H_rgb of the RGB camera by P_rgb to get p_rgb, which is also a three-dimensional vector. Its x and y coordinates are the pixel coordinates of the point in the RGB image.
[0032] Sp3-5: Perform the above operation on each pixel in the depth image to obtain the registered depth map. Each point in the depth map corresponds to a pixel in the RGB image.
[0033] Sp4: Selection based on the data of the depth camera: Combined with the logic of equipment construction, the field of view of the depth camera is limited to a certain range d through hardware or software. It can be considered that objects outside d have nothing to do with the current construction of the wall construction equipment, and both depth data and RGB data can be discarded; areas without depth data due to metal reflections or beyond the field of view are regarded as invalid data areas, and their depth and RGB data are discarded; the depth data within the distance d from the camera is retained, and the corresponding RGB point coordinates are calculated, and the calculated RGB points are saved as a set.
[0034] Smart cameras have traditional automatic exposure algorithms: In a response cycle, the traditional automatic exposure algorithm needs to deal with the following specific matters: evaluate the exposure quality of the current image based on the image exposure statistics generated by the ISP hardware; if the exposure quality needs to be adjusted, generate the working parameters of the next frame of the image based on the current working parameters and the ideal exposure target; write the new working parameters to each hardware device, and drive the aperture, sensor shutter and gain to a new position. The modification of the traditional automatic exposure algorithm in this solution is: according to the image generated by the ISP hardware, select the points with depth data matching, and use the exposure statistics to evaluate the exposure quality of the current image. For the selected RGB points, according to the photoelectric signal intensity sent back by the camera photosensitive chip, use fixed exposure setting parameters to generate a color image; the generated color image will be incomplete at the hole position, and the three RGB color channels are assigned 255 to fill the color, and the non-hole area will be normally colored without underexposure or overexposure, which will not affect the obstacle detection of the algorithm.
[0035] In the prior art, when a building wall processing robot performs construction near a window, half of the RGB camera's field of view sees the wall and the other half sees the space outside the window, which easily causes underexposure of the wall and overexposure of the external space. The image forms two areas, one black and one white, and loses the ability to distinguish small objects such as wall obstacles. Therefore, in the present invention, through the field registration of the depth camera and the color camera, the depth camera is used to perform hole detection, and the color camera is used for selective exposure setting. For areas without depth data due to metal reflection or beyond the field of view, they can be automatically regarded as invalid data areas, and their depth and RGB data are discarded, so that the image near the window is clear, the distinction of wall obstacles is not affected, and there will be no overexposure or underexposure.
[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for selective imaging of images using a depth color camera, which is used for collecting images of windows by a building wall processing robot, and is characterized by the following features: The depth camera performs hole detection, and the color camera performs selective exposure setting. The imaging method includes the following steps: Sp1: Get the depth image of the depth camera; Sp2: Get the image from the RGB camera; Sp3: Attach the corresponding RGB color to each pixel in the depth image; Sp4: Selection based on the data of the depth camera: Combined with the logic of equipment construction, the field of view of the depth camera is limited to a certain range d through hardware or software. Objects outside d are considered to have no relationship with the current construction of the wall construction equipment, and both depth data and RGB data can be discarded; areas without depth data due to metal reflection or beyond the field of view are regarded as invalid data areas, and their depth and RGB data are discarded; the depth data within the distance d from the camera is retained, and the corresponding RGB point coordinates are calculated, and the calculated RGB points are saved as a set; In this method, based on the image generated by the ISP hardware, points with matching depth data are selected, and exposure statistics are used to evaluate the exposure quality of the current image; For the selected RGB points, a color image is generated using fixed exposure setting parameters according to the intensity of the photoelectric signal sent back by the camera's photosensitive chip; The generated color image will be incomplete in the empty areas. Assign 255 to the three RGB color channels to fill the color. The non-empty areas will be displayed normally without underexposure or overexposure, which will not affect the obstacle detection of the algorithm.
2. The image selective imaging method of a depth color camera according to claim 1, characterized in that: The method of attaching a corresponding RGB color to each pixel in the depth image includes: Sp3-1: Construct a three-dimensional vector p_ir = (x, y, z), where x, y are the pixel coordinates of the point, and z is the depth value of the pixel; Sp3-2: Use the inverse of the intrinsic matrix H_ir and multiply it by p_ir to get the corresponding spatial point coordinates P_ir, P_ir = (x1, y1, z1), which means that in the depth coordinate system, the xyz distances from the origin of the depth camera are (x1, y1, z1), respectively. The intrinsic matrix H_ir is obtained by calibration by the camera manufacturer or calibration board; Sp3-3: Since P_ir is the coordinate of the point in the depth coordinate system, convert it to the coordinate system of the RGB camera, multiply P_ir by the rotation matrix R, add a translation vector T, and get P_rgb, which is the coordinate of the point in the RGB coordinate system. The positional relationship between the depth camera and the RGB camera is represented by the rotation matrix R and the translation vector T, which are provided by the manufacturer or calibrated by the calibration plate; Sp3-4: Multiply the intrinsic parameter matrix H_rgb of the RGB camera by P_rgb to get p_rgb, which is also a three-dimensional vector. Its x and y coordinates are the pixel coordinates of the point in the RGB image. Sp3-5: Perform the above operation on each pixel in the depth image to obtain the registered depth map. Each point in the depth map corresponds to a pixel in the RGB image.