A RGB-D image acquisition system
By combining linear array camera, surface array camera and linear laser, dynamic acquisition and synthesis of RGB-D images is achieved, which solves the problems of large size, slow calculation and low resolution of existing equipment, and improves the measurement accuracy of weld detection and speed.
Patent Information
- Application Number
- CN202211466845.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-11-22
AI Technical Summary
The existing RGB-D image acquisition equipment is large in size, has a large amount of calculation, cannot enter cramped space, and has low resolution, which cannot meet the needs of high-precision weld detection.
The combination of linear array camera and surface array camera is adopted, and through multiple linear lasers and image processing modules, dynamic continuous acquisition and synthesis of depth images and RGB images is realized, improving the resolution and measurement accuracy of the image.
Realize high-precision dynamic measurements in a narrow space, improve image acquisition speed and resolution, and improve the measurement accuracy of weld detection.
Smart Images

Figure CN115942096B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image acquisition, and in particular to an RGB-D image acquisition system. Background Art
[0002] At present, RGB-D images are usually used in the extraction process of 3D objects, such as the extraction of weld toes during weld inspection. However, existing RGB-D image acquisition equipment, whether based on RGB array cameras combined with structured light projection or based on time-of-flight methods, has corresponding problems and cannot meet the needs of high-precision extraction: First, they are large in size and cannot enter cramped spaces. When inspecting small and complex structural parts, they are prone to optical path or structural interference; second, the amount of calculation is large and the calculation is slow. It is impossible to measure continuously and dynamically, and cannot be applied to online measurement scenarios when the workpiece is in motion; third, in order to ensure the controllability of the amount of calculation, the resolution of current RGB-D cameras is relatively low, and it is impossible to obtain fine points.
[0003] In summary, it is necessary to design an RGB-D image acquisition system to solve the above problems. Summary of the invention
[0004] The present invention provides an RGB-D image acquisition system, which solves the problems of the current RGB-D camera, such as large size, easy interference, slow acquisition and calculation, inability to perform dynamic measurement, and low resolution.
[0005] In order to achieve the above object, the present invention adopts the following technical scheme:
[0006] A RGB-D image acquisition system, comprising:
[0007] light source;
[0008] A linear array camera, which forms a first field of view along the Y direction and is used to collect an RGB image of the object to be tested;
[0009] A plurality of line lasers are located on both sides of the line array camera, and the illumination planes of the plurality of line lasers are coplanar and coincide with the central plane of the line array camera;
[0010] An area array camera, which forms a second field of view along the X direction and the Y direction, and is used to collect laser line images of the plurality of line lasers;
[0011] The optical axis of the line array camera is coplanar with the optical axis of the area array camera, and the width of the second field of view in the Y direction is equal to the width of the first field of view in the Y direction;
[0012] An image processing module, which is used to calculate a depth image based on the received laser line image; and is also used to synthesize the depth image and the RGB image into the RGB-D image.
[0013] In some embodiments of the present invention, the trigger frequency of the line laser is the same as the trigger frequency of the line array camera, and the line laser and the line array camera are triggered alternately.
[0014] In some embodiments of the present invention, when the line array camera acquires the RGB image, the line laser is in an off state.
[0015] In some embodiments of the present invention, the length of the laser line formed by the line laser on the plate is not less than the width of the first field of view in the Y direction.
[0016] In some embodiments of the present invention, when the workpiece to be measured is in a stationary state, the position of the laser line image acquired by the area array camera coincides with the position of the RGB image acquired by the line array camera.
[0017] In some embodiments of the present invention, during the movement of the workpiece to be measured, the laser line image acquired by the area array camera and the RGB image acquired by the line array camera are spatially staggered.
[0018] In some embodiments of the present invention, the image processing module is further used to upsample the RGB image and the depth image along the X direction by a factor of 2 in length to obtain an interpolated RGB image and an interpolated depth image.
[0019] In some embodiments of the present invention, the image processing module is further used to add the matrix of the interpolated depth image in the RGB direction to complete the synthesis of the RGB-D image.
[0020] In some embodiments of the present invention, the number of pixels of the RGB image in the Y direction is m, and the number of pixels of the depth image in the Y direction is n; when m>n, the image processing module is further used to upsample the depth image in the Y direction until the number of pixels in its Y direction is equal to m; when m<n, the image processing module is further used to upsample the RGB image in the Y direction until the number of pixels in its Y direction is equal to n.
[0021] The technical solution of the present invention has the following technical effects compared with the prior art:
[0022] The present invention cooperates with the area array camera and the line laser and adds a line array camera, so that it can dynamically and continuously measure and calculate the depth image and RGB image of the test piece in a relatively small space, thereby improving the speed of image acquisition; at the same time, because the line array camera and the line laser are triggered alternately, the depth image and the RGB image are intertwined in space, and a high-quality RGB-D image can be obtained after fusion, thereby improving the resolution of the image.
[0023] In addition, the RGB-D image synthesized by the depth image and the RGB image has high accuracy, which is beneficial to improving the subsequent extraction and measurement accuracy of the test piece. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0025] Figure 1 It is a schematic diagram of a first field of view of the area array camera shown in the embodiment.
[0026] Figure 2 It is an image schematic diagram of the synthesis process of the RGB-D image shown in the embodiment.
[0027] Figure 3 It is a schematic diagram of the process of obtaining the RGB-D image shown in the embodiment.
[0028] Figure 4 It is a schematic diagram of the process of extracting the 3D data of the weld toe according to the RGB-D image shown in the embodiment.
[0029] Reference numerals: 100 - linear array camera; 200 - area array camera; 310 - first line laser; 320 - second line laser; 400 - light source. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0032] Example 1
[0033] Reference Figure 1 As shown, a RGB-D image acquisition system includes:
[0034] Light source 400;
[0035] A linear array camera 100, which forms a first field of view along the Y direction and is used to collect an RGB image of the object to be tested;
[0036] A plurality of line lasers are located on both sides of the line array camera 100, and the illumination planes of the plurality of line lasers are coplanar and coincide with the central plane of the line array camera 100;
[0037] An area array camera 200, which forms a second field of view along the X direction and the Y direction, and is used to collect laser line images of the plurality of line lasers;
[0038] The optical axis of the line array camera 100 is coplanar with the optical axis of the area array camera 200, and the width of the second field of view in the Y direction is equal to the width of the first field of view in the Y direction;
[0039] An image processing module (not shown in the figure) is used to calculate a depth image based on the received laser line image; and is also used to synthesize the depth image and the RGB image into the RGB-D image.
[0040] Specifically, in this embodiment, the line laser includes a first line laser 310 and a second line laser 320 , which are located on both sides of the line array camera 100 , and the illumination plane of the first line laser 310 is coplanar with the illumination plane of the second line laser 320 , and coincides with the central plane of the line array camera 100 .
[0041] In some embodiments of the present invention, the triggering frequency of the first line laser 310 and the second line laser 320 is the same as the triggering frequency of the line array camera 100, and the line lasers and the line array camera 100 are triggered alternately; the laser line image and the RGB image on the weld surface are collected. When collecting the RGB image, the line laser must be turned off to prevent the line array camera 100 from failing to collect the RGB image with high color distinction.
[0042] In some embodiments of the present invention, Figure 1 The solid line portion is the first field of view formed by the linear array camera 100 , the dotted line portion is the second field of view formed by the area array camera 200 ; the dotted line portion is the laser lines of the first line laser 310 and the second line laser 320 .
[0043] The setting of the line array camera 100 satisfies the following conditions: the central plane of the first field of view coincides with the illumination plane of each line laser; the optical axis of the line array camera 100 is coplanar with the optical axis of the area array camera 200, and their common plane is perpendicular to the Y direction; the field of view size in the Y direction of the first field of view is the same as the field of view size in the Y direction of the second field of view, and they are parallel to each other.
[0044] Through the above conditions, when the test piece is in a stationary state, the position of the laser line image captured by the area array camera 200 coincides with the position of the RGB image captured by the line array camera 100 .
[0045] During the movement of the test piece, the laser line image captured by the area array camera 200 and the RGB image captured by the line array camera 100 are spatially interlaced with each other.
[0046] Using the line array camera 100 to collect RGB images can ensure a sufficiently high sampling frequency, specifically not less than 1 kHz, to meet full dynamic sampling requirements.
[0047] Continue to refer to Figure 1 As shown, the second field of view formed by the area array camera 200 is rectangular, the length direction of the second field of view is parallel to the X direction, and the width direction is parallel to the Y direction; during the image acquisition process, the plate with the weld moves along the X direction, the surface normal of the weld is parallel to the Z direction, and the welding direction of the weld is parallel to the X direction.
[0048] In some embodiments of the present invention, the illumination plane formed by the first line laser 310 and the second line laser 320 is referenced to Figure 1As shown by the dashed line in the figure, the lengths of the laser lines formed by the first line laser 310 and the second line laser 320 on the sheet are not less than the width of the first field of view in the Y direction. The projection of the laser line formed by the line laser on the weld surface can fill the field of view of the area array camera 200 in the Y direction. This can not only meet the line accuracy requirements but also avoid the optical path interference of the laser. During the acquisition process of the depth image, laser is used as the light source, and the area array camera 200 can perform high-frame-rate image acquisition with a low exposure time, achieving a sampling frequency of not less than 1 kHz.
[0049] In some embodiments of the present invention, when the line array camera 100 acquires the RGB image, the first line laser 310 and the second line laser 320 are in the off state.
[0050] In some embodiments of the present invention, regarding the light source 400, it is located between the area array camera 200 and the line array camera 100, and the light source 400 uses a white light bar light source.
[0051] The image processing module is also used to synthesize the depth image and the RGB image into an RGB-D image;
[0052] Since the RGB image and the depth image are acquired by alternately triggering the line array camera 100 and the line laser, the acquired RGB image and depth image are spatially interleaved; linear interpolation needs to be performed on the RGB image and the depth image;
[0053] Refer to Figure 2 and Figure 3 As shown, the synthesis of the RGB-D image specifically includes the following steps:
[0054] S1. Upsample the RGB image and the depth image along the X direction by a factor of 2 in length to obtain the interpolated RGB image and the interpolated depth image; when the number of pixels of the RGB image and the depth image in the Y direction is different, the image processing module first upsamples in the Y direction until the number of pixels in the width direction is equal to the larger value of the number of pixels in the Y direction of the RGB image and the depth image; for example, the number of pixels of the RGB image in the Y direction is m, and the number of pixels of the depth image in the Y direction is n; when m > n, the image processing module is also used to upsample the depth image in the Y direction until the number of pixels in its Y direction is equal to m; when m < n, the image processing module is also used to upsample the RGB image in the Y direction until the number of pixels in its Y direction is equal to n.
[0055] S2. The interpolated RGB image and the interpolated depth image have the same size and the corresponding positions in the three-dimensional space are exactly the same; the matrix of the interpolated depth image is added in the direction of RGB to complete the synthesis of the RGB-D image.
[0056] In some embodiments of the present invention, the working principle of the acquisition system is described in combination with the application of the RGB-D image acquisition system in the field of weld detection. Figure 4 As shown:
[0057] After the measurement starts, the light source 400 is turned on first, and the object to be measured moves along the X direction relative to the area array camera 200 and the line array camera 100 to enter the measurement cycle;
[0058] In one measurement cycle, the first line laser 310 and the second line laser 320 are first turned on, and the area array camera 200 collects laser line images; then the first line laser 310 and the second line laser 320 are turned off, and the line array camera 100 collects RGB images of the weld surface;
[0059] As the object to be tested moves, multiple measurement cycles are repeated to obtain multiple laser line images and RGB images, and the light source 400 is turned off;
[0060] Then, the image processing module calculates a depth image based on the laser line image based on the laser triangulation method, and synthesizes an RGB-D image according to the above steps;
[0061] For the extraction process of weld toe coordinates:
[0062] The image processing module first extracts the RGB-D image into a standard depth image and a standard RGB image;
[0063] Then, the standard RGB image is converted into a grayscale image and then Gaussian filtering is used to remove interference, that is, to remove noise interference during image acquisition;
[0064] Then, the Sobel operator is used to calculate the gradient value along the Y direction of the grayscale image to obtain the first-order partial derivative matrix of each pixel point along the Y direction, and then the absolute value is solved to obtain the amplitude matrix Gy of the gradient image along the Y direction; in addition, since the X direction is parallel to the welding direction, the position of the actual weld toe may not be different along the X direction, so the gradient amplitude matrix Gx along the X direction will not have obvious amplitude difference at the weld toe relative to other pixel positions on the left and right of the weld toe, which is essentially a noise interference term, so Gx shall not be involved in subsequent calculations;
[0065] Then, the two points with the largest amplitude are extracted in turn from each row of the amplitude matrix Gy along the X direction, which are the left and right weld toes of this row, and their corresponding XY coordinates are the left and right 2D coordinates of the weld toe;
[0066] Based on the principle that the weld toe positions of adjacent rows will not change suddenly under high-frequency sampling, the step S63 further includes performing mean and / or median filtering on the 2D coordinates along the X direction to remove weld toe position abnormalities caused by noise;
[0067] Finally, according to the 2D coordinates of the weld toe, the depth of the corresponding 2D coordinates is queried in the depth image to obtain the 3D coordinates of the weld toe, thus completing the 3D coordinate extraction of the weld toe.
[0068] The technical solution of the present invention has the following technical effects compared with the prior art:
[0069] The present invention achieves dynamic and continuous measurement of RGB-D images of weld surface in a relatively narrow space by cooperating with area array cameras and line lasers and adding line array cameras. In addition, the weld toe coordinates are extracted by synthesizing the RGB-D image using the depth image and the RGB image, so that the extraction result is more accurate and the measurement accuracy of weld surface size and shape and position is improved.
[0070] In the description of the above embodiments, specific features, structures, materials or characteristics may be combined in a suitable manner in any one or more embodiments or examples.
[0071] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. An RGB-D image acquisition system, characterized in that, comprising: a light source; a line array camera that forms a first field of view in the Y direction for acquiring the RGB image of the workpiece to be measured; a plurality of line lasers located on both sides of the line array camera, the illumination planes of the plurality of line lasers being coplanar and coinciding with the central plane of the line array camera; a area array camera that forms a second field of view in the X and Y directions for acquiring the laser line images of the plurality of line lasers; wherein the optical axis of the line array camera and the optical axis of the area array camera are coplanar, and the width of the second field of view in the Y direction is equal to the width of the first field of view in the Y direction; an image processing module that calculates a depth image based on the received laser line images; and is also used to synthesize the depth image and the RGB image into the RGB-D image; the trigger frequency of the line laser is the same as the trigger frequency of the line array camera, and the line laser and the line array camera are alternately triggered; the image processing module is also used to upsample the RGB image and the depth image along the X direction by a factor of 2 in length to obtain the interpolated RGB image and the interpolated depth image; the image processing module is also used to add the matrix of the interpolated depth image in the RGB direction to complete the synthesis of the RGB-D image.
2. The RGB-D image acquisition system according to claim 1, characterized in that when the line array camera acquires the RGB image, the line laser is in the off state.
3. The RGB-D image acquisition system according to claim 1, characterized in that, the length of the laser line formed by the line laser on the plate is not less than the width of the first field of view in the Y direction.
4. The RGB-D image acquisition system according to claim 1, characterized in that, when the workpiece to be measured is in a stationary state, the position of the laser line image acquired by the area array camera coincides with the position of the RGB image acquired by the line array camera.
5. The RGB-D image acquisition system according to claim 1, characterized in that, during the movement of the workpiece to be measured, the laser line image acquired by the area array camera and the RGB image acquired by the line array camera are spatially staggered.
6. The RGB-D image acquisition system according to claim 1, characterized in that, the number of pixels of the RGB image in the Y direction is m, and the number of pixels of the depth image in the Y direction is n; when m > n, the image processing module is also used to upsample the depth image in the Y direction until the number of pixels in its Y direction is equal to m; when m < n, the image processing module is also used to upsample the RGB image in the Y direction until the number of pixels in its Y direction is equal to n.
