Carrying platform control mode and hardware composition based on machine vision guidance

Abstract

The invention provides a carrying platform control mode and hardware composition based on machine vision guidance, and relates to the technical field of carrying platform control. The carrying platform control hardware based on machine vision guidance comprises an omni-directional carrying device and a target detection target board, the omni-directional carrying device comprises a parking mechanism, anti-collision radars are fixedly arranged on the two sides and the centers of the front end and the rear end of the parking mechanism, steering wheel bodies are fixedly arranged at the four corners of the lower end face of the parking mechanism, and the steering wheel bodies are fixedly connected with the parking mechanism. Auxiliary steering wheels are fixedly arranged on the two sides, close to the center, of the lower end face of the parking mechanism, and a plurality of lithium batteries are embedded in the lower end face of the parking mechanism. According to the carrying platform, four sets of steering wheels are used for achieving omni-directional movement, space posture adjustment is achieved in combination with a servo parking lifting device, meanwhile, a binocular vision detection device is installed on the transfer platform, and in combination with a target plate, the relative posture relation between the carrying platform and a target station can be recognized and detected, and station positioning navigation control is conducted.

Description

Technical Field

[0001] The present invention relates to the technical field of carrier platform control, and in particular to a carrier platform control method and hardware composition based on machine vision guidance. Background Art

[0002] The navigation and control schemes commonly used in traditional large-component transport platforms are wired guidance, QR code guidance, laser radar guidance, and deep vision guidance. Wired guidance uses pre-buried wires on the ground for navigation, which is suitable for fixed-path scenarios; QR code guidance sets a QR code on the ground or at the target station, and the transport platform determines the position by identifying the QR code, which is suitable for scenarios that require precise positioning; laser radar guidance uses laser radar to scan the surrounding environment, establish an environmental map and achieve navigation, which is suitable for scenarios that require high-precision positioning and obstacle avoidance; deep vision guidance uses deep vision technology to obtain the three-dimensional position information of the target, which is suitable for scenarios that require three-dimensional spatial positioning.

[0003] However, these existing technologies have the following shortcomings when applied to the scenarios of high-precision transportation of large aircraft parts: the problem of station fixity. These technologies are mainly suitable for application scenarios with fixed stations, but have poor flexibility and adaptability for non-fixed relative positions; insufficient positioning accuracy. For application scenarios of high-precision transportation of large aircraft parts, the positioning accuracy of existing technologies may not meet the requirements, especially the limited recognition accuracy and robustness of traditional QR code guidance and lidar guidance in complex backgrounds and dynamic environments; limited spatial posture adjustment capabilities. After the carrier platform arrives at the station, the existing technology often lacks the ability to finely adjust the platform in six degrees of freedom (left and right, front and back, up and down, roll, pitch, and roll), making it difficult to meet the needs of high-precision transportation. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a control method and hardware composition of a carrier platform based on machine vision guidance, which solves the problem of weak fine adjustment capability of the prior art.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a carrier platform control hardware based on machine vision guidance, including an omnidirectional carrier and a target detection target plate, the omnidirectional carrier including a parking mechanism, anti-collision radars are fixedly arranged on both sides and at the center of the front and rear ends of the parking mechanism, a steering wheel body is fixedly arranged at the four corners of the lower end surface of the parking mechanism, auxiliary steering wheels are fixedly arranged on both sides of the lower end surface of the parking mechanism near the center, a plurality of lithium batteries are embedded in the lower end surface of the parking mechanism, an electric control box is fixedly arranged on the upper end surface of the parking mechanism near the front end and the rear end, and a visual detection device is fixedly arranged at the center of the opposite sides of the two electric control boxes;

[0006] The target detection target plate includes a detection target plate body, a cross positioning target is arranged on the front side of the detection target plate body, a QR code is arranged on the front side of the detection target plate body located at the center of the cross positioning target, cooperative target and calibration target ball mounting holes are arranged at the front end of the detection target plate body located at the four areas of the cross positioning target, and laser ranging positioning targets are arranged at the front end of the detection target plate body located at the two lower areas of the cross positioning target.

[0007] Preferably, the visual inspection device comprises a mounting plate, a cross laser is fixedly arranged at the center of the lower end surface of the mounting plate, and light sources are fixedly arranged on both sides of the rear end of the mounting plate near the center.

[0008] Preferably, a laser rangefinder is fixedly provided on both sides of the rear end of the mounting plate close to the light source, and a visual camera is fixedly provided on both sides of the rear end of the mounting plate close to the laser rangefinder.

[0009] Preferably, the mounting plate is fixedly connected to the electric control box, the visual camera, laser ranging and light source are all located inside the electric control box, and the output end is emitted through the hole of the mounting plate.

[0010] Preferably, connection and mounting holes are provided at two corners at the upper front end of the detection target plate body.

[0011] The control method of the carrier platform based on machine vision guidance specifically includes the following steps:

[0012] S1. Use the omnidirectional steering wheel body wheel set to achieve omnidirectional movement of the omnidirectional carrier;

[0013] S2. Determine the relative position relationship between the omnidirectional carrier and the target station by detecting the QR code mark on the target board;

[0014] S3. Perform preliminary posture adjustment on the spot according to the measurement data, then the omnidirectional carrier moves forward, and during the forward process, the posture is corrected in real time through the data measured by the visual detection device, and fine-tuning is performed after reaching the parking point.

[0015] Preferably, the visual detection device improves the accuracy of landmark point positioning through a sub-pixel edge detection algorithm based on the lens defocus mechanism, and then uses SVD singular value decomposition to align the identified cooperative target points.

[0016] Preferably, in the method, when the target detection target plate exceeds the distance of the visual detection device, the rough position adjustment of the omnidirectional carrier is performed through two laser ranging and a cross laser, and the laser ranging calculates the roll angle and the front-to-back distance between the omnidirectional carrier and the target detection target plate through the distance difference, and the relationship between the cross laser and the cross positioning target on the target detection target plate represents the up and down, left and right and rotation angles of the omnidirectional carrier.

[0017] The present invention provides a control method and hardware composition of a carrier platform based on machine vision guidance. It has the following beneficial effects:

[0018] The present invention provides a control method and hardware composition of a transport platform guided by machine vision. In the application scenario of high-precision transportation of large aircraft parts, the transport platform guided by visual space positioning can adapt to non-fixed relative positions, improve the flexibility and adaptability of the system, and realize high-precision station positioning. Especially in complex backgrounds and dynamic environments, by using two-dimensional code markers with high-contrast edges and fault tolerance, the accuracy and robustness of positioning are improved, and the platform has a six-degree-of-freedom spatial posture adjustment capability, ensuring that the platform can perform fine posture adjustment after arriving at the station, meeting the needs of high-precision transportation, and combining non-visual guidance methods, rough positioning is performed through laser ranging and cross lasers, which increases the reliability and safety of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the omnidirectional carrier of the present invention when viewed from the bottom;

[0020] Figure 2 It is a schematic diagram of the omnidirectional carrier of the present invention from the front view axle side;

[0021] Figure 3 It is a schematic diagram of the front view axis side of the visual inspection device of the present invention;

[0022] Figure 4 It is a schematic diagram of the rear-view axis side of the visual inspection device of the present invention;

[0023] Figure 5 It is a front view schematic diagram of a target detection target plate of the present invention;

[0024] Figure 6 It is a schematic diagram of the system control flow of the present invention;

[0025] Figure 7 This is a schematic diagram of the cooperative target positioning principle of the present invention;

[0026] Figure 8 This is a schematic diagram showing the distribution of the edge grayscale value of a mark point in a blurred camera image and its first-order gradient and second-order gradient along the X-axis or Y-axis when the present invention uses a Gaussian function as a lens defocus diffusion function;

[0027] Fig. 9 This is a schematic diagram of the pixel coordinates of the marker point image using the center of the ellipse formed by the marker point in the camera image in the present invention;

[0028] Fig.10 It is a schematic diagram of the overall algorithm flow chart of the present invention;

[0029] Fig.11 It is a schematic diagram of the non-visual guidance principle and process of the present invention.

[0030] Among them, 1. omnidirectional carrier; 2. target detection target plate; 101. parking mechanism; 102. steering wheel body; 103. auxiliary steering wheel; 104. lithium battery; 105. visual detection device; 106. anti-collision radar; 107. electrical control box; 10501. mounting plate; 10502. cross laser; 10503. visual camera; 10504. laser ranging; 10505. light source; 201. detection target plate body; 202. connection mounting hole; 203. cooperative target; 204. calibration target ball mounting hole; 205. laser ranging positioning target; 206. QR code; 207. cross positioning target. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] like Figure 1-11 As shown, the embodiment of the present invention provides a carrier platform control hardware based on machine vision guidance, including an omnidirectional carrier 1 and a target detection target plate 2. The omnidirectional carrier 1 includes a parking mechanism 101. Anti-collision radars 106 are fixedly arranged on both sides and at the center of the front and rear ends of the parking mechanism 101. A steering wheel body 102 is fixedly arranged at the four corners of the lower end surface of the parking mechanism 101. Auxiliary steering wheels 103 are fixedly arranged on both sides of the lower end surface of the parking mechanism 101 near the center. A plurality of lithium batteries 104 are embedded in the lower end surface of the parking mechanism 101. Electric control boxes 107 are fixedly arranged on the upper end surface of the parking mechanism 101 near the front end and the rear end. Visual detection devices 105 are fixedly arranged at the center of the opposite sides of the two electric control boxes 107.

[0033] The target detection target plate 2 includes a detection target plate body 201, a cross positioning target 207 is arranged on the front side of the detection target plate body 201, a QR code 206 is arranged on the front side of the detection target plate body 201 located at the center of the cross positioning target 207, cooperation targets 203 and calibration target ball mounting holes 204 are arranged at the front end of the detection target plate body 201 located at the four areas of the cross positioning target 207, and laser ranging positioning targets 205 are arranged at the front end of the detection target plate body 201 located at the two lower areas of the cross positioning target 207.

[0034] The visual detection device 105 includes a mounting plate 10501, a cross laser 10502 is fixedly arranged at the center of the lower end surface of the mounting plate 10501, and light sources 10505 are fixedly arranged on both sides of the center of the rear end of the mounting plate 10501. Laser rangefinders 10504 are fixedly arranged on both sides of the rear end of the mounting plate 10501 near the light source 10505, and visual cameras 10503 are fixedly arranged on both sides of the rear end of the mounting plate 10501 near the laser rangefinder 10504. The mounting plate 10501 is fixedly connected to the electric control box 107, and the visual camera 10503, the laser rangefinder 10504, and the light source 10505 are all located inside the electric control box 107, and the output end is emitted through the hole of the mounting plate 10501. The two corners of the upper front end of the detection target plate body 201 are provided with connection mounting holes 202.

[0035] The control method of the carrier platform based on machine vision guidance specifically includes the following steps:

[0036] S1. Using the omnidirectional steering wheel body 102 wheelset to achieve omnidirectional movement of the omnidirectional carrier 1;

[0037] S2. Determine the relative position relationship between the omnidirectional carrier 1 and the target station by detecting the QR code 206 on the target plate 2;

[0038] S3. Perform preliminary posture adjustment on the spot according to the measurement data, then the omnidirectional carrier 1 moves forward, and during the moving process, the posture is corrected in real time through the measurement data of the visual detection device 105, and fine-tuning is performed after reaching the parking point.

[0039] The visual detection device 105 improves the accuracy of the positioning of the marker point by using a sub-pixel edge detection algorithm based on the lens defocus mechanism, and then uses SVD singular value decomposition to align the identified cooperative target (203) points. In the method, when the target detection target plate 2 exceeds the distance of the visual detection device 105, the rough position adjustment of the omnidirectional carrier 1 is performed by two laser rangefinders 10504 and a cross laser 10502. The laser rangefinder 10504 calculates the roll angle and the front-to-back distance between the omnidirectional carrier 1 and the target detection target plate 2 by the distance difference. The relationship between the cross laser 10502 and the cross positioning target 207 on the target detection target plate 2 represents the up-down, left-right and rotation angles of the omnidirectional carrier 1.

[0040] Specifically:

[0041] Visual system recognition principle (such as Figure 6 (shown)

[0042] Target plate positioning:

[0043] When the target plate is mounted on a non-fixed position, in order to improve the efficiency and accuracy of the visual system positioning, an ARUCO two-dimensional code mark is set on the target plate. This special mark has a high-contrast edge, can be clearly seen under various backgrounds, and has fault tolerance, so that even if part of the mark is blocked, the system can work normally. The basic principle of ARUCO two-dimensional code positioning is to match the predefined ARUCO marking pattern by detecting the edges and line patterns in the image. The control system in the present invention determines the four corner points of the pattern based on the outline of the two-dimensional code. Using these four corner points, the internal and external parameters of the pattern, such as the rotation matrix, translation vector, etc., can be calculated. Based on these parameters, the pattern can be placed in three-dimensional space. According to the positioning of the two-dimensional code, the position of the cooperative target can be accurately extracted for further high-precision positioning.

[0044] Cooperative targeting principle (e.g. Figure 7 shown):

[0045] Lens defocus has a certain impact on the positioning accuracy of the marker points. Figure 7 As shown in the figure, in the actual camera imaging process, several rays of light emitted by the object point are refracted by the lens and incident on the camera imaging plane to form an image. When the lens aperture is large, if the object point is not in the camera lens focusing plane, the refracted light will be incident on different positions of the imaging plane, forming a light spot. This phenomenon is called "lens defocus". The defocused image can be regarded as the convolution result of the focused image and the point spread function (PointSpread Function). Usually, the point spread function is estimated using a Gaussian function.

[0046] During the calibration process, due to the influence of lens defocus, the edges of the marker points in the collected calibration plate image are blurred, making it impossible for the traditional Canny operator to obtain the accurate edge position of the marker points, thus affecting the calibration accuracy of the system. Therefore, a sub-pixel edge detection algorithm based on the lens defocus mechanism is used.

[0047] like Figure 8 The figure shows the distribution of the grayscale value of the edge of the mark point in the blurred camera image and its first-order gradient and second-order gradient along the X-axis or Y-axis when the Gaussian function is used as the lens defocus diffusion function. The exact position of the edge should be the point where its second-order gradient is zero. Therefore, the sub-pixel edge detection algorithm is as follows:

[0048] (1) Use Gaussian filtering and threshold segmentation to preprocess the original image to eliminate noise;

[0049] (2) Use the Canny operator to obtain the pixel-level edge position of the marker point, and use the edge position to extract the region of interest (ROI) from the image to reduce the amount of calculation and improve the reliability of the algorithm. The region of interest is set to include the entire marker point edge and the extended edge, and the scale of the extended edge is proportional to the scale of the marker point edge;

[0050] (3) Calculate the first-order gradient and second-order gradient of the grayscale value of all points in the ROI to find candidate points. All points whose first-order gradient is less than the average value in the region of interest will not become edge candidate points and are therefore excluded. Then, find the area where the positive and negative signs of the second-order gradient change along the X-axis and Y-axis respectively, as the edge candidate area;

[0051] (4) In the candidate area, the second-order gradient of the gray value is linearly fitted using the least squares method, and then the point where the linear fitting function value is equal to zero is calculated, which is the exact position of the edge.

[0052] The center of the ellipse formed by the marker points in the camera image is used as the pixel coordinate of the marker point image. Fig. 9 As shown in the figure, since the actual imaging process of the camera is a transmission transformation and contains lens distortion, there is a certain deviation between the center of the ellipse formed by the spatial circle projection and the projection of the center of the spatial circle, resulting in calibration errors. The deviation caused by the linear transmission transformation is taken into account and compensated in the calibration process. However, the actual lens will have nonlinear lens distortion due to reasons such as uneven lens thickness and assembly errors. The resulting deviation is not taken into account, resulting in the algorithm being unable to achieve good compensation accuracy when the lens distortion is obvious. Minh Vo et al. proposed using the initial calibration results to convert the captured calibration plate image into a front view, and then use template matching to optimize the position of the marker points, thereby suppressing the influence of perspective projection and lens distortion on the camera calibration accuracy; however, the disadvantage of this method is that the amount of calculation required to convert to the front view is large and requires a special concentric circle calibration plate. When a dot calibration plate is used, the accuracy improvement is limited.

[0053] Therefore, an iterative compensation method for marker point coordinates based on the camera imaging model is used, while considering the influence of linear transmission transformation and nonlinear lens distortion to improve the calculation accuracy of the marker point center, thereby further improving the system calibration accuracy.

[0054] For a space circle in the camera coordinate system, the three-dimensional coordinates of its center in the calibration plate coordinate system are (x og ,y og , z og ), the three-dimensional coordinates in the camera coordinate system are (x oc ,y oc , z oc ), the normal vector is (n xoc , nyoc , n zoc ), with a radius of r oc , then the points on its edge satisfy

[0055]

[0056] Let the camera image pixel coordinate corresponding to the center of the space circle be q c , the center of the ellipse formed by the projection of the edge of the space circle is q e The steps of the circle center iterative compensation algorithm are as follows:

[0057] (1) Using the center of the ellipse projected by the landmark point {q e}, perform initial calibration on the camera and obtain R gc , t gc Initial estimate of (0) R gc , (0) t gc ;

[0058] (2)Use (0) R gc , (0) t gc , calculate (x oc ,y oc , z oc ), (n xoc , n yoc , n zoc ):

[0059]

[0060] (3) construct a set of virtual space circles;

[0061] (4) The virtual space circle generated by the camera imaging model containing lens distortion is projected onto the camera imaging plane to obtain the pixel coordinate q corresponding to the center of the virtual space circle w And the ellipse corresponding to the edge of the virtual circle, and then the ellipse center q can be obtained by ellipse fitting v ;

[0062] (5) Calculate the projection q of the center of the space circle after compensation d :

[0063] q d =q e +(q w -q v )

[0064] (6) Use q d Recalibrate the camera and update the camera internal parameters (intrinsic parameter matrix and distortion parameters) and R gc , tgc .

[0065] (7) Repeat steps (1) to (6) until q v With q e If the distance between them is less than the preset threshold, it means that the position of the virtual calibration plate converges to the position of the actual calibration plate and the compensation of the spatial circle projection deviation is completed.

[0066] The overall algorithm flow chart is as follows: Fig.10 After the iterative compensation of the spatial circle projection deviation is completed, more accurate coordinates of the center of the circular marker can be obtained.

[0067] Finally, SVD singular value decomposition is used to align the identified cooperative target points to obtain the relative pose relationship between the camera and the target.

[0068] Principle and process of vision-guided delivery platform

[0069] First, the carrier platform is navigated to a fixed station by traditional fixed station navigation. Then, the target positioning target plate on the mobile station is identified by the visual system for relative posture measurement. Then, the initial posture adjustment is performed on the spot through the measurement data. Then, the carrier platform moves forward, and the posture is corrected in real time through the visual measurement data during the process. Then, after reaching the parking point, fine-tuning is performed at the point, and the adjustment directions include left and right, front and back, and roll. Then, the vehicle is extended to park. Then, fine-tuning is performed through parking, including up and down, pitch, and roll.

[0070] Non-visual guidance principles and processes (such as Fig.11 (shown)

[0071] After the target plate exceeds the detection distance of the visual system, the present invention can guide the carrier device through a non-visual system using two laser range finders and a cross laser.

[0072] The two laser rangefinders can calculate the rolling angle and front-to-back distance between the carrier and the target plate through the distance difference. The relationship between the cross laser and the cross laser target on the target plate can indicate the up-down, left-right and rotation angles.

[0073] The vehicle can be first adjusted to a rough position using non-visual guidance methods.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A carrier platform control hardware based on machine vision guidance, comprising an omnidirectional carrier device (1) and a target detection target plate (2), characterized in that: The omnidirectional carrier (1) comprises a parking mechanism (101), anti-collision radars (106) are fixedly arranged on both sides and at the center of the front and rear ends of the parking mechanism (101), a steering wheel body (102) is fixedly arranged at the four corners of the lower end surface of the parking mechanism (101), auxiliary steering wheels (103) are fixedly arranged on both sides of the lower end surface of the parking mechanism (101) near the center, a plurality of lithium batteries (104) are embedded in the lower end surface of the parking mechanism (101), an electric control box (107) is fixedly arranged on the upper end surface of the parking mechanism (101) near the front end and the rear end, and a visual detection device (105) is fixedly arranged at the center of the opposite sides of the two electric control boxes (107); The target detection target plate (2) comprises a detection target plate body (201), a cross positioning target (207) is arranged on the front side of the detection target plate body (201), a two-dimensional code (206) is arranged on the front side of the detection target plate body (201) located at the center of the cross positioning target (207), cooperation targets (203) and calibration target ball mounting holes (204) are arranged at the front end of the detection target plate body (201) located at four areas of the cross positioning target (207), and laser ranging positioning targets (205) are arranged at the front end of the detection target plate body (201) located at two areas below the cross positioning target (207).

2. The control hardware of the carrier platform based on machine vision guidance according to claim 1, characterized in that: The visual inspection device (105) comprises a mounting plate (10501), a cross laser (10502) is fixedly arranged at the center of the lower end surface of the mounting plate (10501), and light sources (10505) are fixedly arranged on both sides of the rear end of the mounting plate (10501) near the center.

3. The control hardware of the carrier platform based on machine vision guidance according to claim 2, characterized in that: A laser rangefinder (10504) is fixedly arranged on both sides of the rear end of the mounting plate (10501) close to the light source (10505), and a visual camera (10503) is fixedly arranged on both sides of the rear end of the mounting plate (10501) close to the laser rangefinder (10504).

4. The control hardware of the carrier platform based on machine vision guidance according to claim 3, characterized in that: The mounting plate (10501) is fixedly connected to the electric control box (107); the visual camera (10503), the laser distance measurement (10504), and the light source (10505) are all located inside the electric control box (107); and the output end is emitted through the hole of the mounting plate (10501).

5. The control hardware of the carrier platform based on machine vision guidance according to claim 1, characterized in that: The detection target plate body (201) has connection and mounting holes (202) at two upper corners of the front end.

6. The control method of the carrier platform based on machine vision guidance according to any one of claims 1 to 5, characterized in that: The specific steps include: S1. Using the omnidirectional steering wheel body (102) wheel set to achieve omnidirectional movement of the omnidirectional carrier (1); S2. Determine the relative position relationship between the omnidirectional carrier (1) and the target station by detecting the QR code (206) on the target plate (2); S3. Perform preliminary posture adjustment on the spot based on the measurement data, then the omnidirectional carrier (1) moves forward, and during the moving process, the posture is corrected in real time using the measurement data of the visual detection device (105), and fine-tuning is performed after reaching the parking point.

7. The control method of the carrier platform based on machine vision guidance according to claim 1 is characterized in that: The visual detection device (105) improves the accuracy of positioning the marker points by using a sub-pixel edge detection algorithm based on the lens defocus mechanism, and then uses SVD singular value decomposition to align the identified cooperative target (203) points.

8. The control method of the carrier platform based on machine vision guidance according to claim 1 is characterized in that: In the method, when the target detection target plate (2) exceeds the distance of the visual detection device (105), the rough position adjustment of the omnidirectional carrier (1) is performed by two laser distance measuring devices (10504) and a cross laser (10502). The laser distance measuring device (10504) calculates the rolling angle and the front-to-back distance between the omnidirectional carrier (1) and the target detection target plate (2) by the distance difference. The relationship between the cross laser (10502) and the cross positioning target (207) on the target detection target plate (2) indicates the up-down, left-right and rotation angles of the omnidirectional carrier (1).

Images