Servo positioning method and system based on 3D vision

Abstract

The invention discloses a servo positioning method and system based on 3D vision, and relates to the technical field of drill arm servo positioning, and the method comprises the steps: presetting the relative pose of an RGBD depth camera and a drill arm, and obtaining the 3D vision information of a drilling working face scene through the RGBD depth camera; layering a drilling working face scene into sub-scenes, and positioning a to-be-drilled target point in the sub-scenes; screening areas, performing 3D visual information analysis, fitting a to-be-drilled target area by using a curve fitting method, and positioning a to-be-drilled target point in combination with preset information; and the position to be drilled is positioned in real time, and a servo strategy is adopted to control the drill bit to move towards the target positioning point. According to the servo positioning method based on 3D vision provided by the invention, the edge detection algorithm is adopted to carry out specific-shape region positioning on depth data of different layers, and the precision and speed of coarse positioning are improved. The method has better effects in the aspects of drilling precision, drilling efficiency and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of drill arm servo positioning, and in particular to a servo positioning method and system based on 3D vision. Background Art

[0002] With the continuous development of industrial technology, the application of drill arm systems in mining is becoming more and more common. Its positioning accuracy and drill arm operating efficiency are key factors affecting the progress and quality of the entire project. Traditional drill arm positioning technology mainly relies on mechanical structure and sensor signals. However, in the complex and changeable mining working environment, its positioning accuracy and adaptability are usually greatly limited. Therefore, the study of new drill arm positioning technology can effectively improve the automation level and operating efficiency of the drill arm system in mining work.

[0003] In recent years, 3D vision technology has developed rapidly and has achieved remarkable results in object recognition, scene understanding, navigation and positioning. As one of the important implementation methods of 3D vision technology, RGB-D camera can simultaneously obtain color images and depth information, making it possible to accurately perceive the three-dimensional structure of the target object. Applying RGB-D camera to drill arm servo positioning can realize comprehensive perception and precise control of the drill arm working environment, fully automatically realize the movement and drilling of the drill arm, reduce the inaccurate positioning caused by human interference, thereby improving the positioning accuracy and operating efficiency of the drill arm system while reducing the consumption of human resources. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing drill arm positioning and precise drilling method requires professional technicians to control the drill arm to move to the position to be drilled, and it is difficult to accurately move to the position to be drilled near the final drilling positioning point by manual control. While consuming a lot of manpower, the positioning accuracy is limited, and inaccurate positioning may make the subsequent working environment more complicated and unsafe.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a servo positioning method based on 3D vision, including presetting the relative posture of an RGBD depth camera and a drill arm, and using the RGBD depth camera to obtain 3D visual information of a drilling working face scene; stratifying the drilling working face scene into sub-scenes, and locating the target point to be drilled in the sub-scenes; screening the area, performing 3D visual information analysis and fitting the target area to be drilled using a curve fitting method, and locating the target point to be drilled in combination with the preset information; locating the position to be drilled in real time and using a servo strategy to control the drill head to move to the target positioning point.

[0007] As a preferred solution of the servo positioning method based on 3D vision described in the present invention, wherein: the relative posture of the preset RGBD depth camera and the drill arm, and the 3D visual information of the drilling working face scene obtained by using the RGBD depth camera include the relative posture of the preset RGBD depth camera and the drill arm, so that the camera and the drill bit move synchronously, ensuring that the position and posture of the drill bit in the camera coordinate system remain unchanged;

[0008] The 3D visual information of the drilling working face scene is obtained by using an RGBD depth camera, including the RGB information and depth information of the scene.

[0009] As a preferred solution of the 3D vision-based servo positioning method described in the present invention, wherein: the drilling work surface scene is layered into sub-scenes, and locating the target point to be drilled in the sub-scene includes dividing the drilling work surface scene into multiple sub-scenes using depth segmentation technology.

[0010] As a preferred solution of the 3D vision-based servo positioning method described in the present invention, the 3D vision information analysis includes simultaneously locating the target point to be drilled in RGB vision information and depth vision information, and fusing the two positioning results to achieve accurate and robust positioning of the target point to be drilled.

[0011] As a preferred solution of the 3D vision-based servo positioning method described in the present invention, wherein: the screening area, fitting the target area to be drilled using a curve fitting method, locating the target point to be drilled in combination with preset information includes screening the area as an area where the depth change is greater than a threshold, denoising the sub-scene information, calculating the image gradient and depth gradient, and applying non-maximum suppression and dual threshold detection strategies.

[0012] As a preferred solution of the servo positioning method based on 3D vision described in the present invention, wherein: the real-time positioning of the position to be drilled and the use of a servo strategy to control the drill bit to move to the target positioning point include, when it is not the initial positioning, using the prior information of the last successful positioning to accelerate the speed and efficiency of this positioning;

[0013] The prior information is the spatial coordinates of the camera and the drill in the world coordinate system, the target point to be drilled, the sub-scene in which it is located, and the curve equation of the specific shape geometric area.

[0014] As a preferred solution of the 3D vision-based servo positioning method described in the present invention, the real-time positioning of the position to be drilled and the use of a servo strategy to control the drill bit to move toward the target positioning point includes applying a servo strategy to control the movement of the drill arm, repeating the positioning, control and inspection processes until the relative position relationship between the drill bit and the target position to be drilled meets the preset drilling conditions.

[0015] Another object of the present invention is to provide a servo positioning system based on 3D vision, which can solve the problem of inaccurate positioning of the current drill arm positioning and precise drilling method by applying the RGB-D camera to the drill arm servo positioning.

[0016] As a preferred solution of the 3D vision-based servo positioning system described in the present invention, it includes: a 3D vision information acquisition and depth data processing module, an edge detection and target point fitting module, a servo control and drill bit positioning module;

[0017] The path planning module is used to use the RGBD depth camera to obtain 3D visual information of the working face drilling scene, output the depth change interval, and pre-process the scene depth data through a variety of depth filtering methods to remove noise; the PID control module is used to determine the target position of the hole to be drilled; the smoothing processing module is used to detect whether the drilling requirements are met.

[0018] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a servo positioning method based on 3D vision.

[0019] A computer-readable storage medium stores a computer program, which implements the steps of a servo positioning method based on 3D vision when executed by a processor.

[0020] Beneficial effects of the present invention: The servo positioning method based on 3D vision provided by the present invention uses Gaussian depth filter and surrounding neighborhood depth value to fill missing points, thereby reducing the interference of noise on depth information and ensuring the validity and stability of collected data. By refining and stratifying the depth information and combining the preliminary depth change interval, hierarchical processing of the scene is realized, the use efficiency of the scene data is optimized, and the complexity of subsequent search and positioning is reduced by prior information. Finally, the effect of improving positioning efficiency and shortening calculation time is achieved. The edge detection algorithm is used to locate the area of ​​specific shape of the depth data at different levels, and iterative search is performed in each level until the target area is found. This method effectively reduces the size of the search area and improves the accuracy and speed of coarse positioning. Through fine positioning after fitting, the influence of mechanical error on positioning accuracy is reduced. Finally, the effect of improving drilling accuracy is achieved. The servo control system continuously performs positioning, control and detection according to the relative position of the target point and the drill bit to ensure that the drill bit finally reaches the preset drilling position, so that the drill arm can gradually optimize the path and posture during the execution process, ensure the accuracy of the drill bit position and angle, improve the stability of the drilling process, and ensure that the system can cope with complex field environments. The present invention achieves better results in drilling accuracy, drilling efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 An overall flow chart of a 3D vision-based servo positioning method provided for the first embodiment of the present invention.

[0023] Figure 2 An algorithm flow chart of a 3D vision-based servo positioning method provided for the first embodiment of the present invention.

[0024] Figure 3 A functional schematic diagram of the main algorithm steps of a 3D vision-based servo positioning method provided in the first embodiment of the present invention.

[0025] Figure 4 An original depth map of a 3D vision-based servo positioning method provided by the first embodiment of the present invention.

[0026] Figure 5 A depth self-interval diagram of a 3D vision-based servo positioning method provided in the first embodiment of the present invention.

[0027] Figure 6 A diagram showing successful drilling positioning of a 3D vision-based servo positioning method provided in the first embodiment of the present invention.

[0028] Figure 7 An overall flow chart of a 3D vision-based servo positioning system provided for the third embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0030] Example 1, reference Figure 1-Figure 6 , as an embodiment of the present invention, provides a servo positioning method based on 3D vision, comprising:

[0031] S1: Preset the relative posture of the RGBD depth camera and the drill arm, and use the RGBD depth camera to obtain the 3D visual information of the drilling working face scene.

[0032] Furthermore, the acquisition of working face scene information mainly relies on the RGBD depth camera. First, the relative position of the RGBD depth camera and the drill arm is preset so that the camera and the drill bit can be moved and selected synchronously, while ensuring that the drill bit is in the same position in the camera coordinate system and maintains the same posture. Within the measurable range of the scene depth at a certain distance between the camera and the area to be drilled in the working face, the 3D visual information of the scene in the area to be drilled is obtained through the depth camera at an appropriate position and angle. The collected data includes scene depth information and RGB information. Since the operating environment of the drill arm is relatively complex and its lighting conditions are relatively limited, the reliability of RGB image information is low, while the depth information is not affected by lighting conditions. Therefore, the depth information is mainly used to implement subsequent functions. At this time, the acquired data contains a lot of noise information, which affects its reliability.

[0033] It should be noted that the depth distribution of the working surface to be drilled is uneven, and the scene lighting conditions are relatively limited. The depth camera uses realsenseD435, fixes the camera to the appropriate position of the drill arm and adjusts the posture of the drill arm to ensure that the drilling target position and the drill bit can be observed at the same time. Initially, the drilling rig is manually moved within the range where the D435 depth camera can measure the depth, and the RGB sensor and depth sensor of the D435 are called at the same time to obtain the 3D visual information in the current scene. At this time, the RGB data information obtained cannot fully observe the information of each visible area in the scene, so the reliability is low, and the scene depth information obtained by the depth sensor can fully observe the depth characteristics and changes of each part of the scene. Two coordinate systems are involved in the implementation, the camera coordinate system and the world coordinate system. Initially, the initial camera coordinate system will be set to the world coordinate system.

[0034] Furthermore, based on the position and angle of the camera when acquiring data, the approximate range of depth change in the area to be drilled is calculated, and the scene depth of the area to be drilled on the working surface that has been collected is processed by a variety of methods based on the preliminary change range to remove noise caused by various reasons in the collected data, thereby achieving the purpose of reducing the impact of noise and improving the reliability of the data used.

[0035] The data preprocessing process is as follows:

[0036] For the acquired scene depth data, a Gaussian depth filter is used to process it. A two-dimensional 3*3 Gaussian kernel is used to smooth each point of the depth image. It should be noted that the acquired scene depth data will have certain depth missing points. When there are depth missing points around a certain pixel point being processed, before depth filtering, the depth of the missing point is filled with the mean of the depth values ​​of the points in the surrounding 3*3 neighborhood that are not missing, and then subsequent calculations are performed. The smoothed image will reduce the points with abnormal depth values, and at the same time, the missing depth value points will be filled. Then, the depth change interval of the current scene is counted, and a preliminary change interval [d min ,d max ].

[0037] S2: Layering the drilling work surface scene into multiple sub-scenes, locating the target point to be drilled in the sub-scenes; screening out the area with obvious depth change in the sub-scenes, fitting the target area to be drilled in the area with obvious depth change using a curve fitting method, and locating the target point to be drilled in combination with preset information.

[0038] Furthermore, the pre-processed data will have a preliminary depth change interval. Based on this depth change interval, the scene is further layered. The layering is based on the change in scene depth. The scene information is divided into multiple more detailed depth change intervals according to a certain depth change range. The depth change intervals of each level are allowed to cross and the size of the change intervals is not necessarily the same. After layering, multiple depth scenes of different levels can be obtained. In the non-initial stage, the depth level prior information used in the previous successful positioning can be used to reduce the computational overhead of subsequent search and positioning.

[0039] Based on the known initial scene depth change interval, we assume that the maximum depth change around the positioning target area is s, then we can convert the large interval [d min ,d max ], according to the size interval v, it is divided into the following: {[d min ,d min +s],[d0,d0+s],…,[d n ,d n +s],[d max -s,d max ]}, where {d min <d0<…<d n +s <d max}, ensure that each small interval contains the initial depth change interval, so that multiple subdivided small intervals can be obtained according to the fixed interval length. It should be noted that elements are allowed to cross between small intervals.

[0040] It should be noted that for the more refined change intervals after layering, first isolate the scene depth area within the range of the change interval, generate a sub-scene depth map. In this sub-scene depth map, according to the change range of the depth change interval at this level, set the detection threshold of the depth gradient, and combine with the edge detection algorithm to calculate all the edge information in this level. Among these edge information, search for the depth area with a specific shape. If the current area is not found in one level, hierarchical traversal will be performed in a certain way, and search for the current area in each level until the target area is found.

[0041] For the depth change interval [d lower , d upper that has been subdivided, sort it according to the upper limit value d upper of the interval, and traverse from the interval with a larger d upper value to the interval with a smaller d upper value. Each depth interval corresponds to one level. Then use the canny edge detection algorithm. The specific implementation steps are as follows: first perform Gaussian blur to reduce the influence of noise in the image, which has been done in the initial preprocessing stage; then apply the sobel operator to calculate the depth gradient and gradient change direction of each pixel value. The sobel operator also has good anti-noise ability; then perform non-maximum suppression (NMS) to retain the local maximum values in the image and suppress other values to eliminate most non-edge values; there are still some false boundary points in the image at this time, so finally perform double-threshold detection. According to the depth change situation of the target positioning area, set two thresholds t1 and t2 (t1 < t2). The points detected by the smaller threshold t1 are called weak boundary points, and the points detected by the larger threshold t2 are called strong boundary points. Connect the weak boundary points existing between the strong boundary points and adjacent strong boundary points to obtain the final boundary information we want. Then perform the detection of a specific shape. In this embodiment, the detected shape is a quasi-circular shape. Combining the geometric features of a circle, the preliminary position of the quasi-circular shape to be searched can be located, and the target positioning point to be searched is inside the circle-like shape.

[0042] Furthermore, in the target area with this specific shape, separate this area, obtain all the points included in the edge of the specific shape in this small area, use the quadratic residual as the energy function to fit this specific shape, and according to the fitted equation, locate a specific position inside this specific shape area, that is, find the target point to be drilled.

[0043] Separate the preliminarily found quasi-circular area, and then fit this quasi-circular shape, applying the quadratic residual as the energy function where x0 and r represent the center of the circle and the radius respectively, and x iRepresents the boundary point. The energy function mainly penalizes the distance between the boundary point in the area and the circle boundary. Then the Gauss-Newton method is used to solve the energy function. After n iterations, the equation converges and the circle and radius value of the fitted circle are obtained, that is, the equation of the circle is determined. If it is assumed that the positioning target point is at the center of this type of circle, then the target point to be drilled can be directly positioned through the fitted equation.

[0044] At the same time, the above method is also used to locate the points to be drilled in the RGB image information of the same sub-scene. At this time, the target points to be drilled are located in the depth vision information and the RGB vision information respectively. The position coordinates of the two target points to be drilled in space are weighted fused, and the two types of information are combined to improve the accuracy and robustness of the algorithm.

[0045] S3: Locate the hole to be drilled in real time and use a servo strategy to control the drill head to move toward the target positioning point.

[0046] Furthermore, the relative position and posture between the target positioning point and the drill bit is calculated, and the drill arm is controlled to move according to the position of the drilling target point in space. After moving an appropriate distance, the depth information of the scene is obtained again, the target position to be drilled is relocated, and the relative position and posture between the target position to be drilled and the drill bit are calculated. It is detected whether the distance and relative position between the drill head and the drilling target position meet the following requirements: the distance is less than a preset threshold and the drill head is perpendicular to the drilling position. If so, the positioning is successful. Otherwise, the drill head is controlled to move an appropriate distance toward the target position again, and the positioning, control and inspection process is repeated until the relative position relationship between the drill head and the target position to be drilled meets the preset drillable conditions.

[0047] When the drilling target position has been located, the relative position between the positioning target point and the drill bit is calculated, and the relative position information is transmitted to the servo control system. The servo control system will plan a suitable path and control the drill arm to move. The target of each movement is the distance from the drilling target point to the drill bit. When the drill arm moves to the preset position, the relative position relationship between the drilling target point and the drill bit is recalculated to determine whether the distance between them is less than the set threshold and the drill head is vertically facing the target point. If these two conditions are met, the positioning function is realized. If the requirements are not met, since the positions of the drill arm and the camera have changed compared to before the movement, and the control and movement of the drill arm will introduce some mechanical errors into the algorithm, it is necessary to re-detect the environment, re-position the target point, and calculate the new and more accurate relative position relationship between the drilling target point and the drill bit. The three steps of drilling target point positioning, controlling the movement of the drill arm, and checking whether the drill bit reaches the target drilling position are iteratively performed until the drill bit reaches the target drilling position. It should be noted that it takes relatively more time to locate the drilling target position for the first time. Subsequent positioning can reduce the search range of the depth level based on the priori of the previous successful positioning, combine the equation of the previous fitted circle as the initial circle, and then iterate and solve on this basis, which can greatly reduce the time cost and ensure a faster positioning speed.

[0048] Depth map Figure 5 As shown, the pure black part represents the infinity and depth loss area, and the other areas are based on different depths.

[0049] The core task of drilling hole positioning is to find the drilling hole in the depth map. In order to improve the efficiency of positioning, the proposed method is:

[0050] First, the scene is layered according to different depth intervals. In the layered small intervals, the depth information that needs to be processed will become less, which is more conducive to processing. If the depth information of the entire working surface scene is processed every time, the efficiency will be slower. The initial layering may take more time, but after the first successful positioning, the second depth scene layering will have prior information to provide, which will accelerate the subsequent positioning process.

[0051] During the positioning process, the depth variation around the area to be drilled is usually inside an irregular geometric shape, which may be closed or not. Therefore, it is necessary to locate the special geometric area first. In the example provided, the edge detection algorithm is used to achieve this. After successful positioning, Figure 6 shown.

[0052] The application of servo control strategy can better reduce the impact of various interference factors. For example, when it is initially detected that the drill bit and the borehole are relatively vertical and the distance is 20cm, the relative position of the drill bit and the borehole at this time is transmitted to the control center. The control center controls the drill arm and the drill bit to move 5cm toward the drilling position based on the servo strategy. However, due to interference from various factors, when the relative position between the drill bit and the borehole is detected again, it may be found that the distance between the drill bit and the borehole is 14cm instead of 15cm. At this time, it is necessary to promptly feedback information to the control center and adjust the control instructions.

[0053] Embodiment 2, an embodiment of the present invention, provides a servo positioning method based on 3D vision. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0054] First, in order to verify the actual effect of the servo positioning method based on 3D vision, an RGBD depth camera is used to collect 3D visual data of the drilling scene on the working face, and the detailed implementation is carried out according to the steps. The experiment uses a Realsense D435 depth camera, which is fixed at the appropriate position of the drill arm to ensure that the depth and RGB image data of the drilling scene are collected when the camera coordinate system and the drill bit move synchronously. Due to the complex lighting environment of the working face, it is necessary to integrate the depth information and RGB information for processing.

[0055] The experiment first obtains the preliminary depth data of the area to be drilled on the working surface, and pre-processes the data using the Gaussian filtering method according to the depth change interval to remove noise. The depth change interval is further divided into different depth levels through the calculated depth change interval, and the edge information of each level is extracted through the edge detection algorithm. In different levels, the depth area that meets the target shape is found, and the area is fitted by the fitting equation using 3D visual information analysis, and finally the target position to be drilled is located. The servo system then moves the drill bit according to the calculated positional relationship between the target point and the drill bit until the drilling requirements are met. Each experiment is carried out in different depth ranges to verify the stability and applicability of the servo positioning method.

[0056] Table 1 Experimental data table

[0057]

[0058] It can be seen from the experimental data that by using an RGBD depth camera to obtain 3D visual data of the working surface scene, and then through a multi-level layering and edge detection algorithm, the experimental results show the significant advantages of the invention. In the noise elimination parameters, the data shows that the average noise elimination rate reached about 86%, proving the effectiveness of the depth filter in complex scenes on the working surface. In addition, in terms of layering accuracy, the experimental results are stable at more than 90%, reflecting that the positioning accuracy can be effectively improved by layering and refining the depth change interval.

[0059] In the fitting error, the experimental error was controlled between 0.3-0.6 mm, which is much lower than the average error range of the existing technology, showing the high accuracy of the fitting algorithm in different scenarios. The final positioning accuracy reached more than 95%, further proving the efficiency and accuracy of the servo system in iterative correction.

[0060] By comparing with the existing technology, the traditional positioning method often encounters problems such as large noise interference and unstable positioning accuracy when dealing with complex working surface scenes. The present invention effectively solves these problems by combining deep stratification and edge detection, and demonstrates excellent stability and accuracy in experiments. Therefore, the present invention improves the efficiency of drilling arm positioning in complex environments while reducing dependence on human intervention, and has significant innovation and application prospects.

[0061] Example 3, reference Figure 7 , as an embodiment of the present invention, provides a servo positioning system based on 3D vision, including a 3D vision information acquisition and depth data processing module, an edge detection and target point fitting module, and a servo control and drill positioning module.

[0062] The path planning module is used to use the RGBD depth camera to obtain 3D visual information of the working face drilling scene, output the depth change range, and pre-process the scene depth data through a variety of depth filtering methods to remove noise. The PID control module is used to determine the target position of the hole to be drilled, and the smoothing processing module is used to detect whether the drilling requirements are met.

[0063] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0065] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0066] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0067] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A servo positioning method based on 3D vision, characterized in that: include: The relative posture of the RGBD depth camera and the drill arm is preset, and the RGBD depth camera is used to obtain the 3D visual information of the drilling working face scene; The drilling work surface scene is divided into sub-scenes, and the target points to be drilled are located in the sub-scenes; Screen the area, perform 3D visual information analysis and use curve fitting method to fit the target area to be drilled, and locate the target point to be drilled based on preset information; Locate the hole to be drilled in real time and use servo strategy to control the drill head to move to the target positioning point.

2. The 3D vision-based servo positioning method according to claim 1, characterized in that: The relative position and posture of the preset RGBD depth camera and the drill arm, and the 3D visual information of the drilling working face scene obtained by using the RGBD depth camera include the relative position and posture of the preset RGBD depth camera and the drill arm, so that the camera and the drill bit move synchronously, ensuring that the position and posture of the drill bit in the camera coordinate system remain unchanged; The 3D visual information of the drilling working face scene is obtained by using an RGBD depth camera, including the RGB information and depth information of the scene.

3. The 3D vision-based servo positioning method according to claim 2, characterized in that: The step of stratifying the drilling work surface scene into sub-scenes and locating the target point to be drilled in the sub-scenes includes dividing the drilling work surface scene into a plurality of sub-scenes using a depth segmentation technology.

4. The 3D vision-based servo positioning method according to claim 3, characterized in that: The 3D visual information analysis includes locating the target point to be drilled in the RGB visual information and the depth visual information at the same time, and fusing the two positioning results to achieve accurate and robust positioning of the target point to be drilled.

5. The 3D vision-based servo positioning method according to claim 4, characterized in that: The screening area uses a curve fitting method to fit the target area to be drilled, and locates the target point to be drilled in combination with preset information, including screening the area as an area with a depth change greater than a threshold, denoising the sub-scene information, calculating the image gradient and the depth gradient, and applying non-maximum suppression and dual threshold detection strategies.

6. The 3D vision-based servo positioning method according to claim 5, characterized in that: The real-time positioning of the position to be drilled and the use of a servo strategy to control the drill bit to move toward the target positioning point includes, when it is not the initial positioning, using the prior information of the last successful positioning to accelerate the speed and efficiency of the current positioning; The prior information is the spatial coordinates of the camera and the drill in the world coordinate system, the target point to be drilled, the sub-scene in which it is located, and the curve equation of the specific shape geometric area.

7. The 3D vision-based servo positioning method according to claim 6, characterized in that: The real-time positioning of the position to be drilled and the use of a servo strategy to control the drill bit to move toward the target positioning point includes applying a servo strategy to control the movement of the drill arm, and repeating the positioning, control and inspection processes until the relative position relationship between the drill bit and the target position to be drilled meets the preset drilling conditions.

8. A system using the 3D vision-based servo positioning method according to any one of claims 1 to 7, characterized in that: Including 3D visual information acquisition and depth data processing module, edge detection and target point fitting module, servo control and drill positioning module; The path planning module is used to obtain 3D visual information of the working face drilling scene using an RGBD depth camera, output a depth change interval, and pre-process the scene depth data through a variety of depth filtering methods to remove noise; The PID control module is used to determine the target position of the hole to be drilled; The smoothing processing module is used to detect whether the drilling requirements are met.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the 3D vision-based servo positioning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the 3D vision-based servo positioning method according to any one of claims 1 to 7 are implemented.

Images