A method and system for robotic bolting based on pairs of visual information

Abstract

The application provides a robot support method and system based on a visual information pair, relates to the technical field of computer vision, and is applied to a robot device; the method comprises the following steps: obtaining stereo image data of a target area to be collected; determining time information of a current collection time of the stereo image data, and storing the stereo image data and the corresponding time information to obtain a visual information pair associated with time and space; solving the visual information pair to obtain visual space-time information; converting the visual space-time information into an action instruction used for instructing an execution module to execute a corresponding support task action, and updating and dynamically correcting the action instruction in real time according to the visual space-time information; and guiding corresponding working components to a target work point through the execution module according to the action instruction, and executing a support task action. The application can provide an efficient and accurate support operation process in a complex coal mine roadway environment, reduces manual intervention, and improves construction safety and operation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a robot supporting method and system based on a pair of visual information. BACKGROUND

[0002] The development of intelligent coal mines is an inevitable trend for safe and efficient exploitation of coal resources. In particular, in the coal mine roadway supporting operation, the anchor rod (cable) drilling construction and installation work as an important process of roadway excavation occupies a large amount of manpower and time. Due to the high degree of dependence on manpower in this link, and the high risk in construction, failure to complete the support in time or poor support effect will lead to roof fall, rib spalling and other safety hazards, which seriously restricts the rapid excavation and safe production of coal mines.

[0003] Related technologies based on vision and other sensing means for operation assistance generally do not use information pairs to analyze and reconstruct four-dimensional (x, y, z, t) space-time relationships. Although some methods use stereo vision technology to capture images at different angles to obtain three-dimensional information of objects, such as binocular distance measurement or holographic projection. These technologies can recover the scene depth information from two or more perspective images, and measure the distance between objects without direct contact, but these methods do not fully exploit the full potential of different perspective information, and the perspective is parallel to the vision, which cannot focus on key areas. SUMMARY

[0004] The embodiment of the present application provides a robot supporting method and system based on a pair of visual information, which aims to solve the problems in the above background technology.

[0005] In order to solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiment of the present application provides a robot supporting method based on a pair of visual information, applied to a robot device, the robot device comprising at least one execution module; the method comprises:

[0007] Obtaining stereo image data of a target area to be collected, the stereo image data comprising raster format data and vector format data, the stereo image data being a pair of binocular focusing data, the target area comprising a target operation point;

[0008] Determining time information of the current collection time of the stereo image data, and storing the stereo image data and the corresponding time information to obtain a pair of visual information associated with space-time;

[0009] Solving the pair of visual information to obtain visual space-time information, the visual space-time information being used to represent the time and space relationship between the execution module and the target operation point;

[0010] The visual spatio-temporal information is converted into an action instruction for instructing the execution module to perform a corresponding supporting task action, and the action instruction is updated and dynamically corrected in real time according to the visual spatio-temporal information;

[0011] According to the action instruction, a corresponding work component is guided to the target work point by the execution module, and the supporting task action is performed, and the work component is an operation terminal of the execution module.

[0012] Optionally, the target work point includes a positioning hole site; the visual information is calculated to obtain visual spatio-temporal information, including:

[0013] By means of image processing technology, key visual features in the stereoscopic image data are extracted, and the key visual features include texture features, shape features and edge features of the positioning hole site and the work component;

[0014] Based on binocular parallax of the stereoscopic image data, depth information is calculated;

[0015] In combination with the depth information, spatial position information and time information of the target work point and the work component are three-dimensionally positioned to obtain the visual spatio-temporal information.

[0016] Optionally, the key visual features in the stereoscopic image data are extracted by means of image processing technology, including:

[0017] The stereoscopic image data is preprocessed, and the preprocessing includes denoising processing, contrast enhancement processing and image registration processing;

[0018] By means of texture analysis algorithm, texture features of the positioning hole site and texture features of the work component are extracted;

[0019] By means of shape recognition algorithm, shape features of the stereoscopic image data and shape features of the work component are extracted;

[0020] By means of edge detection algorithm, edge features of the target work point and the work component are extracted;

[0021] The texture features of the positioning hole site, the texture features of the work component, the shape features of the stereoscopic image data, the shape features of the work component, the edge features of the target work point and the edge features of the work component are determined as the key visual features in the stereoscopic image data.

[0022] Optionally, the depth information is calculated based on binocular parallax of the stereoscopic image data, including:

[0023] In the visual information pair, disparity information of each pixel point is calculated;

[0024] According to the disparity information of each pixel point, the depth information of each pixel point is calculated.

[0025] Optionally, the depth information is combined with the key visual feature to perform three-dimensional spatial positioning on the target work point, and the visual space-time information is obtained, including:

[0026] The key visual feature in the stereoscopic image data is vectorized to generate a multi-dimensional feature vector of the key visual feature, and the multi-dimensional feature vector at least includes a texture feature vector, a shape feature vector, and an edge feature vector;

[0027] The multi-dimensional feature vector of the key visual feature is mapped with a preset drilling template data to generate feature encoding information, and the drilling template data includes depth information, texture features, shape features, and edge features of a standard drilling hole;

[0028] The multi-dimensional feature vector is compared and matched with the drilling template data;

[0029] According to the comparison and matching result, the spatial position of the positioning hole is determined;

[0030] According to the spatial position of the positioning hole, visual space-time information representing the spatial relationship between the execution module and the positioning hole is generated.

[0031] Optionally, the visual space-time information is converted into an action instruction for instructing the execution module to perform a corresponding support task action, and the action instruction is updated and dynamically corrected in real time according to the visual space-time information, including:

[0032] Based on the visual space-time information, the spatial relationship between the execution module and the target work point is solved in real time;

[0033] According to the spatial relationship between the execution module and the target work point, the action instruction is generated, and the action instruction is updated and dynamically corrected in real time.

[0034] Optionally, the action instruction is updated and dynamically corrected in real time, including:

[0035] The adaptive control algorithm is used to update and dynamically correct the action instruction in real time, and the adaptive control algorithm includes Kalman filtering, PID control algorithm, H∞ control algorithm, and neural network control algorithm.

[0036] Optionally, the stereoscopic image data of the target region to be collected is obtained, including:

[0037] In the tunneling roadway or the roadway scene to be reinforced, a pair of visual acquisition devices are used to acquire a pair of binocular focus data in the drilling direction of the roadway scene; the size of the field of view overlapping area acquired by the two visual acquisition devices is not less than a preset area size.

[0038] The pair of binocular focus data in the drilling direction of the roadway scene is determined as the stereoscopic image data of the target region.

[0039] In a second aspect, an embodiment of the present application provides a robot support system based on a pair of visual information, which is applied to the steps of the robot support method based on a pair of visual information as described in the first aspect, and the system comprises a robot device, a data acquisition module, a data storage module, a data processing module and a control module, wherein the robot device at least comprises an execution module.

[0040] The data acquisition module is configured to acquire stereoscopic image data of a target region to be acquired, wherein the stereoscopic image data comprises raster format data and vector format data, the stereoscopic image data is a pair of binocular focus data, and the target region comprises a target work point.

[0041] The data storage module is configured to determine time information of a current acquisition time of the stereoscopic image data, and store the stereoscopic image data and the corresponding time information to obtain a pair of visual information associated with time and space.

[0042] The data processing module is configured to solve the pair of visual information to obtain visual space-time information, wherein the visual space-time information is used to represent a spatial relationship between the execution module and the target work point.

[0043] The control module is configured to convert the visual space-time information into an action instruction used to instruct the execution module to perform a corresponding support task action, and to update and dynamically correct the action instruction in real time according to the visual space-time information.

[0044] The execution module is configured to guide a corresponding work component to the target work point by the execution module according to the action instruction, and to perform the support task action, wherein the work component is an operation terminal of the execution module.

[0045] Optionally, the robot device is any one of the following: a multi-axis degree of freedom mechanical arm, a humanoid robot, a quadruped robot, a wheeled robot or a tracked robot.

[0046] The technical scheme provided by the embodiment of the present application at least brings the following beneficial effects:

[0047] The application can accurately obtain the visual space-time information of the target area by combining the stereoscopic image data with the time information, effectively realize the accurate representation of the spatial relationship between the robot device and the target work point, and enable the robot to dynamically adjust the action instruction according to the real-time updated visual space-time information when performing the support task, thereby ensuring the high-precision execution of the support task. Through the depth calculation and dynamic correction of the visual information, the application improves the adaptability of the robot to the complex environment, reduces the dependence on manual intervention, and improves the safety, automation degree and work efficiency of the coal mine operation. In addition, the visual information associated with space-time can fully capture the spatial position of the target work point, ensure the accurate guidance of the work component to the target position, and realize the precise support operation. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0049] Figure 1 is a step schematic diagram of a robot support method based on a pair of visual information provided by an embodiment of the present application;

[0050] Figure 2 is an application scenario schematic diagram of a robot support system based on a pair of visual information provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in this paper only represents the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0052] With the continuous development of intelligent technology in coal mines, the automation and safety of underground operations in coal mines have gradually become the key to improving production efficiency and ensuring safety. However, traditional coal mine roadway support operations rely on manual operation, especially in the process of anchor (cable) drilling and installation, which is not only laborious but also has a high risk coefficient. Related support operation methods mainly rely on two-dimensional images for target recognition and prediction, but due to the complexity of the environment and limited perspective, two-dimensional images cannot effectively reflect the spatial position and shape of the target object, especially in complex scenarios, with large appearance differences and missing depth information, resulting in insufficient accuracy of perception, which in turn affects the efficiency and safety of the operation. The core idea of the present application is to accurately solve the problem by combining the stereo information of the focused vision based on the stereo image data and the spatiotemporal information of the visual information pair, thereby realizing the intelligentization and automation of the coal mine roadway support operation. By obtaining the stereo image data of the target region and associating it with the time information, the spatial relationship between the robot execution module and the target operation point is represented using visual spatiotemporal information, and the action instructions are dynamically updated, which can ensure the accurate execution of the support task, significantly improve the efficiency and safety of the operation, and solve the technical defects in the traditional method.

[0053] Figure 1 is a step schematic diagram of a robot support method based on a visual information pair provided by an embodiment of the present application, as Figure 1 shown, the method is applied to a robot device, and the robot device at least includes an execution module. In an optional embodiment, the robot device is any of the following: a multi-axis freedom mechanical arm, a humanoid robot, a quadruped robot, a wheeled robot, or a tracked robot.

[0054] The robot device described in this embodiment refers to a physical entity that performs support tasks, and its types include but are not limited to: multi-axis freedom mechanical arm (such as industrial robot arm), mobile platform robot (such as quadruped, wheeled or tracked robot), humanoid robot (with complex action ability), and the robot device has the ability to move, position and operate in the coal mine roadway.

[0055] The execution module is a component in the robot device responsible for physical actions and is the direct execution terminal of the operation task. The execution module is used to guide the work components and perform support actions. Specifically, the execution module accurately moves tools such as drill bits and anchor rods to target operation points (such as anchor drilling hole positions) according to control instructions, completes specific procedures such as drilling, drill rod lengthening, cartridge installation, and anchor rod fixation. The work components include but are not limited to drill bits, drill rods, cartridges, and anchor rods (cables) according to different work scenarios and task conditions.

[0056] Single execution module (such as single robot arm) completes all actions, and is suitable for simple scenes. The embodiment also supports cooperation of multiple execution modules (such as robot arm + mobile chassis), and is suitable for complex tasks (such as drilling and installing anchor rod at the same time).

[0057] The method comprises the following steps: S101, acquiring stereo image data of a target region to be collected, wherein the stereo image data comprises raster format data and vector format data, the stereo image data is paired binocular focus data, and the target region comprises a target work point.

[0058] Image acquisition is performed on a target region or object to be collected by using an imaging device or a sensing device, and a focus position is controlled. Paired binocular focus data acquired is used as stereo image data. In the application, the stereo image data comprises a bimodal data fusion structure of vector format data and raster format data. The raster format data is in the form of pixel matrix image raw data, and the vector format data is in the form of geometric feature vectorization conversion.

[0059] The geometric feature vectorization conversion is generated.

[0060] The stereo image data is paired binocular focus data, the binocular focus data is video image or graphic data, paired visual acquisition devices focus in the same direction or the same general direction to form a stereo pair, and have certain parallax information, which is used for subsequent depth estimation and spatial positioning. The imaging device or the sensing device comprises but is not limited to a binocular vision camera, an infrared imaging system and a laser radar system. The target region refers to a specific region in which a robot device needs to work, and comprises a plurality of target work points. The target region comprises but is not limited to an anchor rod (cable) drilling position, water exploration and drainage, gas extraction, pressure relief drilling construction position, mesh, W steel belt and internal hole in a roadway roof or two sides in a support work and other drilling and construction processes.

[0061] It can be understood that the application is based on a cooperative perception technology of a visual information pair, and aims to build a complete technical closed loop from data acquisition to dynamic control. It is different from a traditional monocular vision or static binocular imaging method.

[0062] In an optional implementation, the step S101 specifically comprises the following steps S1011 and S1012:

[0063] In the step S1011, paired binocular focus data in a drilling direction in a tunneling roadway or a roadway scene to be reinforced is acquired by using paired visual acquisition devices; and the size of a field of view overlapping region acquired by the two paired visual acquisition devices is not less than a preset region size.

[0064] In the coal mine tunnel excavation or the roadway environment to be reinforced, a pair of visual acquisition devices (such as a binocular camera system, a stereo vision camera, etc.) are used to collect data of the roadway scene. The pair of visual acquisition devices are installed side by side on the robot device, and through focusing setting, it is ensured that the data can be collected from two perspectives synchronously to obtain binocular focusing data.

[0065] When collecting data, the size of the field of view overlapping area collected by the pair of visual acquisition devices is not less than the preset area size, that is, the field of view overlapping area collected by the pair of visual acquisition devices is large enough to effectively perform subsequent three-dimensional space reconstruction and positioning accuracy calculation. In this process, the pair of visual devices are controlled synchronously to make the collected binocular focusing data have high precision and high consistency. In this embodiment, the pair of visual acquisition devices are different from the traditional parallel vision binocular camera, simulate the visual line of the human eye with a certain visual angle, focus on a point, and can more accurately collect binocular focusing data.

[0066] Step S1012, the pair of binocular focusing data in the drilling direction in the roadway scene is determined as the stereoscopic image data of the target region.

[0067] The collected pair of binocular focusing data is determined as the stereoscopic image data of the target region, and is prepared for subsequent analysis and processing. The stereoscopic image data of the target region will be used as the visual input of the robot to perform the task, supporting the positioning and navigation of the robot in the target region. Through subsequent data processing and image analysis, the three-dimensional space coordinates of the target work point can be obtained, so as to accurately guide the robot device to perform the supporting work. Optionally, an inertial navigation system (INS) and a laser radar are introduced to cooperate with the calibration, the visual information pair (binocular focusing data) and the motion trail of the robot device are fused, and a four-dimensional space-time relationship model is constructed. The supporting path can be dynamically planned in the subsequent supporting task execution process, and the positioning error caused by rock creep can be avoided.

[0068] Step S102, time information of the current collection time of the stereoscopic image data is determined, and the stereoscopic image data and the corresponding time information are stored to obtain the space-time correlated visual information pair.

[0069] The obtained stereoscopic image data of the target region is stored for calculation of key spatio-temporal information such as three-dimensional spatial state and depth information. The acquisition time of each frame of collected stereoscopic image data is recorded. This is because the stereoscopic image data needs to contain not only spatial information but also time information to form a complete spatio-temporal data pair. Specifically, after the binocular image data of the target region is collected, the time information (such as a time stamp) of the current acquisition time is obtained at the same time, and the collected stereoscopic image data is associated with the corresponding time information and stored together.

[0070] In an optional embodiment, the target operation point includes a positioning hole site.

[0071] In the coal mine underground tunneling or supporting operation, the hole site (i.e., the positioning hole site) of the anchor rod drilling can be the target operation point of the supporting operation. In the case where the target operation point is the positioning hole site, the corresponding supporting task action is automatic drilling or automatic installation of the anchor rod.

[0072] Step S102 specifically includes steps S1021 to S1023:

[0073] Step S1021, by image processing technology, extracts key visual features in the stereoscopic image data, including texture features, shape features and edge features of the positioning hole site and the working component.

[0074] By image processing technology, the collected stereoscopic image data is processed to extract key visual features, including texture features, shape features and edge features of the positioning hole site and the working component. The image processing technology includes Canny edge detection algorithm, Gabor filter, color histogram or shape descriptor. The texture features are used to describe the texture details of the positioning hole site and the surface of the working component, to determine the attributes such as surface roughness, contrast, directionality, etc. The shape features are used to describe the geometric shape or contour features of the positioning hole site and the working component, to determine the attributes such as shape size, geometric structure, symmetry, etc. of the positioning hole site and the working component. The edge features are used to describe the contour or boundary of the positioning hole site and the working component in the image.

[0075] In an optional embodiment, step S1021 specifically comprises: pre-processing the stereoscopic image data, the pre-processing comprising denoising processing, contrast enhancement processing and image registration processing; extracting texture features of the positioning hole and texture features of the work component by using a texture analysis algorithm; extracting shape features of the stereoscopic image data and shape features of the work component by using a shape recognition algorithm; extracting edge features of the target work point and the work component by using an edge detection algorithm; and determining the texture features of the positioning hole, the texture features of the work component, the shape features of the stereoscopic image data, the shape features of the work component, the edge features of the target work point and the edge features of the work component as key visual features in the stereoscopic image data.

[0076] Before extracting the key visual features, the collected stereoscopic image data is pre-processed. The pre-processing steps include: denoising processing, contrast enhancement processing and image registration processing.

[0077] After the pre-processing is completed, texture features of the positioning hole and the work component are extracted by using a texture analysis algorithm. The gray level co-occurrence matrix (GLCM) can be used to analyze the roughness, contrast and directionality of the texture, the local binary pattern (LBP) can be used to describe the local pattern of the image texture, and the Gabor filter can be applied to extract the frequency information of the texture to identify different texture features. Further, shape features in the stereoscopic image data are extracted by using a shape recognition algorithm. The Hough transform can be used to detect the circular or linear shapes in the image to identify the shape of the hole, and the contour detection algorithm such as the boundary tracking method can be applied to extract the contour information of the hole. Further, edge features of the target work point and the work component are extracted by using an edge detection algorithm. The Canny edge detection algorithm can be used to identify the edges in the image, the Sobel operator can be applied for edge detection, the edge position can be determined by calculating the gradient of the image gray scale, and the Laplacian operator can be used for edge detection, the second derivative is used to highlight the edge region.

[0078] Finally, the extracted features are integrated to determine the key visual features in the stereoscopic image data. Specifically, the texture features of the positioning hole, the texture features of the work component, the shape features of the stereoscopic image data, the shape features of the work component, the edge features of the target work point and the edge features of the work component.

[0079] Step S1022, based on the binocular disparity of the stereoscopic image data, calculating the depth information.

[0080] In this embodiment, the depth information is calculated based on the binocular disparity of the collected stereo image data to obtain the spatial position and depth information of the target work point and related work components. The binocular disparity refers to the parallax phenomenon caused by the cameras of two cameras shooting the same scene from different positions. By analyzing the pixel difference between the left and right images, the depth information of the objects in the scene can be derived, simulating the visual mechanism of the human eye. The larger the parallax, the closer the object is to the camera; the smaller the parallax, the farther the object is from the camera.

[0081] In an optional embodiment, the binocular disparity based on the stereo image data is used to calculate the depth information, which includes calculating the disparity information of each pixel point in the pair of visual information, and calculating the depth information of each pixel point according to the disparity information of each pixel point. At the same time, the multi-dimensional scene grid information with depth information is further encoded. The encoding process uses feature extraction, multi-dimensional code generation, and redundancy check multi-dimensional code encoding technology to convert complex scene information into efficient and compressed encoding form, and stores these encodings in a database. By quickly encoding the current scene and matching with the stored multi-dimensional code information, the scene can be quickly understood and accurately identified.

[0082] In the stereo image data, each pair of images contains pixel data from two different perspectives. Optionally, the disparity value of each pixel point between the left and right images can be calculated by an image matching algorithm (such as block matching, SAD (Sum of Absolute Differences) method, etc.), that is, the disparity information of each pixel point. Specifically, the pixels of the left and right images are compared to find the matching of the corresponding pixel points. Further, by calculating the horizontal or vertical offset between the matching pixels, the disparity information of each pixel point is obtained, which is used to describe the difference between the two camera perspectives.

[0083] The depth information of each pixel point is calculated by known camera parameters (such as focal length, baseline distance, etc.). The calculation can be performed according to the following formula:

[0084]

[0085] In the formula, is the focal length of the camera, is the baseline between the two cameras (i.e. the physical distance between the cameras), is the parallax value.

[0086] Step S1023, combining the depth information, the spatial position information and the time information of the target work point and the work components are three-dimensionally positioned to obtain the visual space-time information.

[0087] By utilizing depth information obtained from stereoscopic image data, the spatial location and temporal information of the target work point and working components are comprehensively processed to achieve accurate three-dimensional spatial positioning. In an optional embodiment, combining the depth information with the key visual features to perform three-dimensional spatial positioning of the target work point and obtain the visual spatiotemporal information includes:

[0088] The key visual features in the stereo image data are vectorized to generate multidimensional feature vectors of the key visual features. The multidimensional feature vectors include at least texture feature vectors, shape feature vectors, and edge feature vectors.

[0089] Key visual features in the acquired stereo image data are vectorized. Image processing techniques are used to transform key visual features (such as texture, shape, and edges) into multidimensional feature vectors. These multidimensional feature vectors can effectively represent the visual information of the target work point in a multidimensional space. The generated multidimensional feature vectors include at least the following features: texture feature vectors, representing the surface texture information of the target work point; shape feature vectors, representing the geometric shape information of the target work point or workpiece; and edge feature vectors, reflecting the geometric information of the target area's edges, such as the outline of a hole.

[0090] The multidimensional feature vectors of the key visual features are mapped to the preset drilling template data to generate feature encoding information. The drilling template data includes the depth information, texture features, shape features and edge features of the standard drilling.

[0091] The obtained multidimensional feature vectors are mapped to preset borehole template data to generate feature encoding information. The borehole template data contains various information about standard boreholes, including depth information, texture features, shape features, and edge features. Specifically, through feature encoding technology, the key visual features actually collected are associated with the standard template data to ensure the accuracy and consistency of the target work points.

[0092] The multidimensional feature vector is compared and matched with the borehole template data.

[0093] Feature matching is performed by comparing the generated multidimensional feature vector with the preset feature information in the borehole template data. The aim is to find the optimal matching position between the target work point in the image data and the standard borehole template, ensuring accurate identification of the positioning hole.

[0094] Based on the comparison and matching results, the spatial position of the positioning hole is determined.

[0095] According to the matching result, the spatial position of the target work point, i.e., the positioning hole site, is determined. Through accurate feature matching, the coordinate information of the target hole site in the three-dimensional space is identified. The coordinate information is a comprehensive solution based on the depth-of-field data and the visual features.

[0096] According to the spatial position of the positioning hole site, visual spatiotemporal information is generated to represent the spatial relationship between the execution module and the positioning hole site.

[0097] Finally, according to the spatial position of the positioning hole site, visual spatiotemporal information is generated to describe the spatial relationship between the execution module and the target work point. The visual spatiotemporal information combines the spatial positioning of the target work point and the time information, providing accurate guidance for subsequent automated operations of the execution module.

[0098] In step S103, the visual information pair is solved to obtain visual spatiotemporal information, which is used to represent the spatial relationship between the execution module and the target work point.

[0099] The spatiotemporal correlation visual information pair stored in step S102 is data fused to combine the time information and the spatial information, and a spatiotemporal data set is constructed. The spatiotemporal data set includes key visual features of stereoscopic image data, depth-of-field information, three-dimensional coordinates of the target work point and the working component, and corresponding acquisition time stamps. Through spatiotemporal data fusion, a dynamic spatiotemporal feature correlation model is established to reflect the relative position changes of the target work point and the execution module in the three-dimensional space in real time.

[0100] Distance, object contour, and spatial distribution relationship, which are related to depth-of-field information, are extracted from the visual spatiotemporal information using depth-of-field feature extraction technology, which includes stereoscopic ranging technology.

[0101] Based on the fused spatiotemporal data set and depth-of-field feature data, a dynamic spatial relationship model is established. The dynamic spatial relationship model is solved through the following steps:

[0102] First, the three-dimensional coordinates of the target work point and the execution module are unified to the reference coordinate system of the robot device. Coordinate system alignment is performed using rigid body transformation matrices (such as rotation matrices and translation vectors) to ensure the consistency of the spatial relationship solution. According to the real-time pose (including position and attitude angle) of the execution module and the three-dimensional coordinates of the target work point, the relative pose relationship between the two is calculated. The relative pose relationship includes the Euclidean distance between the execution module and the target work point, the directional deviation between the execution module motion trajectory and the target work point, and the included angle between the execution module attitude angle and the target work point normal vector.

[0103] Further, the relative pose relationship and the time information are combined to generate dynamic encoded visual spatio-temporal information. The visual spatio-temporal information is stored in the form of a multi-dimensional vector, including spatial coordinates, time stamps, pose deviation amounts, and motion trend prediction values.

[0104] Finally, visual spatio-temporal information is obtained, which includes real-time relative position coordinates of the execution module and the target work point, motion path planning of the execution module, dynamic correction parameters (such as speed adjustment amount and attitude correction angle), and spatial state description of the target work point (such as hole depth and hole diameter).

[0105] In step S104, the visual spatio-temporal information is converted into an action instruction for instructing the execution module to perform a corresponding support task action, and the action instruction is updated and dynamically corrected in real time according to the visual spatio-temporal information.

[0106] The visual spatio-temporal information represents the real-time spatial position relationship between the execution module and the target work point. According to the visual spatio-temporal information, a corresponding support task action instruction is generated to instruct the execution module to perform tasks such as drilling and support installation. In order to ensure the accuracy of the work, the generated action instruction is updated and dynamically corrected in real time according to the visual spatio-temporal information. The real-time correction mechanism provided in the embodiment can timely adjust the action instruction according to the detected deviation or environmental change in the execution process, thereby optimizing the execution process and ensuring the accuracy and safety of the work. For example, if the target work point position deviates, the execution module can accurately complete the support task by dynamically adjusting the instruction.

[0107] In an optional embodiment, step S104 specifically includes steps S1041 to S1042:

[0108] In step S1041, the spatial relationship between the execution module and the target work point is solved in real time based on the visual spatio-temporal information.

[0109] The real-time spatial relationship between the execution module and the target work point is accurately solved based on the visual spatio-temporal information. Specifically, the visual spatio-temporal information includes the three-dimensional coordinates of the target work point, the real-time pose (including position and attitude angle) of the execution module, and the dynamic relative relationship between the two. To realize real-time solving of the spatial relationship, first, the three-dimensional coordinates of the target work point and the real-time pose of the execution module are unified to the reference coordinate system of the robot device. Through the rigid body transformation matrix (including the rotation matrix and the translation vector), the target work point coordinates in the visual spatio-temporal information are converted to the local coordinate system of the execution module. For example, if the target work point is located at the drilling position of the roadway roof, the three-dimensional coordinates of the drilling need to be mapped to the operation space of the end effector of the mechanical arm in combination with the current position and attitude of the execution module in the roadway.

[0110] Secondly, based on the converted coordinate system, the real-time relative pose relationship between the end effector (such as a drill bit or an anchor rod clamp) of the execution module and the target work point is calculated. The relative pose relationship includes the following parameters: the Euclidean distance (the straight-line distance between the execution module and the target work point), the directional deviation (the directional angle between the motion trajectory of the execution module and the target work point), and the attitude deviation (the angle between the attitude angle of the execution module and the normal vector of the target work point).

[0111] In addition, in combination with the timestamp information, the dynamic change trend of the execution module and the target work point is analyzed. For example, when the chassis of the mobile robot device adjusts the position, the relative motion trajectory of the target work point needs to be predicted to avoid the positioning error caused by the movement of the robot device.

[0112] Step S1042, according to the spatial relationship between the execution module and the target work point, the action instruction is generated, and the action instruction is updated and dynamically corrected in real time.

[0113] Based on the calculated real-time spatial relationship, the control module generates specific action instructions to guide the execution module to complete the support task. The action instruction includes the movement path, the speed parameter, the attitude adjustment amount, and the operation time sequence (such as drill hole starting, drill rod lengthening, anchor rod installation, etc.) of the end effector.

[0114] According to the spatial position of the target work point, the optimal motion path of the execution module is planned. For example, if the drill bit needs to be guided to the center of the drill hole, a straight-line interpolation trajectory is generated based on the Euclidean distance and the directional deviation, and the initial motion speed is set. During the action execution process, the actual relative pose of the execution module and the target work point is continuously monitored through the visual space-time information.

[0115] In an optional embodiment, the real-time updating and dynamic correction of the action instruction includes: using an adaptive control algorithm to update and dynamically correct the action instruction in real time, and the adaptive control algorithm includes: Kalman filtering, PID control algorithm, H∞ control algorithm, and neural network control algorithm.

[0116] It should be noted that the adaptive control algorithms used in the embodiments of the present application can be combined with each other. For example, in the case of a support task action of drilling operation, when it is detected that the actual position of the drill bit deviates from the target hole position by 2 mm, the PID control algorithm is used to calculate that each joint needs to be additionally rotated by 0.5°, the H∞ control algorithm is used to suppress the shaking caused by mechanical vibration, and the neural network control algorithm is used to optimize the correction amount according to the historical data. Optionally, the steps of detection, correction, and execution are cyclically executed until the error is lower than the threshold.

[0117] The modified action instruction is issued to the execution module through the control module to drive the mechanical joint or the moving chassis to complete real-time adjustment.

[0118] In step S105, according to the action instruction, the corresponding work component is guided to the target work point through the execution module, and the support task action is executed, and the work component is an operation terminal of the execution module.

[0119] After receiving the dynamically modified action instruction, the execution module drives the work component (such as a drill bit or an anchor rod clamp) to accurately move to the target work point based on the movement path, speed parameter and attitude adjustment amount in the instruction. Specifically, the execution module guides the work component to the target work point along the planned path through mechanical arm joint movement or moving chassis adjustment. In this process, the action instruction is updated and dynamically modified in real time by using an adaptive control algorithm.

[0120] When the work component reaches the target work point, the execution module triggers the corresponding support task action according to the instruction. For example, in the case of drilling as the support task action, the drill bit is controlled to drill at a preset rotation speed and feed speed, and the drilling pressure is dynamically adjusted to avoid sticking by monitoring the drill bit resistance through a force sensor. For another example, in the case of anchor rod installation as the support task action, the anchor rod clamp is aligned with the drilled hole position, the anchor rod is pushed to the set depth, and then the rotation locking mechanism is started to fix the tray and the lock.

[0121] When the support task action is completed, the next standard operation cycle is automatically entered, and steps S101 to S105 are repeatedly executed to realize continuous and efficient automatic support operation.

[0122] By combining stereoscopic image data with time information, the application accurately obtains the visual spatiotemporal information of the target region, effectively realizes accurate representation of the spatial relationship between the robot device and the target work point, enables the robot to dynamically adjust the action instruction according to the real-time updated visual spatiotemporal information when performing the support task, and thus ensures high-precision execution of the support task. Through depth calculation and dynamic correction of the visual information, the application improves the adaptability of the robot to complex environments, reduces the dependence on manual intervention, and improves the safety, automation level and operation efficiency of coal mine operation. In addition, the spatiotemporally correlated visual information can fully capture the spatial position of the target work point, ensure accurate guidance of the work component to the target position, and thus realize precise support operation.

[0123] Figure 2 is an application scenario schematic diagram of a robot support system based on a pair of visual information provided by an embodiment of the application, and the system is applied to steps of a robot support method based on a pair of visual information as described above. Please refer to Figure 2The system comprises a robot device, a data acquisition module, a data storage module, a data processing module and a control module, and the robot device comprises at least one execution module.

[0124] The data acquisition module is configured to acquire stereoscopic image data of a target region to be acquired, wherein the stereoscopic image data comprises raster format data and vector format data, the stereoscopic image data is paired binocular focus data, and the target region comprises a target work point.

[0125] The data acquisition module (i.e. Figure 2 The visual information pair shown in the figure is used for image acquisition of a target region (such as Figure 2 a roadway 205) or an object to be acquired, and controls the focus position. The acquired paired binocular focus data is used as stereoscopic image data, wherein the binocular focus data is video image or graphic data, has certain parallax information, and is used for subsequent depth estimation and spatial positioning. The imaging device or sensing device comprises but is not limited to a binocular vision camera, an infrared imaging system and a laser radar system. The target region refers to a specific region where the robot device needs to work, and comprises a plurality of target work points (such as Figure 2 a drill hole position of an anchor rod drill hole 201).

[0126] The data storage module is configured to determine time information of a current acquisition time of the stereoscopic image data, and store the stereoscopic image data and the corresponding time information, to obtain a visual information pair associated with time and space.

[0127] The data storage module is configured to store the obtained stereoscopic image data of the target region, for calculation of key time and space information such as three-dimensional spatial state and depth information. The acquisition time of each frame of acquired stereoscopic image data is recorded. This is because the stereoscopic image data not only needs to contain spatial information, but also needs to be combined with time information to form a complete time and space data pair. Specifically, after the binocular image data of the target region is acquired, the time information (such as a time stamp) of the current acquisition time is acquired at the same time, and the acquired stereoscopic image data is associated with the corresponding time information and stored together.

[0128] The data processing module is configured to calculate the visual information pair to obtain visual time and space information, and the visual time and space information is used to represent the spatial relationship between the execution module and the target work point.

[0129] The data processing module is configured to perform data fusion on the time and space associated visual information pair stored in step S102, combine the time information and the spatial information, and construct a time and space data set. The time and space data set comprises key visual features of stereoscopic image data, depth information, target work points and work components (such asFigure 2 The three-dimensional coordinates of drill bit 202 and the corresponding acquisition timestamp are obtained. Through spatiotemporal data fusion, a dynamic spatiotemporal feature association model is established to reflect the relative position changes of the target operation point and the execution module in three-dimensional space in real time.

[0130] The control module is used to convert the visual spatiotemporal information into action commands that instruct the execution module to perform corresponding support tasks, and to update and dynamically correct the action commands in real time based on the visual spatiotemporal information.

[0131] The control module generates corresponding support task action commands based on visual spatiotemporal information, instructing the execution module to perform tasks such as drilling and support installation. To ensure operational accuracy, the generated action commands are updated and dynamically corrected in real time based on visual spatiotemporal information. The real-time correction mechanism provided in this embodiment can adjust the action commands promptly based on deviations or environmental changes detected during execution, thereby optimizing the execution process and ensuring operational accuracy and safety. For example, if the target work point location shifts, the commands are dynamically adjusted to ensure the execution module can accurately complete the support task.

[0132] The execution module is used to guide the corresponding working component to the target work point according to the action instruction, and to execute the support task action. The working component is the operation terminal of the execution module.

[0133] The execution module (such as) Figure 2 The drilling rig 203 is used to precisely move its working components (such as drill bits and anchor clamps) towards the target work point based on the movement path, speed parameters, and attitude adjustment amounts in the dynamically corrected motion commands received in the commands. Specifically, the execution module guides the working components to the target work point along the planned path through the movement of the robotic arm joints or the adjustment of the moving chassis. During this process, an adaptive control algorithm is used to update and dynamically correct the motion commands in real time.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, electronic devices, and storage media. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods and apparatuses according to the embodiments of the present application. It is understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing unit, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1 Figure 1 Figure 1 Figure 1 Figure 1

[0136] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to cover all such variations and modifications that fall within the scope of the embodiments of the present application.

[0137] ​​​​​​Finally, it needs to be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of additional same elements in the process, method, article or terminal device including the elements. The above describes in detail the configuration-based multi-level index calculation system and method provided by the present application, and the principles and implementation modes of the present application are described by applying specific examples in this article. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description of the present application should not be understood as a limitation of the present application.

Claims

1. A robot support method based on visual information pairs, characterized in that, Applied to a robotic device, the robotic device including at least one execution module; the method includes: Acquire stereo image data of the target area to be collected. The stereo image data includes raster format data and vector format data. The stereo image data is paired binocular focusing data. The target area includes target work points. Determine the time information of the current acquisition time of the stereoscopic image data, and store the stereoscopic image data and the corresponding time information to obtain a spatiotemporally related visual information pair; The visual information pairs are processed to obtain visual spatiotemporal information, which is used to characterize the temporal and spatial relationship between the execution module and the target work point. The visual spatiotemporal information is converted into action commands to instruct the execution module to perform corresponding support tasks, and the action commands are updated and dynamically corrected in real time based on the visual spatiotemporal information. According to the action command, the corresponding working component is guided to the target work point by the execution module and the support task action is executed. The working component is the operation terminal of the execution module. The target work point includes positioning holes; the process of solving the visual information pair to obtain visual spatiotemporal information includes: Using image processing technology, key visual features are extracted from the stereoscopic image data. These key visual features include the texture, shape, and edge features of the positioning holes and the working parts. Based on the binocular parallax of the stereo image data, depth information is calculated. By combining the depth information, the spatial location and temporal information of the target work point and the working component are used to perform three-dimensional spatial positioning to obtain the visual spatiotemporal information. Also includes: The key visual features in the stereo image data are vectorized to generate multi-dimensional feature vectors of the key visual features. The multi-dimensional feature vectors include at least texture feature vectors, shape feature vectors, and edge feature vectors. The multidimensional feature vectors of the key visual features are mapped to the preset drilling template data to generate feature encoding information. The drilling template data includes the depth information, texture features, shape features and edge features of the standard drilling. The multidimensional feature vector is compared and matched with the borehole template data; Based on the comparison and matching results, the spatial position of the positioning hole is determined; Based on the spatial location of the positioning hole, visual spatiotemporal information is generated to characterize the spatial relationship between the execution module and the positioning hole.

2. The method according to claim 1, characterized in that, The step of extracting key visual features from the stereoscopic image data using image processing technology includes: The stereo image data is preprocessed, including denoising, contrast enhancement, and image registration. Using texture analysis algorithms, the texture features of the positioning holes and the texture features of the working parts are extracted; The shape features of the stereo image data and the shape features of the working component are extracted using a shape recognition algorithm. Edge detection algorithms are used to extract the edge features of the target work point and the work component; The texture features of the positioning holes, the texture features of the working parts, the shape features of the stereoscopic image data, the shape features of the working parts, the edge features of the target work point, and the edge features of the working parts are identified as the key visual features in the stereoscopic image data.

3. The method according to claim 1, characterized in that, The process of calculating depth information based on the binocular disparity of the stereo image data includes: In the visual information pair, the disparity information of each pixel is calculated; Based on the disparity information of each pixel, the depth information of each pixel is calculated.

4. The method according to claim 1, characterized in that, The step of converting the visual spatiotemporal information into action commands to instruct the execution module to perform corresponding support tasks, and updating and dynamically correcting the action commands in real time based on the visual spatiotemporal information, includes: Based on the visual spatiotemporal information, the spatial relationship between the execution module and the target work point is calculated in real time. Based on the spatial relationship between the execution module and the target work point, the action instructions are generated, and the action instructions are updated and dynamically corrected in real time.

5. The method according to claim 4, characterized in that, The real-time updating and dynamic correction of the action commands includes: The action commands are updated and dynamically corrected in real time using adaptive control algorithms, which include Kalman filtering, PID control algorithm, H∞ control algorithm, and neural network control algorithm.

6. The method according to claim 1, characterized in that, The acquisition of stereoscopic image data of the target area to be acquired includes: In the scenario of tunneling or tunnels to be reinforced and supported, a pair of visual acquisition devices are used to acquire a pair of binocular focusing data in the drilling direction within the tunnel scenario; the size of the overlapping area of ​​the field of view acquired by the pair of visual acquisition devices is not less than the preset area size. The paired binocular focusing data along the drilling direction in the tunnel scene are determined as the stereoscopic image data of the target area.

7. A robot support system based on visual information pairs, characterized in that, The system is applied to the steps of performing a robot support method based on visual information pairs as described in any one of claims 1-6, the system comprising: a robot device, a data acquisition module, a data storage module, a data processing module, and a control module, wherein the robot device includes at least one execution module; The data acquisition module is used to acquire stereoscopic image data of the target area to be acquired. The stereoscopic image data includes raster format data and vector format data. The stereoscopic image data is paired binocular focusing data. The target area includes target work points. The data storage module is used to determine the time information of the current acquisition time of the stereoscopic image data, and store the stereoscopic image data and the corresponding time information to obtain a spatiotemporally related visual information pair; The data processing module is used to solve the visual information pair to obtain visual spatiotemporal information, which is used to characterize the spatial relationship between the execution module and the target operation point. The control module is used to convert the visual spatiotemporal information into action commands that instruct the execution module to perform corresponding support tasks, and to update and dynamically correct the action commands in real time based on the visual spatiotemporal information. The execution module is used to guide the corresponding working component to the target work point according to the action instruction, and to execute the support task action. The working component is the operation terminal of the execution module.

8. The system according to claim 7, characterized in that, The robotic device is any one of the following: a multi-axis robotic arm, a humanoid robot, a quadruped robot, a wheeled robot, or a tracked robot.

Images