Multi-source fusion perception system and perception method of spatial dexterous robotic arm

Abstract

A multi-source fusion perception system for a dexterous space manipulator and its perception method. The perception system includes a morphological perception subsystem, a proximity perception subsystem, a visual subsystem, and a multi-source data fusion processing unit. The morphological perception subsystem sends the manipulator's morphological information, and the proximity perception subsystem sends the distance information between the manipulator and the surrounding environment to the multi-source data fusion processing unit; the visual subsystem sends the global three-dimensional point cloud information and local image information to the multi-source data fusion processing unit. The multi-source data fusion processing unit establishes a synchronous registration of the multi-source information of each subsystem, and through the collaborative expression of the multi-source and multi-dimensional information of each subsystem and the fusion interaction of dynamic scene information, it obtains the overall shape of the manipulator, the dynamic scene map, and the relative position of the operation target. It solves the problem of autonomous perception of the overall shape, dynamic scene, and the position of the operation target of the manipulator during the process of reaching typical unstructured and narrow spaces such as solar panel assisted deployment and internal fault detection of space components during on-orbit service and maintenance in space.

Description

Technical Field

[0001] The present invention relates to the field of space on-orbit service and maintenance technology, and in particular to a multi-source fusion perception system and a perception method thereof for a space dexterous manipulator arm. Background Art

[0002] With the rapid development of aerospace science and technology, higher and higher requirements are being placed on space operation technology. Among them, on-orbit service and maintenance technology is of great significance for maintaining spacecraft in good working condition and improving the reliability of mission execution. For typical on-orbit service and maintenance tasks such as assisted deployment of sail panels and fault detection of space components, the space available for monitoring and maintenance operations is very narrow due to the complex structure of spacecraft equipment. Traditional space rigid arms have limited degrees of freedom and have shortcomings such as inflexible movement, flexibility and environmental adaptability. These shortcomings make it difficult to achieve free passage and terminal arrival in unstructured and narrow spaces. For the perception of the arrival and traversal process of space operation tasks of robotic arms, traditional methods mostly use multi-source information separation processing to complete the identification and measurement of the operating environment and target, without considering the changes in the shape of the robotic arm itself. There are problems with incomplete perception information in narrow spaces and poor perception performance. Summary of the Invention

[0003] The technical problem addressed by this invention is to overcome the shortcomings of existing technologies and provide a multi-source fusion perception system for a dexterous spatial manipulator. This system can achieve full autonomous perception of the manipulator's overall shape, dynamic scenes, and target position during the entire process of reaching and traversing typical unstructured, confined spaces, such as for solar panel assisted deployment and internal fault detection of space components. This system offers advantages such as good morphological flexibility, strong environmental adaptability, and comprehensive and accurate perception.

[0004] The technical solution of the present invention is:

[0005] A multi-source fusion perception system for a spatial dexterous robotic arm comprises a morphological perception subsystem, a proximity perception subsystem, a visual subsystem, and a multi-source data fusion processing unit; the morphological perception subsystem is used to obtain morphological information of the robotic arm's bending changes and send it to the multi-source data fusion processing unit; the proximity perception subsystem is used to obtain distance information between the robotic arm and its surroundings and send it to the multi-source data fusion processing unit; the visual subsystem comprises a global vision module and a local vision module, the global vision module is used to obtain a global three-dimensional point cloud and send it to the multi-source data fusion processing unit, and the local vision module is used to obtain a local image and send it to the multi-source data fusion processing unit; the multi-source data fusion processing unit establishes information synchronization among the morphological perception subsystem, the proximity perception subsystem, and the visual subsystem, and obtains the overall morphology of the robotic arm, the dynamic scene map, and the relative position of the operation target through the collaborative expression of multi-source and multi-dimensional information and the fusion interaction of dynamic scene information among the morphological perception subsystem, the proximity perception subsystem, and the visual subsystem.

[0006] The morphology subsystem includes morphology sensors and morphology control units connected in sequence. The morphology sensors are evenly distributed around the entire spatial dexterous manipulator in segments, groups, and multiple viewing angles. The morphology control unit sends the bending change morphology information of each unit manipulator to the multi-source data fusion processing unit through the morphology control unit.

[0007] The proximity sensing subsystem includes a proximity sensing sensor and a proximity sensing control unit connected in sequence. The proximity sensing sensors are evenly distributed on the spatial dexterous manipulator in a one-to-one correspondence with the segmented manipulator units. The distance information between each unit manipulator and the surrounding environment is sent to the multi-source data fusion processing unit through the morphological sensing control unit.

[0008] The visual subsystem consists of a global vision module and a local vision module. The global vision module includes a global vision sensor and a global vision control unit connected in sequence. The global vision control unit sends the acquired global 3D point cloud to the multi-source data fusion processing unit. The local vision module includes a local vision sensor and a local vision control unit connected in sequence. The local vision control unit sends the acquired local image to the multi-source data fusion processing unit.

[0009] The robotic arm includes a plurality of shape bending units, each of which includes a plurality of spacer disks and a central axis fixed drive cable, the central axis fixed drive cable fixes the spacer disks, and the spacer disks include a shape sensor spacer disk and a proximity sensor spacer disk;

[0010] A plurality of shape perception sensors are evenly distributed in the circumference of the shape perception sensor spacer disk;

[0011] A plurality of proximity sensors are evenly distributed in the circumferential direction of the proximity sensor spacer disk.

[0012] The morphology sensors are evenly distributed throughout the spatially dexterous manipulator, using segmented morphology bending units, grouped morphology sensor spacers, and 360-degree multi-viewing angles. Each sensor is coded and labeled. A morphology control unit detects the spatial posture changes of each manipulator unit and transmits the morphology information to a multi-source data fusion processing unit in parallel, segmented, and coded.

[0013] The proximity sensors are evenly distributed on the spatial dexterous robotic arm according to segmented morphological bending units and 360-degree multi-viewing angles, and are coded and marked corresponding to the morphological perception subsystem. The distance information between each unit robotic arm and the surrounding environment is sent to the multi-source data fusion processing unit in parallel according to the coded segments through the morphological perception control unit.

[0014] One end of the robotic arm is fixedly connected to the tracking star body, and the tracking star drives the robotic arm toward the target star;

[0015] The global vision sensor is set at the front viewing angle above the base of the tracking star body robotic arm. The global vision control unit controls the global vision sensor to collect global three-dimensional point cloud information and send it to the multi-source data fusion processing unit; the local vision sensor is installed at the center of the end of the robotic arm. The local vision control unit controls the local vision sensor to collect images of the local area in front of the robotic arm and send them to the multi-source data fusion processing unit.

[0016] By establishing a multi-source information synchronization registration for each subsystem, the data synchronously acquired by the morphological perception subsystem, proximity perception subsystem, and visual subsystem are sent to the multi-source data fusion processing unit. The establishment of the multi-source information synchronization registration for each subsystem includes the synchronization of spatial reference coordinates and the synchronization of associated data.

[0017] A multi-source fusion perception method for a space dexterous manipulator adopts any of the multi-source fusion perception systems for a space dexterous manipulator described above, comprising the following steps:

[0018] S1, perceive the global scene and plan the initial safe path of the dexterous robotic arm;

[0019] S2. Determine the interactive strategy for multi-source information fusion of vision, form perception, and proximity perception;

[0020] S3. Based on global scene perception, the initial safe path of the dexterous robot arm, and the multi-source information fusion interaction strategy of vision / morphological perception / proximity perception, the safe reachable area of ​​the dynamic scene during the arrival process in a confined space is planned and perceived in real time.

[0021] The step S1 includes

[0022] (1) Construct a database of target coding templates of interest, collect global scene point cloud data including the target of interest based on the global vision sensor, and obtain an initial global scene map;

[0023] (2) Based on the improved supervoxel B-spline region segmentation method, the three-dimensional boundary segmentation of the target in the initial global scene map is performed to obtain the segmentation result;

[0024] (3) Filter the segmentation results according to the target coding template database to obtain the segmentation results of the target of interest, which includes the dexterous robot arm, local visual sensor, and narrow gap area;

[0025] (4) Based on the segmentation results of the target of interest, a relative position relationship map between scene perception targets is constructed to obtain the relative position relationship between the dexterous manipulator and surrounding obstacles, and between the end of the dexterous manipulator and the slit;

[0026] (5) According to the relative position relationship between the dexterous manipulator and the surrounding obstacles, and between the end of the dexterous manipulator and the slit, the initial safe path of the manipulator is planned based on the artificial potential field method constrained by the minimum distance map relationship of the scene perception target.

[0027] In step (2), the improved supervoxel B-spline method first uses the point cloud over-segmentation method to segment the initial global scene map to obtain an over-segmented block set T of the three-dimensional scene point cloud:

[0028]

[0029] Among them, t i is an over-segmented block, k is the number of segments;

[0030] Secondly, the over-segmented block set T is processed using the quintic B-spline curve equation to obtain the target optimal boundary curve. The quintic B-spline curve equation is:

[0031]

[0032] Among them, P j It is t i Discrete point cloud data on the edge, n is t i The number of discrete point clouds on the edge, F j，k (t i ) is the k-order B-spline basis function, k=5

[0033]

[0034] P(t i )=P0*F 0，5 (t i )+P1*F 1，5 (t i)+P2*F 2，5 (t i )+P3*F 3，5 (t i )+P4*F 4，5 (t i )+P5*F 5，5 (t i ).

[0035] In the step (5), planning the initial safe path of the manipulator includes: setting the attraction point on the local visual sensor at the end of the dexterous manipulator based on the artificial potential field method of the minimum distance spectrum relationship constraint of the scene perception target, and the attraction point is selected at the minimum intersection position of the distance between the boundary of the local visual sensor at the end and the boundary of the satellite body and the boundary of the solar sail panel; setting the repulsion point at different positions of the dexterous manipulator body, and the repulsion point is selected at the minimum intersection position of the distance between the boundary of the dexterous manipulator body and the boundaries of different obstacles around it;

[0036] The artificial potential field force U(X) includes the gravitational field formed by the moving target and the repulsive field formed by the obstacle:

[0037] U(X)=U att (X)+U rep (X)

[0038]

[0039]

[0040] Among them, U att (X) is the gravitational field, which guides the dexterous robotic arm to move toward the target position in the slit area; U rep (X) is the repulsive field, which guides the dexterous robot arm to avoid obstacles. att and k rep are the attraction and repulsion gain coefficients;

[0041] ρ0 is the obstacle influence distance; m j is the minimum distance between the local visual sensor boundary at the end and the satellite body boundary and the solar panel boundary; d i is the minimum distance between the dexterous robot arm's body boundary and the boundaries of different obstacles around it; d i and m j Obtained based on the relative position relationship between the robotic arm and the obstacle; N is the number of obstacles.

[0042] The step S3 includes

[0043] (1) Based on the morphological perception, the dexterous manipulator perceives its overall morphology during its dynamic motion process and obtains the overall morphological perception model of the dexterous manipulator;

[0044] (2) Based on the interaction of proximity and morphological perception, the dexterous manipulator perceives obstacles around it and obtains a relative position relationship map between scene perception targets based on multi-source fusion;

[0045] (3) Combining the overall morphological perception model of the dexterous manipulator and the relative position relationship map between scene perception targets based on multi-source fusion, and the dynamic and safe path planning of the dexterous manipulator based on the circumferential multi-distributed point artificial potential field method, the autonomous visual perception of the target position in the narrow gap is completed.

[0046] In the step (1), the output data of the multiple evenly distributed morphological perception sensors of the spatial dexterous manipulator perception system are used as variables to form a multidimensional variable space, and the overall morphological perception model of the dexterous manipulator is obtained by the least squares morphological fitting method based on the grouped and segmented joint graph optimization with equal arc constraints;

[0047] In the step (2), the proximity sense / morphology sense are fused and interacted in combination with the overall morphology perception model of the dexterous manipulator, a number of virtual proximity sense sensors corresponding to the morphology sensors are constructed, and the relative position relationship between the virtual proximity sense sensors on the dexterous manipulator and the surrounding obstacles is calculated. Based on the multi-source complementary information of proximity sense and morphology sense, more accurate and complete dexterous arm morphology and surrounding obstacle perception information is obtained through coordinated optimization; with the dexterous manipulator as the center, a relative position relationship map between scene perception targets, as well as the overall morphology perception model of the dexterous manipulator, the dexterous arm morphology and surrounding obstacle perception information, is constructed based on a multi-source fusion-based relative position relationship map between scene perception targets;

[0048] In step (3), based on the relative position relationship between the dexterous manipulator and the surrounding obstacles and between the end of the dexterous manipulator and the slit, the optimal safe path for the manipulator to dynamically reach the narrow gap area is planned based on the circumferential multi-distributed point artificial potential field method.

[0049] In summary, this application has at least the following beneficial technical effects:

[0050] Compared with traditional spatial rigid arms, which have limited degrees of freedom, they have disadvantages such as inflexible movement, flexibility and environmental adaptability. Traditional perception methods mostly use multi-source information separation processing to complete the identification and measurement of the operating environment and target, without considering the changes in the shape of the robot arm itself. There are problems such as incomplete perception information in small spaces and poor perception performance. The present invention adopts a spatial dexterous robot arm with high degrees of freedom, good flexibility, flexible and changeable shape, and excellent accessibility. It can avoid obstacles and arrive flexibly and freely cross small spaces. For the perception of the arrival and crossing process of the robot arm's spatial operation tasks, the present invention combines artificial intelligence and multi-source information, and obtains deeper semantic information through the synchronous collaborative expression and fusion interaction of multi-source and multi-dimensional information, thereby improving the full-process autonomous perception capability of its own shape, dynamic scenes and the position of the operation target. It has the advantages of good morphological flexibility, strong environmental adaptability, and high perception accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The multi-source fusion perception system and perception method of the spatial dexterous robotic arm of the present invention are given in the following embodiments and drawings.

[0052] Figure 1 This is the overall flow chart of the multi-source fusion perception system of the spatial dexterous manipulator of the present invention;

[0053] Figure 2 This is a schematic diagram of the structural layout of the flexible units of the spatial dexterous robotic arm of the present invention;

[0054] Figure 3 This is a schematic diagram of the multi-source fusion perception environment of the spatial dexterous manipulator of the present invention;

[0055] Figure 4 This is a flow chart of the multi-source fusion interaction strategy of vision / morphological perception / proximity perception in step S1 of the present invention;

[0056] Figure 5 This is a flowchart for planning the initial safe path of the manipulator based on the artificial potential field method of the scene perception target minimum distance spectrum relationship constraint in step S1 of the present invention;

[0057] Figure 6 This is a flowchart of the implementation of the supervoxel B-spline method based on the rough matching of the target three-dimensional model in step S1 of the present invention.

[0058] Explanation of the accompanying symbols: 1. Morphological perception sensor; 2. Proximity perception sensor; 3. Global vision sensor; 4. Local vision sensor; 5. Morphological perception control unit; 6. Proximity perception control unit; 7. Global vision control unit; 8. Local vision control unit; 9. Multi-source data fusion processing unit. DETAILED DESCRIPTION

[0059] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0060] The present application discloses a multi-source fusion perception system for a spatial dexterous manipulator. Figure 1 The multi-source fusion perception system of the space dexterous manipulator of the present invention is set on the tracking spacecraft. The perception system includes a morphological perception sensor 1, a proximity perception sensor 2, a global vision sensor 3, a local vision sensor 4, a morphological perception control unit 5, a proximity perception control unit 6, a global vision control unit 7, a local vision control unit 8 and a multi-source data fusion processing unit 9.

[0061] The robotic arm consists of multiple morphological bending units, each of which includes multiple spacer discs and a central axis-fixed drive cable that secures the spacer discs. One end of the robotic arm is fixedly connected to the tracking satellite, which drives the robotic arm toward the target satellite.

[0062] The morphological sensor 1 is evenly distributed on the entire space dexterous manipulator according to the segmented morphological bending unit, grouped morphological sensor spacer, and 360-degree multi-viewing angle, and is coded and marked one by one according to different segments M, different groups n, and different viewing angles i. The coding format is "morphological sensor F M-n-i ”.

[0063] The morphological perception control unit 5 is connected to the morphological perception sensor 1. The morphological perception control unit 5 is used to control the morphological perception sensor 1 and obtain the morphological information of the bending changes of each unit robot arm, and send it to the multi-source data fusion processing unit 9 in parallel according to the coding segmentation;

[0064] The proximity sensor 2 is evenly distributed on the space dexterous manipulator according to the segmented morphological bending unit and 360-degree multi-viewing angle, and is coded with the corresponding morphological subsystem. The coding format is "Proximity Sensor T M-i ”.

[0065] The proximity control unit 6 is connected to the proximity sensor 2. The proximity control unit 6 is used to control the proximity sensor 2 and obtain the distance information between each unit robot arm and the surrounding environment, and send it to the multi-source data fusion processing unit 9 in parallel according to the coding segmentation;

[0066] See also Figure 2 In a preferred embodiment of the present invention, the number of spacer discs in each bending unit is 6, including 5 shape sensor spacer discs and 1 proximity sensor spacer disc. Each shape sensor spacer disc has 5 shape sensors, and each proximity sensor spacer disc has 5 proximity sensors. The total number of spacer discs in the entire space dexterous manipulator is N. total =5*6=30, the total number of sensors for the entire space dexterous manipulator N TENG=5*5*5=125, the total number of proximity sensors of the entire space dexterous manipulator N TOF =5*1*5=25.

[0067] See also Figure 2 In a preferred embodiment of the present invention, the form sensor 1 can be a TENG (triboelectric nanogenerator) form sensor. It is composed of five consecutive segments (M = 1, 2, 3, 4, 5) of form-bending units. Each segment of the form-bending units is evenly distributed with five groups (n = 1, 2, 3, 4, 5) of form sensor spacers. Each group of form sensor spacers is distributed with five viewing angles (i = 1, 2, 3, 4, 5), that is, a form sensor is arranged every 72 degrees, forming a 360-degree full viewing angle distribution.

[0068] See also Figure 2 In a preferred embodiment of the present invention, the proximity sensor 2 can be a TOF photoelectric ranging proximity sensor. Five groups (M = 1, 2, 3, 4, 5) of proximity sensor spacers are arranged at the position of the third group of spacers in each morphological bending unit. Each group of proximity sensor spacers is distributed across five viewing angles (i = 1, 2, 3, 4, 5), that is, a proximity sensor is arranged every 72 degrees, forming a 360-degree full viewing angle distribution.

[0069] The global vision sensor 3 is arranged at a front viewing angle above the base of the tracking star body robotic arm; the global vision control unit 7 is connected to the global vision sensor 3, and the global vision control unit 7 is used to control the global vision sensor 3 to collect the three-dimensional point cloud information of the global scene and send it to the multi-source data fusion processing unit 9;

[0070] See also Figure 3 In a preferred embodiment of the present invention, the global vision sensor 3 can adopt a binocular structured light 3D camera, which can obtain three-dimensional point cloud information of scenes such as the dexterous robotic arm, space environment obstacles, local vision sensors, and narrow gap areas before arriving in a narrow space in real time.

[0071] The local vision sensor 4 is installed at the end center of the dexterous robotic arm; the local vision control unit 8 is connected to the local vision sensor 4, and the local vision control unit 8 is used to control the local vision sensor 4 to collect images of the local area in front of the robotic arm movement and send them to the multi-source data fusion processing unit.

[0072] See also Figure 3 In a preferred embodiment of the present invention, the local vision sensor 4 can adopt a TOF (Time of Flight) 3D camera, which can obtain in real time the depth image and color image of the local area in front of the robot arm during the process of arriving in a narrow space and the operating area in the narrow gap after arrival.

[0073] The multi-source data fusion processing unit 9 establishes a synchronous registration of multi-source information of each subsystem, and obtains the overall shape of the robotic arm, the dynamic scene map and the relative position of the operation target through the collaborative expression of multi-source and multi-dimensional information of each subsystem and the fusion interaction of dynamic scene information.

[0074] The establishment of the multi-source information synchronous registration of each subsystem includes synchronization of spatial reference coordinates and synchronization of associated data;

[0075] See also Figure 3 In a preferred embodiment of the present invention, the synchronization of the spatial reference coordinates is achieved by transforming the data obtained by the morphological perception, proximity perception, and visual subsystem sensors into the same reference coordinate system, which is defined on the base of the tracking star and uniquely determined by the global calibration system.

[0076] In a preferred embodiment of the present invention, the synchronization of the associated data ensures that the data time transmitted by the morphological perception, proximity perception, and visual subsystems at the current moment remains consistent by setting a synchronization trigger signal, wherein the morphological perception sensor data is synchronously and parallelly acquired from multiple perspectives of segmented morphological bending units, grouped morphological sensor spacers, and the same morphological perception spacer; the proximity perception sensor data is synchronously and parallelly acquired from multiple perspectives of segmented morphological bending units and the same proximity perception spacer; the morphological perception and proximity perception data are sent in parallel according to 5 segmented bending units, and each segmented bending unit performs associated synchronization of the morphological perception and proximity perception data according to a one-to-one corresponding coding mark.

[0077] Another technical solution provided by the present invention is a multi-source fusion perception method for a spatial dexterous manipulator, which uses the multi-source fusion perception system for a spatial dexterous manipulator. The perception method includes the following steps:

[0078] S1, perceive the global scene and plan the initial safe path of the dexterous robotic arm;

[0079] S2. Determine the interactive strategy for multi-source information fusion of vision, form perception, and proximity perception;

[0080] S3. Based on global scene perception, the initial safe path of the dexterous robot arm, and the multi-source information fusion interaction strategy of vision / morphological perception / proximity perception, the safe reachable area of ​​the dynamic scene during the arrival process in a confined space is planned and perceived in real time.

[0081] In a preferred embodiment of the present invention, in step S1, the supervoxel B-spline method based on the coarse matching of the target three-dimensional model is used to realize the rapid three-dimensional recognition and boundary segmentation of targets such as dexterous robotic arms, spatial environment obstacles, local visual sensors, and narrow gap areas in the initial global scene map, obtain the target three-dimensional boundary curve equation, calculate the relative positions between targets based on the target three-dimensional boundary curve equation, and construct a relative position relationship map between scene perception targets.

[0082] See also Figure 4 and Figure 5 Specifically, step S1 is implemented as follows:

[0083] (1) Construct a database of target coding templates of interest, collect global scene point cloud data including the target of interest based on the global vision sensor, and obtain an initial global scene map;

[0084] (2) Based on the improved supervoxel B-spline region segmentation method, the three-dimensional boundary segmentation of the objects in the initial global scene map is performed to obtain the three-dimensional segmentation results of all objects in the scene;

[0085] (3) Filter the 3D segmentation results of all targets in the scene according to the target coding template database to obtain the segmentation results of the targets of interest, which include the dexterous manipulator, local visual sensor, and narrow gap area;

[0086] (4) Based on the segmentation results of the target of interest, a relative position relationship map between scene perception targets is constructed to obtain the relative position relationship between the dexterous manipulator and surrounding obstacles, and between the end of the dexterous manipulator and the slit;

[0087] (5) According to the relative position relationship between the dexterous manipulator and the surrounding obstacles, and between the end of the dexterous manipulator and the slit, the initial safe path of the manipulator is planned based on the artificial potential field method constrained by the minimum distance map relationship of the scene perception target.

[0088] The target coding template database in step (1) includes three-dimensional CAD models of key components of the space satellite, such as a dexterous manipulator 1, a local visual sensor 2, a satellite body 3, a solar sail panel support 4, a narrow gap area 6, and a solar sail panel 7, which have coding information;

[0089] The three-dimensional boundary segmentation of the global scene map target in step (2) includes three-dimensional recognition and boundary segmentation of the dexterous manipulator, local visual sensor, and narrow gap area in the global scene map;

[0090] As described above, the improved supervoxel B-spline method first uses the point cloud over-segmentation method to segment the initial global scene map to obtain an over-segmented block set T of the three-dimensional scene point cloud:

[0091]

[0092] Among them, t i is an over-segmented block, k is the number of segments;

[0093] Secondly, the over-segmented block set T is processed using the quintic B-spline curve equation to obtain the target optimal boundary curve (i.e., the target three-dimensional boundary curve equation mentioned above), where the quintic B-spline curve equation is:

[0094]

[0095] Among them, P j It is t i Discrete point cloud data on the edge, n is t i The number of discrete point clouds on the edge, F j，k (t i ) is the k-order B-spline basis function, k=5

[0096]

[0097] P(t i )=P0*F 0，5 (t i )+P1*F 1，5 (t i )+P2*F 2，5 (t i )+P3*F 3，5 (t i )+P4*F 4，5 (t i )+P5*F 5，5 (t i )

[0098] In the step (3), according to the target coding template database, firstly, a point cloud segmentation method based on three-dimensional model registration is used to quickly segment the coded targets in the scene, and then a segmentation result of the target of interest is obtained by differential screening with the three-dimensional segmentation results of all targets in the scene. The target of interest includes a dexterous robotic arm, a local visual sensor, and a narrow gap area;

[0099] The relative position relationship map between scene perception targets in step (4) is centered on the dexterous manipulator and calculates the relative position relationship between the dexterous manipulator and surrounding obstacles, and between the end of the dexterous manipulator and the slit based on the target boundary curve equation.

[0100] In the step (5), the initial safe path of the robotic arm is planned based on the artificial potential field method constrained by the minimum distance graph relationship of the scene perception target.

[0101] like Figure 5 and Figure 6As shown, in a preferred embodiment of the present invention, in order to achieve the arrival of the dexterous manipulator in an unstructured narrow space, taking into account the possibility that the dexterous manipulator body will also encounter obstacles, the artificial potential field method based on the minimum distance map relationship constraint of the scene perception target is used to set the attraction point on the local visual sensor at the end of the dexterous manipulator, and the attraction point is selected at the minimum intersection position of the distance between the boundary of the local visual sensor at the end and the boundary of the satellite body and the boundary of the solar panel; the repulsion point is set at different positions on the body of the dexterous manipulator, and the attraction point is selected at the minimum intersection position of the distance between the boundary of the dexterous manipulator body and the boundaries of different obstacles around it.

[0102] Specifically, the artificial potential field force U(X) based on the minimum distance graph relationship constraint of the scene perception target includes the gravitational field formed by the moving target and the repulsive field formed by the obstacle:

[0103] U(X)=U att (X)+U rep (X)

[0104]

[0105]

[0106] Among them, U att (X) is the gravitational field, which guides the dexterous robotic arm to move toward the target position in the slit area; U rep (X) is the repulsive field, which guides the dexterous robot arm to avoid obstacles. att and k rep are the attraction and repulsion gain coefficients.

[0107] ρ0 is the obstacle influence distance; m j is the minimum distance between the local visual sensor boundary at the end and the satellite body boundary and the solar panel boundary; d i is the minimum distance between the dexterous robot arm's body boundary and the boundaries of different obstacles around it; d i and m j Obtained based on the relative position relationship between the robotic arm and the obstacle in step 4; N is the number of obstacles.

[0108] In the above-mentioned multi-source fusion perception method of the spatial dexterous robotic arm, in step S2, the visual / morphological / proximity multi-source information fusion interaction strategy adopts a strategy that combines the visual perception of the operation target with the morphological / proximity fusion obstacle perception around the robotic arm.

[0109] See also Figure 6In a preferred embodiment of the present invention, in step S2, the obstacles in the front observation field of view and the operating targets in the slit area during the arrival process of the dexterous manipulator are perceived by the local visual sensor; the obstacles in the overall surrounding environment of the spatial dexterous manipulator during the arrival process are perceived by the fusion of morphological perception and proximity perception information; finally, the visual information of the spatial dexterous manipulator and the morphological perception / proximity perception fusion information are subjected to multi-source information fusion interaction.

[0110] In the multi-source fusion perception method for the spatial dexterous manipulator, step S3 includes the following steps:

[0111] S31 Based on the shape perception, the dexterous manipulator perceives its own overall shape during the dynamic motion process and obtains the overall shape perception model of the dexterous manipulator;

[0112] S32 is based on the fusion interaction of proximity and morphology perception to perceive obstacles around the dexterous manipulator, and obtains a relative position relationship map between scene perception targets based on multi-source fusion;

[0113] S33 combines the overall morphological perception model of the dexterous robotic arm and the relative position relationship map between scene perception targets based on multi-source fusion, and the dynamic and safe path planning of the dexterous robotic arm based on the circumferential multi-distributed point artificial potential field method to complete the autonomous visual perception of the target position in the narrow gap.

[0114] The step S31 uses the measurement output data of multiple evenly distributed morphological sensors of the spatial dexterous manipulator perception system as variables to form a multidimensional variable space, and obtains the overall morphology of the dynamic motion process of the dexterous manipulator by the least squares morphological fitting method based on the grouped segmented joint graph optimization with equal arc constraints.

[0115] The step S32 is combined with the overall morphological perception model of the dexterous manipulator in step S31, and the proximity perception / morphological perception are integrated and interacted to construct a number of virtual proximity sensors corresponding to the morphological sensors, and the relative position relationship between the proximity sensors on the dexterous manipulator and the surrounding obstacles is calculated. Based on the multi-source complementary information of proximity perception and morphological perception, more accurate and complete dexterous arm morphology and surrounding obstacle perception information is obtained through coordinated optimization; with the dexterous manipulator as the center, combined with the relative position relationship between targets based on the scene perception of the visual sensor in step 4 of step S1, as well as the self-morphological information obtained in S31, and the distance information between the proximity sensors of the dexterous manipulator and the surrounding obstacles obtained in S32, a relative position relationship map between scene perception targets based on multi-source fusion is constructed;

[0116] The step S33 is based on the relative position relationship map between the scene perception targets of multi-source fusion, and plans the optimal safe path for the robot arm to dynamically reach the narrow gap area based on the circumferential multi-distributed point artificial potential field method according to the relative position relationship between the dexterous robot arm and the surrounding obstacles and between the end of the dexterous robot arm and the narrow gap.

[0117] In a preferred embodiment of the present invention, the attraction points are set on the local visual sensor at the end of the dexterous manipulator based on the circumferential multi-distributed point artificial potential field method, and the attraction points are respectively selected at the minimum intersection positions of the boundary of the local visual sensor at the end and the boundary of the satellite body and the boundary of the solar sail panel; the repulsion points are set at different positions on the body of the dexterous manipulator, and the repulsion points are selected at the minimum intersection positions of the boundary of the dexterous manipulator and the boundaries of different obstacles around it and at the proximity sensing position (including virtual proximity sensing);

[0118] The artificial potential field force U(X) includes the gravitational field formed by the moving target and the repulsive field formed by the obstacle:

[0119] U(X)=U att (X)+U rep (X)

[0120]

[0121]

[0122] Among them, U att (X) is the gravitational field, which guides the dexterous robotic arm to move toward the target position in the slit area; U rep (X) is the repulsive field, which guides the dexterous robot arm to avoid obstacles. att and k rep are the attraction and repulsion gain coefficients;

[0123] ρ0 is the obstacle influence distance; m j is the minimum distance between the local visual sensor boundary at the end and the satellite body boundary and the solar panel boundary; d i is the minimum distance between the dexterous robot arm's body boundary and the boundaries of different obstacles around it; d i and m j Obtained based on the relative position relationship between the robotic arm and the obstacle; N is the number of obstacles, M is the number of proximity sensors (M = 6 * 5 * 5 = 150), L i is the distance between the proximity sensor and different obstacles around the robotic arm.

[0124] In a preferred embodiment of the present invention, the optimal safe path for the robotic arm to dynamically reach a narrow gap area is planned based on the multi-distributed point artificial potential field method, and after arrival, a target augmentation data set is constructed by combining virtual and real local visual sensors, and rapid annotation of point cloud targets is completed based on adaptive segmentation coarse annotation and manual fine annotation. Autonomous visual perception of the target posture in the narrow gap is completed based on the fast ICP and PointNet++ fusion method.

[0125] The described fast ICP and PointNet++ fusion method is based on the PointNet++ layered architecture network model. It uses a set of 3D CNN convolutions and downsampling to extract and group 3D point cloud features. It then processes the data in two separate steps, depending on the task. One step feeds the segmentation network, which uses transposed convolutions on the point cloud features to interpolate and ultimately segment the point cloud. The other step feeds the classification network, which further classifies the point cloud objects through feature encoding and a fully connected network. The ICP algorithm is then used to estimate the pose of the target point cloud. ICP implementation process:

[0126] Given two 3D data point sets from different coordinate systems, find the spatial transformation of the two point sets so that they can be spatially matched. Assume that {P i , i = 1, 2, ..., N} represents the first point set in space, and the alignment matching transformation of the second point set is to minimize the objective function of the following formula.

[0127]

[0128] In the present invention, the multi-source data fusion processing unit 9 obtains the overall shape of the robotic arm, the dynamic scene map and the relative position of the operation target based on the collaborative expression and fusion interaction of multi-source and multi-dimensional information of morphological perception, proximity perception and vision. Compared with the perception method of the prior art (which completes the identification and measurement of the operating environment and the target through multi-source information separation processing without considering the changes in the shape of the robotic arm itself), it has the advantages of good flexibility in small space shape, strong environmental adaptability, comprehensive perception and high accuracy.

[0129] Although the contents of the present invention have been described in detail through the above preferred embodiments, it should be recognized that the above description should not be regarded as a limitation of the present invention. The image processing method and solution method involved in the process of the multi-source fusion perception system of the spatial dexterous manipulator of the present invention for the multi-source data fusion processing unit to obtain the overall form of the manipulator, the dynamic scene map and the relative posture of the operation target are not the only methods of the present invention. After reading the above content, various modifications and substitutions of the present invention will be obvious to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A multi-source fusion perception system for a spatial dexterous manipulator, characterized by: It includes morphological perception subsystem, proximity perception subsystem, visual subsystem, and multi-source data fusion processing unit; The morphological perception subsystem is used to obtain the bending change morphological information of the robotic arm and send it to the multi-source data fusion processing unit; The proximity sensing subsystem is used to obtain the distance information between the robotic arm and the surrounding environment and send it to the multi-source data fusion processing unit; The visual subsystem includes a global vision module and a local vision module. The global vision module is used to obtain the global three-dimensional point cloud and send it to the multi-source data fusion processing unit. The local vision module is used to obtain the local image and send it to the multi-source data fusion processing unit. The multi-source data fusion processing unit establishes information synchronization among the morphological perception subsystem, proximity perception subsystem, and visual subsystem. Through the collaborative expression of multi-source and multi-dimensional information and the fusion interaction of dynamic scene information among the morphological perception subsystem, proximity perception subsystem, and visual subsystem, the overall shape of the robotic arm, the dynamic scene map, and the relative position of the operation target are obtained. The morphological perception subsystem includes morphological perception sensors and morphological perception control units connected in sequence. The morphological perception sensors are evenly distributed on the entire spatial dexterous manipulator in segments, groups, and multiple perspectives. The bending change morphological information of each unit manipulator is sent to the multi-source data fusion processing unit through the proximity perception control unit. The proximity sensing subsystem includes a proximity sensing sensor and a proximity sensing control unit connected in sequence. The proximity sensing sensors are evenly distributed on the spatial dexterous manipulator in a one-to-one correspondence with the segmented manipulator units. The distance information between each unit manipulator and the surrounding environment is sent to the multi-source data fusion processing unit through the morphological sensing control unit. The visual subsystem consists of a global vision module and a local vision module. The global vision module includes a global vision sensor and a global vision control unit connected in sequence, and the global vision control unit sends the acquired global three-dimensional point cloud to the multi-source data fusion processing unit; the local vision module includes a local vision sensor and a local vision control unit connected in sequence, and the local vision control unit sends the acquired local image to the multi-source data fusion processing unit.

2. The multi-source fusion perception system for a spatial dexterous manipulator according to claim 1 is characterized by: The robotic arm includes a plurality of shape bending units, each of which includes a plurality of spacer disks and a central axis fixed drive cable, the central axis fixed drive cable fixes the spacer disks, and the spacer disks include a shape sensor spacer disk and a proximity sensor spacer disk; A plurality of shape perception sensors are evenly distributed in the circumference of the shape perception sensor spacer disk; A plurality of proximity sensors are evenly distributed in the circumferential direction of the proximity sensor spacer disk.

3. The multi-source fusion perception system for a spatial dexterous manipulator according to claim 1 is characterized by: One end of the robotic arm is fixedly connected to the tracking star body, and the tracking star drives the robotic arm toward the target star; The global vision sensor is set at the front viewing angle above the base of the tracking star body robotic arm. The global vision control unit controls the global vision sensor to collect global three-dimensional point cloud information and send it to the multi-source data fusion processing unit; the local vision sensor is installed at the center of the end of the robotic arm. The local vision control unit controls the local vision sensor to collect images of the local area in front of the robotic arm and send them to the multi-source data fusion processing unit.

4. A multi-source fusion perception method for a spatial dexterous robotic arm, characterized by: The multi-source fusion perception system for a spatial dexterous manipulator according to any one of claims 1 to 3 is used, comprising the following steps: S1, perceive the global scene and plan the initial safe path of the dexterous robotic arm; S2. Determine the interactive strategy for multi-source information fusion of vision, form perception, and proximity perception; S3. Based on global scene perception, the initial safe path of the dexterous robot arm, and the multi-source information fusion interaction strategy of vision / morphological perception / proximity perception, the safe reachable area of ​​the dynamic scene during the arrival process in a confined space is planned and perceived in real time.

5. The multi-source fusion perception method for a spatial dexterous manipulator according to claim 4 is characterized in that: The step S1 includes (1) Construct a database of target coding templates of interest, collect global scene point cloud data including the target of interest based on the global vision sensor, and obtain an initial global scene map; (2) Based on the improved supervoxel B-spline region segmentation method, the three-dimensional boundary segmentation of the target in the initial global scene map is performed to obtain the segmentation result; (3) Filter the segmentation results according to the target coding template database to obtain the segmentation results of the target of interest, which includes the dexterous robot arm, local visual sensor, and narrow gap area; (4) Based on the segmentation results of the target of interest, a relative position relationship map between scene perception targets is constructed to obtain the relative position relationship between the dexterous robot arm and the surrounding obstacles, and between the end of the dexterous robot arm and the slit; (5) Based on the relative position relationship between the dexterous manipulator and the surrounding obstacles, and between the end of the dexterous manipulator and the slit, the initial safe path of the manipulator is planned based on the artificial potential field method constrained by the minimum distance map relationship of the scene perception target.

6. The multi-source fusion perception method for a spatial dexterous manipulator according to claim 5 is characterized by: In step (2), the improved supervoxel B-spline method first uses the point cloud over-segmentation method to segment the initial global scene map to obtain an over-segmented block set T of the three-dimensional scene point cloud: in, is an over-segmented block, is the number of blocks to be split; Secondly, the over-segmented block set T is processed using the quintic B-spline curve equation to obtain the target optimal boundary curve. The quintic B-spline curve equation is: in, yes Discrete point cloud data on the edge, n is The number of discrete point clouds on the edge, is the k-order B-spline basis function, k=5; 。 7. The multi-source fusion perception method for a spatial dexterous manipulator according to claim 5 is characterized by: In step (5), planning the initial safe path of the robotic arm includes: The artificial potential field method based on the minimum distance graph relationship constraint of scene perception targets sets the attraction point on the local visual sensor at the end of the dexterous manipulator. The attraction point is selected at the minimum intersection position of the distance between the boundary of the local visual sensor at the end and the boundary of the satellite body and the boundary of the solar panel. The repulsion point is set at different positions on the body of the dexterous manipulator. The repulsion point is selected at the minimum intersection position of the distance between the boundary of the dexterous manipulator and the boundaries of different obstacles around it. Artificial potential field force Including the gravitational field formed by the moving target and the repulsive field formed by the obstacles: in, It is the gravitational field that guides the dexterous robotic arm to move toward the target position in the slit area; It is the repulsive field that guides the dexterous robotic arm to avoid obstacles; and are the attraction and repulsion gain coefficients; The distance affected by obstacles; The minimum distance between the local visual sensor boundary at the end and the satellite body boundary and the solar panel boundary; The minimum distance between the dexterous robot arm's body boundary and the boundaries of different obstacles around it; d i and m j Obtained based on the relative position relationship between the robotic arm and the obstacle; N is the number of obstacles.

8. The multi-source fusion perception method for a spatial dexterous manipulator according to claim 5 is characterized by: The step S3 includes (1) Based on morphological perception, the dexterous manipulator perceives its overall morphology during its dynamic motion process and obtains the overall morphological perception model of the dexterous manipulator; (2) Based on the interaction of proximity perception and morphological perception, the dexterous manipulator perceives obstacles around it and obtains a relative position relationship map between scene perception targets based on multi-source fusion; (3) Combining the overall morphological perception model of the dexterous manipulator and the relative position relationship map between scene perception targets based on multi-source fusion, and the dynamic and safe path planning of the dexterous manipulator based on the circumferential multi-distributed point artificial potential field method, the autonomous visual perception of the target position in the narrow gap is completed.

9. The multi-source fusion perception method for a spatial dexterous manipulator according to claim 8 is characterized by: In the step (1), the output data of the multiple uniformly distributed morphological perception sensors of the spatial dexterous manipulator perception system are used as variables to form a multidimensional variable space, and the overall morphological perception model of the dexterous manipulator is obtained by the least squares morphological fitting method based on the grouped segmented joint graph optimization with equal arc constraints; In the step (2), the proximity sense / morphology sense are fused and interacted in combination with the overall morphology perception model of the dexterous manipulator, and a number of virtual proximity sense sensors corresponding to the morphology sensors are constructed, and the relative position relationship between the virtual proximity sense sensors on the dexterous manipulator and the surrounding obstacles is calculated. Based on the multi-source complementary information of proximity sense and morphology sense, more accurate and complete dexterous arm morphology and surrounding obstacle perception information is obtained through coordinated optimization; with the dexterous manipulator as the center, a relative position relationship map between scene perception targets, as well as the overall morphology perception model of the dexterous manipulator, the dexterous arm morphology and surrounding obstacle perception information, is constructed based on a multi-source fusion-based relative position relationship map between scene perception targets; In step (3), based on the relative position relationship between the dexterous manipulator and the surrounding obstacles and between the end of the dexterous manipulator and the narrow gap, the optimal safe path for the manipulator to dynamically reach the narrow gap area is planned based on the circumferential multi-distributed point artificial potential field method.

Images