An environmental field acquisition system based on allosteric arm-mounted vision

Through an environmental field acquisition system based on allosteric arm-mounted vision, combined with RGB-D camera and 360-degree panoramic radar, multi-angle surround view monitoring and three-dimensional reconstruction in urban ruins, solving the difficulty of searching and rescue of urban natural disasters and secondary The issue of disaster risk has improved search and rescue efficiency and safety.

CN112446949BActive Publication Date: 2024-11-01TIANJIN UNIV OF TECH & EDUCATION (TEACHER DEV CENT OF CHINA VOCATIONAL TRAINING & GUIDANCE) +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010908441.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-02
Publication Date
2024-11-01
Estimated Expiration
2040-09-02

AI Technical Summary

Technical Problem

Search and rescue due to urban natural disasters is difficult. The existing rescue methods cannot carry out search and rescue in a timely and accurate manner, and may cause secondary disasters, threatening personnel safety, and need an emergency search and rescue in the ruins of the city and sending them back to the disaster in real time. System of environmental information.

Method used

Design an environmental field acquisition system based on allosteric arm-mounted vision, including allosteric robotic arm, RGB-D camera, image processing module and control module, which can form a binocular, trigonometric, and six-eye three-dimensional depth vision system, through images The collection, processing and control instructions realize real-time collection of environmental information and robot motion control, and combine 360-degree panoramic radar for three-dimensional reconstruction and environmental state simulation.

Benefits of technology

Extended visual detection under narrow gaps and multiple occlusions is achieved, search and rescue efficiency is improved, scientific basis is provided to implement rescue plans, and the efficiency and safety of rescue in accident areas is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112446949B_ABST
    Figure CN112446949B_ABST
Patent Text Reader

Abstract

The present invention relates to an environmental field acquisition system based on allosteric arm-mounted vision, comprising: an allosteric manipulator, a robot body, an image acquisition module, an image processing module and a control module. The image acquisition module is connected to the image processing module, and transmits the collected environmental information to the image processing module for image processing; the control module is connected to the image processing module, and can control the robot according to the processed image information; the image acquisition module is installed on the manipulator, and the control module and the image processing module are installed in the robot body; the system can make the robot allosteric group become a binocular, trinocular, hexacular and other stereoscopic depth vision system. The present invention is of great help to the rescue robot in establishing a large-scale field of view monitoring area, and improves the efficiency of rescue in the accident area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of post-disaster rescue robots, and in particular relates to an environmental field acquisition system based on allosteric arm-mounted vision. Background Art

[0002] Urban natural disasters and environmental disasters are currently on the rise. The number of casualties caused by the collapse of various buildings caused by disasters and the lack of timely search and rescue is greater than the number caused by the disasters themselves. Therefore, the development of urban emergency search and rescue equipment has huge social benefits. However, the city has crisscrossing streets and alleys, tall, solid and dense buildings, complex underground engineering facilities, harsh conditions, and many narrow spaces, which greatly increases the difficulty of search and rescue and hinders the entry of ground search teams. The existing rescue methods cannot guarantee the timely and accurate implementation of search and rescue tasks; the secondary disasters that may occur during the search and rescue process also greatly threaten the safety of relevant personnel. In order to carry out emergency search and rescue tasks in the urban ruins environment, and to transmit post-disaster environmental information to the rear personnel in real time, and provide a scientific basis for rescue personnel to implement rescue plans, an environmental field acquisition system based on allosteric arm-mounted vision is needed to solve the problem. Summary of the invention

[0003] To achieve the above object, the technical solution of the present invention is as follows:

[0004] An environmental field acquisition system based on allosteric arm-mounted vision includes an allosteric manipulator, a robot body, an image acquisition module, a control module and an image processing module. The image acquisition module is an RGB-D camera installed at the end of the allosteric manipulator for real-time acquisition of environmental information; the image processing module processes the image acquired by the image acquisition module and transmits the data information to the control module; the control module sends a control instruction to the robot according to the information transmitted by the image processing module to control the movement of the robot. The system can transform the robot into a binocular, trinocular or hexacular stereoscopic depth vision system according to different needs, and realize extended visual detection in narrow gaps and under various occlusions.

[0005] Furthermore, the binocular stereo depth vision system is characterized in that: when binocular vision is required to collect image information, the image acquisition modules of any two metamorphic robotic arms are controlled to form binocular vision, and the remaining robotic arms are used to form a two-legged, three-legged, or six-legged movement mode. The binocular stereo depth vision calibration steps are as follows:

[0006] Step 1: The robot controls two allosteric manipulator image acquisition modules to form a binocular vision system. After the allosteric manipulator position is fixed, the rotation angle and position relationship of each allosteric manipulator motor are known. The position relationship between the robot's allosteric manipulator coordinate system and the world coordinate system is established through the robot forward kinematics solution, which is expressed by the homogeneous transformation matrix. The formula is as follows:

[0007]

[0008] Among them, R is the rotation matrix and P is the position in the world coordinate system.

[0009] According to the rotation angle of the allosteric manipulator, the homogeneous transformation matrix of the link coordinate system {i} relative to the coordinate system {i-1} is obtained by combining the DH parameters, and the following homogeneous transformation matrix is ​​obtained:

[0010]

[0011] Step 2: Use the "Zhang calibration method" to calibrate respectively to obtain the internal and external parameters of the allosteric robotic arm image acquisition module. The calibration formula and coordinate system symbols are as follows:

[0012]

[0013] Step 3: Perform image correction on a single camera in each camera group. For radial distortion, the first two terms of the Taylor series expansion around the principal point are used to describe it, expressed as k1 and k2. According to their distribution positions in the radial direction, the adjustment formula is:

[0014] x0=x(1+k1r 2 +k2r 4 )

[0015] y0=y(1+k1r 2 +k2r 4 )

[0016] For tangential distortion, the distortion model can be described by two additional parameters p1 and p2:

[0017] x0=2p1xy+p2(r 2 +2x 2 )+1

[0018] y0=2p2xy+p2(r 2 +2y 2 )+1

[0019] Step 4: When calibrating the binocular stereo vision system, the relative relationship between the left and right camera coordinate systems is calibrated according to Step 1. According to the internal and external parameter separation method, the relative positions R and T of the two allostatic manipulators are obtained through the external parameter relationship. The calibration equation is:

[0020]

[0021] T=T r -RT1

[0022] Where R1, R r , T1, T r They are the extrinsic parameters obtained by single-target calibration of two cameras respectively;

[0023] Step 5: When performing stereo correction, use the Bouguet correction method to decompose the rotation matrix R and translation matrix T into the rotation and translation matrices R1 and R2 of the left and right cameras respectively. r , T1, T r The decomposition principle is that the distortion caused by the reprojection of the left and right images is minimized and the common area of ​​the left and right views is maximized; the specific steps are as follows:

[0024] 1) Decompose the rotation matrix of the right image plane relative to the left image plane into two matrices Rl and Rr, which are called the composite rotation matrix of the left and right cameras;

[0025] R l =R 1 / 2

[0026] R r =R -1 / 2

[0027] 2) Rotate the left and right cameras by half so that the optical axes of the left and right cameras are parallel. At this time, the imaging planes of the left and right cameras are parallel, but the baseline is not parallel to the imaging plane;

[0028] 3) Construct the transformation matrix R rect Make the baseline parallel to the imaging plane and transform the pole of the left camera to infinity using the matrix R rect ,as follows:

[0029] R rect =[e1 e2 e3]

[0030]

[0031]

[0032] e3=e1×e2

[0033] 4) The overall rotation matrix of the left and right cameras is obtained by multiplying the synthetic rotation matrix with the transformation matrix. The left and right camera coordinate systems are multiplied by their respective overall rotation matrices to make the principal optical axes of the left and right cameras parallel, and the image plane is parallel to the baseline, thus obtaining an ideal parallel configuration binocular stereo image.

[0034] R′ l =R rect R l , R′ r =R rect R r ,

[0035] After correction, the image needs to be cropped as needed. It is necessary to reselect an image center and an image edge so that the left and right overlapping parts are maximized.

[0036] Furthermore, the three-eye or six-eye stereo depth vision system uses the remaining metamorphic manipulators to form a two-legged or three-legged movement mode; its stereo depth vision calibration method is consistent with the binocular stereo vision calibration method;

[0037] The present invention proposes an environment field acquisition system based on variable arm-borne vision, which can be variable-configured into binocular, trinocular, and hexacular stereoscopic depth vision systems according to the vision system carried by each arm-foot, so as to realize multi-angle surround monitoring and extended monitoring of the search and rescue robot, and perceive the visual semantic information of the surrounding environment. At the same time, the system can be combined with a 360-degree panoramic radar for coarse-fine positioning, and the surrounding scene can be reconstructed in three dimensions through multi-source point cloud information to simulate the environmental status of the rescue area. It is of great help to the rescue robot in establishing a large-scale field of view monitoring area, improve the efficiency of rescue in the accident area, and has high economic value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of an environmental field acquisition system based on allosteric arm-mounted vision according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the robot structure of an environmental field acquisition system based on allosteric arm-mounted vision according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the posture of a robot when constructing a binocular vision system in an environment field acquisition system based on allosteric arm-mounted vision according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the posture of a robot when constructing a three-eye vision system in an environmental field acquisition system based on allosteric arm-mounted vision according to an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the posture of a robot when constructing a six-eye vision system in an environmental field acquisition system based on allosteric arm-mounted vision according to an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the posture of a robot when performing extended detection in the presence of obstacles in an environmental field acquisition system based on allosteric arm-mounted vision as described in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0045] like Figure 1 As shown, an environmental field acquisition system based on allosteric arm-borne vision described in an embodiment of the present invention includes an allosteric manipulator, a robot body, an image acquisition module, a control module and an image processing module. The image acquisition module is an RGB-D camera installed at the end of the allosteric manipulator for real-time acquisition of environmental information; the image processing module processes the image acquired by the image acquisition module and transmits the data information to the control module; the control module sends control instructions to the robot according to the information transmitted by the image processing module to control the movement of the robot.

[0046] like Figure 2 As shown, an environmental field acquisition system based on allosteric arm-mounted vision described in an embodiment of the present invention comprises a robot body and eight allosteric robotic arms, at the ends of which are mounted RGB-D cameras for real-time acquisition of environmental information. The system enables the robot to be transformed into a binocular, trinocular, hexacular or other stereoscopic depth vision system, thereby achieving extended visual detection in narrow gaps and under various occlusions.

[0047] like Figure 3 As shown, when binocular vision is needed to collect image information, the image acquisition modules of any two metamorphic robotic arms are controlled to form binocular vision, and the remaining robotic arms are used to form two-legged, three-legged, and six-legged movement modes. The binocular stereo depth vision calibration steps are as follows:

[0048] Step 1: The robot controls two allosteric manipulators (1-2, 1-3, 1-4, 1-5) and the image acquisition module to form a binocular vision system. After the allosteric manipulators are fixed, the rotation angles and positional relationships of the motors of each allosteric manipulator are known. The positional relationship between the coordinate system of the allosteric manipulator and the world coordinate system is established through the robot forward kinematics solution and expressed through the homogeneous transformation matrix. The formula is as follows:

[0049]

[0050] Among them, R is the rotation matrix, and P is the position in the world coordinate system;

[0051] According to the rotation angle of the allosteric manipulator, the homogeneous transformation matrix of the link coordinate system {i} relative to the coordinate system {i-1} is obtained by combining the DH parameters, and the following homogeneous transformation matrix is ​​obtained:

[0052]

[0053] Step 2: Use the "Zhang calibration method" to calibrate respectively to obtain the internal and external parameters of the allosteric robotic arm image acquisition module. The calibration formula and coordinate system symbols are as follows:

[0054]

[0055] Step 3: Perform image correction on a single camera in each camera group. For radial distortion, the first two terms of the Taylor series expansion around the principal point are used to describe it, expressed as k1 and k2. According to their distribution positions in the radial direction, the adjustment formula is:

[0056] x0=x(1+k1r 2 +k2r 4 )

[0057] y0=y(1+k1r 2 +k2r 4 )

[0058] For tangential distortion, the distortion model can be described by two additional parameters p1 and p2:

[0059] x0=2p1xy+p2(r 2 +2x 2 )+1

[0060] y0=2p2xy+p2(r 2 +2y 2 )+1

[0061] Step 4: When calibrating the binocular stereo vision system, the relative relationship between the left and right camera coordinate systems is calibrated according to Step 1. According to the internal and external parameter separation method, the relative positions R and T of the two allostatic manipulators are obtained through the external parameter relationship. The calibration equation is:

[0062]

[0063] T=T r -RT1

[0064] Where R1, R r , T1, T r They are the extrinsic parameters obtained by single-target calibration of two cameras respectively;

[0065] Step 5: When performing stereo correction, use the Bouguet correction method to decompose the rotation matrix R and translation matrix T into the rotation and translation matrices R1 and R2 of the left and right cameras respectively. r , T1, T r The decomposition principle is that the distortion caused by the reprojection of the left and right images is minimized and the common area of ​​the left and right views is maximized; the specific steps are as follows:

[0066] 1): Decompose the rotation matrix of the right image plane relative to the left image plane into two matrices Rl and Rr, which are called the composite rotation matrix of the left and right cameras;

[0067] Rl =R 1 / 2

[0068] R r =R -1 / 2

[0069] 2): Rotate the left and right cameras by half each, so that the optical axes of the left and right cameras are parallel. At this time, the imaging planes of the left and right cameras are parallel, but the baseline is not parallel to the imaging plane;

[0070] 3): Construct the transformation matrix R rest Make the baseline parallel to the imaging plane and transform the pole of the left camera to infinity using the matrix R rect ,as follows:

[0071] R rect =[e1 e2 e3]

[0072]

[0073]

[0074] e3=e1×e2

[0075] 4): The overall rotation matrix of the left and right cameras is obtained by multiplying the synthetic rotation matrix with the transformation matrix. The left and right camera coordinate systems are multiplied by their respective overall rotation matrices to make the principal optical axes of the left and right cameras parallel, and the image plane is parallel to the baseline, thus obtaining an ideal parallel configuration binocular stereo image.

[0076] R′ l =R rect R l , R′ r =R rect R r ,

[0077] After correction, the image needs to be cropped as needed. It is necessary to reselect an image center and an image edge so that the left and right overlapping parts are maximized.

[0078] like Figure 4 , 5 As shown, when a three-eye or six-eye stereo depth vision system is needed, the remaining metamorphic robotic arms are used to form a two-legged or three-legged movement mode, and its stereo depth vision calibration method is consistent with the binocular stereo vision system calibration method.

[0079] like Figure 6 As shown, when the robot is in a small space and needs to perform obstructed extended vision to collect image information, a three-legged movement method is adopted.

Claims

1. An environmental field acquisition system based on allosteric arm-mounted vision, characterized by: A transformative robotic arm (1), a robot body (2), an image acquisition module (1-2), a control module (2-1) and an image processing module (2-2), wherein the image acquisition module (1-2) is an RGB-D camera installed at the end of the transformative robotic arm and used for real-time acquisition of environmental information; the image processing module (2-2) processes the image acquired by the image acquisition module (1-2) and transmits the data information to the control module (2-1); the control module (2-1) sends a control instruction to the robot according to the information transmitted by the image processing module (2-2) to control the movement of the robot; the system can transform the robot into a binocular, trinocular or hexacular stereoscopic depth vision system according to different needs, so as to realize extended visual detection in narrow gaps and under various occlusions, The binocular stereo depth vision system is characterized in that: when binocular vision is required to collect image information, the image acquisition modules of any two modifiable robotic arms are controlled to form binocular vision, and the remaining robotic arms are used to form a two-legged, three-legged, or six-legged movement mode. The binocular stereo depth vision calibration steps are as follows: Step 1: The robot controls two allosteric manipulators (1-2, 1-3, 1-4, 1-5) and the image acquisition module to form a binocular vision system. After the allosteric manipulators are fixed, the rotation angles and positional relationships of the motors of each allosteric manipulator are known. The positional relationship between the coordinate system of the allosteric manipulator and the world coordinate system is established through the robot forward kinematics solution and expressed through the homogeneous transformation matrix. The formula is as follows: Among them, R is the rotation matrix, and P is the position in the world coordinate system; According to the rotation angle of the allosteric manipulator, the homogeneous transformation matrix of the link coordinate system {i} relative to the coordinate system {i-1} is obtained by combining the DH parameters, and the following homogeneous transformation matrix is ​​obtained: Step 2: Use the "Zhang calibration method" to calibrate respectively to obtain the internal and external parameters of the allosteric robotic arm image acquisition module. The calibration formula and coordinate system symbols are as follows: Step 3: Perform image correction on a single camera in each camera group. For radial distortion, the first two terms of the Taylor series expansion around the principal point are used to describe it, expressed as k1 and k2. According to their distribution positions in the radial direction, the adjustment formula is: x0=x(1+k1r 2 +k2r 4 ) y0=y(1+k1r 2 +k2r 4 ) For tangential distortion, the distortion model can be described by two additional parameters p1 and p2: <h2 style=";text-align:left;direction:ltr">x0 = 2p1xy + p2(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2x<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +1 ) y0=2p2xy+p2(r 2 +2y 2 )+1 Step 4: When calibrating the binocular stereo vision system, the relative relationship between the left and right camera coordinate systems is calibrated according to Step 1. According to the internal and external parameter separation method, the relative positions R and T of the two allostatic manipulators are obtained through the external parameter relationship. The calibration equation is: T=T r -RT1 Where R l ,R r ,T l ,T r They are the extrinsic parameters obtained by single-target calibration of two cameras respectively; Step 5: When performing stereo correction, use the Bouguet correction method to decompose the rotation matrix R and translation matrix T into the rotation and translation matrices R of the left and right cameras respectively. l ,R r ,T l ,T r ,The decomposition principle is that the distortion caused by the reprojection of the left and right images is minimized, and the common area of ​​the left and right views is maximized.

2. According to claim 1, an environmental field acquisition system based on allosteric arm-mounted vision is characterized by: The three-eye or six-eye stereo depth vision system utilizes the remaining transformable robotic arms to form a two-legged or three-legged movement mode; its stereo depth vision calibration method is consistent with the steps of claim 1.