A vision and proximity fusion perception system at the end of a snake-like robotic arm

Abstract

The present invention discloses a vision and proximity fusion perception system for the end of a serpentine robotic arm, including: a perception module, which is used to collect working scene environmental information through a visual sensor and a proximity sensor, and send it to an information processing fusion module; an information processing fusion module, which is used to fuse the environmental information, obtain image information and depth information of the working scene, and send it to a host computer to realize real-time monitoring of the working scene of the serpentine robotic arm. The present invention can meet the visual perception and depth information perception requirements of the end of the serpentine robotic arm, using two monocular cameras to form a binocular camera, and at the same time assisting two multi-region DToF proximity sensors to correct the depth information obtained by the binocular camera. The present invention can also fill the gap in the current market for a dedicated module for environmental perception at the end of a serpentine robotic arm, and has the characteristics of small size, light weight, easy installation, and easy replacement, and can meet the requirements of environmental perception at the end of a variety of serpentine robotic arms.

Description

Technical Field

[0001] The invention belongs to the technical field of continuum robot detection and environment perception, in particular to a vision and proximity sense fusion perception system at the end of a snake-like robot arm. Background Art

[0002] With the development of the robotics industry, various robots have played a key role in various fields of national production. At the same time, as the tasks become increasingly complex and sophisticated, the requirements for robotic arms are also getting higher and higher. Traditional rigid robotic arms are large in size and low in dexterity, making it difficult to meet the requirements of tasks in small spaces and unstructured environments. The snake-shaped robotic arm is small in size and highly flexible, which makes it have incomparable advantages over rigid robotic arms in operating in small spaces and complex environments, and has also become an important direction for the development of robots.

[0003] At present, the research on snake-like robotic arms is booming. Domestic research on snake-like robotic arms is still in the laboratory prototype stage. Most of the research on snake-like robotic arms remains in the stage of design of the robotic arm body and preliminary control and verification of the morphology, while there is little research on the perception of the working environment at the end of the snake-like robotic arm. Considering that the original intention of the research on snake-like robotic arms is to explore unknown, small and complex environments and replace manual work in them, it is required that the end of the snake-like robotic arm can complete the environmental perception of the narrow working scene with limited light, avoid collision between the robotic arm and obstacles, and guide the snake-like robotic arm to the working position. The environmental perception module of the snake-like robotic arm must adopt a lightweight, replaceable, and embedded special module. Considering that the operator needs to intuitively understand the working environment, the perception module should be equipped with an image sensor; considering avoiding collisions at the end of the robotic arm, the perception module should be equipped with a DToF proximity sensor; considering that the working environment at the end of the snake-like robotic arm is not illuminated enough, the perception module should be equipped with a controllable lighting source. The existing environmental perception module cannot meet the above requirements, so a visual and proximity fusion perception system at the end of the snake-like robotic arm is needed to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a vision and proximity perception fusion perception system for the end of a snake-like robotic arm, so as to solve the problem that the existing technology cannot meet the environmental perception requirements of the end of a snake-like robotic arm.

[0005] The technical solution to achieve the purpose of the present invention is: a vision and proximity fusion perception system at the end of a snake-like robotic arm, the system comprising a perception module and an information processing fusion module;

[0006] The perception module is used to collect the working scene environment information through the arranged visual sensors and proximity sensors, and send the environment information to the information processing and fusion module;

[0007] The information processing fusion module is used to fuse environmental information to obtain image information and depth information of the working scene, and send it to the host computer to realize real-time monitoring of the working scene of the snake-like robot arm by the snake-like robot arm monitoring system.

[0008] Furthermore, the perception module includes: an image sensor, a multi-zone DToF proximity sensor, an illumination module, a voltage stabilization module and a level shifter;

[0009] The image sensor is used to obtain visual information from the end of the snake-like robotic arm and transmit the video stream signal to the information processing and fusion module;

[0010] The multi-zone DToF proximity sensor is used to detect target objects in multiple zones and measure distances simultaneously;

[0011] The lighting module is used to provide lighting when the end of the snake-like robotic arm enters a shielded space;

[0012] The voltage stabilizing module is used to generate a 2.8V power supply to power the multi-zone DToF proximity sensor, and to generate a 1.5V power supply in conjunction with the 2.8V power supply to power the image sensor;

[0013] The level shifter is used to match the pin level of the multi-zone DToF proximity sensor with the pin level of the information processing fusion module.

[0014] Further, the information processing fusion module includes a processor, a memory unit, a storage unit, a Gigabit Ethernet transceiver unit and a power management unit;

[0015] The processor is installed with an operating system, and is used to process the two-channel video stream signals, fuse the proximity sensor information and the visual information to obtain the working environment perception information, and communicate with the host computer in real time;

[0016] The memory unit is used to provide the memory required for the operation of the operating system and the memory required for processing image data;

[0017] The storage unit is used to store system data, image data and debugging algorithms. The system data is stored by reading the memory unit. At the same time, when the snake-like robot arm is working, the video signal of the working scene collected by the image sensor is stored.

[0018] The Gigabit Ethernet transceiver unit is used to transmit two video stream signals of the operation scene and the depth map obtained by the multi-zone DToF proximity sensor to the host computer through network communication, so as to realize real-time monitoring of the operation scene of the snake-like robot arm;

[0019] The power management unit is used to supply power to other parts of the information processing fusion module.

[0020] Furthermore, the number of the image sensors is two, and the OV5640 image sensor is used, and the processor is i.MX 8M Plus; the XCLK port of the OV5640 image sensor is connected to an active 24M crystal oscillator as the system clock of the image sensor, the STROBE port is connected to the gate of the field effect tube to control the on and off of the field effect tube to control the lighting module, the PWDN port is effective at a high level and is used to control power off, and the RESET port is effective at a low level and is used to reset the sensor, and the two interfaces are finally connected to the GPIO port of the processor through a socket strip; the SIO_C and SIO_D ports are the SCCB control interface clock line SIOC and data line SIOD, and are connected to the clock line and data line of the I2C1 of the processor through a socket strip;

[0021] The Y5, Y4, Y8 and Y9 ports are the differential signal lines of the OV5640 image sensor CSI interface, which are connected to the differential signal lines of the processor MIPI_CSI1_D0 and MIPI_CSI1_D1; the Y6 and Y7 ports are the differential clock lines of the OV5640 image sensor CSI interface, which are connected to the differential clock line of the processor MIPI_CSI1_CLK through the socket strip; the AVDD port, DOVDD port, and AF-VCC port are powered by 2.8V, and the DVDD port is powered by 1.5V.

[0022] Furthermore, the number of the multi-zone DToF proximity sensors is two, using the VL53L5CX sensor, the AVDD and IOVDD ports are both powered by 2.8V, and a filter capacitor is set next to the power supply pin; the SPI_I2C_N port, the GPI01 port and the LP port are respectively the I2C communication reset, interrupt output and I2C control ports in LP mode of the VL53L5CX, and the three ports are pulled up to 2.8V through a 47KΩ pull-up resistor, and the SCL and SDA ports are pulled up to 2.8V through a 2.2KΩ pull-up resistor; the four signals of the LP port, the SCL port, the SDA port and the SPI_I2C_N port are adjusted to 3.3V through a level shifter and then connected to the corresponding pins of the processor through a socket strip. The SDA port and the SCL port are the data line and clock line of the I2C interface of the VL53L5CX. The data line and clock line of the I2C3 of the processor are connected through the socket strip to realize data transmission and communication between the i.MX 8M Plus processor and the VL53L5CX.

[0023] Furthermore, the voltage stabilizing module includes a TLV76728 chip and an XC6206P152 chip. The input port 6 and the enable port 4 of the TLV76728 chip input 3.3V, and through the conversion output, the output voltage is connected to the output voltage reference port 2 through a series 50Ω current limiting resistor, so that the output port 1 is stabilized at a voltage of 2.8V, and then a stable voltage of 1.5V is obtained through the XC6206P152 chip.

[0024] Furthermore, the storage unit adopts the eMMC storage chip KLMAG2WEMB, the CMD port of the chip is connected to the SD1_CMD port of the processor, for receiving commands from the processor and sending requests to the processor; the CLK synchronization clock port is connected to the SD1_CLK port of the processor, for providing a clock signal for data reading; the DATA0~DATA7 data signal port is connected to the SD1_DATA0~SD1_DATA7 port of the processor, for realizing data transmission between the storage chip and the processor, and realizing data access.

[0025] Furthermore, the memory unit adopts LPDDR4 low-power memory MT53D512M32D2DS-053; the power management unit adopts PCA9450CHN chip.

[0026] Furthermore, the Gigabit Ethernet transceiver unit adopts the YT8511C transceiver, the YT8511C transceiver and the Gigabit network communication interface are connected through four combined differential signal lines RJ45_D1~RJ45_D4, the four sending ports RXD0~RXD3 of the YT8511 transceiver are connected to the network signal receiving ports ENET_RD0~ENET_RD3 of the processor; the YT8511 transceiver signal receiving port TXD0~TXD3 is connected to the network signal sending port ENET_TD0~ENET_TD4 of the processor; the receiving signal clock port RX_CLK and the sending signal clock port of the YT5811 are connected to the ENET_TXC and ENET_RXC ports of the processor; the receiving completion port RX_DV and the sending enable port TX_EN of the YT8511 are connected to the ENET_RX_CTL and ENET_TX_CTL ports of the processor; XTL_IN and XTL_OUT are externally connected to a passive 25MHz crystal oscillator.

[0027] Compared with the prior art, the present invention has the following significant advantages:

[0028] (1) The present invention integrates an advanced high-precision multi-area DToF proximity sensor with a binocular vision camera, and can then correct the depth map obtained by the binocular camera to obtain more accurate three-dimensional information perception of the working environment.

[0029] (2) The high-precision multi-area DToF proximity sensor used in the present invention can achieve optimal ranging in the range of 0.02m to 4m under various lighting conditions, overcoming the problem that when the end of the snake-like robot arm approaches the work target or there are obstacles, the image information extracted by the camera vision cannot perceive the depth due to occlusion or when the distance is too close.

[0030] (3) The present invention uses a high-performance application processor that integrates 4 Cortex-A53 processors and 1 Cortex-M7 processor, which can deploy machine learning algorithms and complete visual computing tasks. It also has two camera interfaces and a variety of peripheral interfaces, which fully meet the deployment of the snake-like robot arm end perception solution.

[0031] (4) In addition to being installed at the end of a serpentine robotic arm to realize fusion perception of the working environment, the present invention can also be applied to the end of a traditional rigid robotic arm to realize fusion perception of the working environment, and has a wide range of applications.

[0032] (5) The entire perception module is integrated on the PCB assembly, ensuring its smaller size and lower cost. It is mainly composed of two PCB assemblies. The PCB assembly of the perception part at the end of the snake robot arm adopts a runway-shaped design. While ensuring the longest baseline distance of the binocular camera, the entire device is kept small, which is very suitable for installation at the end of the snake robot arm to enter a small space for operation. The square PCB of the information processing part is easier to match with the circular cross-section of the snake robot arm. Both are easy to install and can be debugged and replaced separately.

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a block diagram of the visual and proximity fusion perception module at the end of the snake-like robot arm.

[0035] Figure 2 This is the three-dimensional structural diagram of the front side of the PCB assembly that the perception module at the end of the snake-like robot arm perceives.

[0036] Figure 3 This is the 3D structural diagram of the back side of the PCB assembly of the perception part of the perception module at the end of the snake-like robot arm.

[0037] Figure 4 This is the three-dimensional structural diagram of the front side of the PCB assembly of the information processing part of the perception module at the end of the snake-like robot arm.

[0038] Figure 5 This is the three-dimensional structural diagram of the back side of the PCB assembly of the information processing part of the perception module at the end of the snake-like robot arm.

[0039] Figure 6This is the flow chart of the fusion of vision and proximity perception of the end perception module of the snake robot arm.

[0040] Figure 7 This is the circuit connection diagram of the image sensor OV5640 of the perception part of the PCB component of the perception module at the end of the snake-like robot arm.

[0041] Figure 8 This is the LED circuit connection diagram of the PCB component of the sensing part of the snake-like robot arm's end sensing module.

[0042] Fig. 9 This is the circuit connection diagram of the VL53L5CX PCB component of the sensing part of the snake-like robot arm's end sensing module.

[0043] Fig.10 This is the circuit connection diagram of the voltage regulator chip of the PCB component of the perception part of the perception module at the end of the snake-like robot arm.

[0044] Fig.11 This is the circuit connection diagram of the eMMC storage chip of the PCB assembly of the information processing part of the snake-like robot arm's end perception module.

[0045] Fig.12 This is the Ethernet communication circuit connection diagram for the PCB assembly of the information processing part of the snake-like robot arm's end perception module.

[0046] Fig.13 Assembly diagram of the vision and proximity fusion perception module at the end of the snake-like robotic arm. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The present invention provides a vision and proximity fusion perception system at the end of a snake-like robotic arm, the system comprising a perception module and an information processing fusion module;

[0049] The perception module is used to collect the working scene environment information through the arranged visual sensors and proximity sensors, and send the environment information to the information processing and fusion module;

[0050] The information processing fusion module is used to fuse environmental information to obtain image information and depth information of the working scene, and send it to the host computer to realize real-time monitoring of the working scene of the snake-like robot arm by the snake-like robot arm monitoring system.

[0051] The sensing module includes: an image sensor, a multi-zone DToF proximity sensor, an illumination module, a voltage stabilization module and a level shifter;

[0052] The image sensor is used to obtain visual information from the end of the snake-like robotic arm and transmit the video stream signal to the information processing and fusion module;

[0053] The multi-zone DToF proximity sensor is used to detect target objects in multiple zones and measure distances simultaneously;

[0054] The lighting module is used to provide lighting when the end of the snake-like robotic arm enters a shielded space;

[0055] The voltage stabilizing module is used to generate a 2.8V power supply to power the multi-zone DToF proximity sensor, and to generate a 1.5V power supply in conjunction with the 2.8V power supply to power the image sensor;

[0056] The level shifter is used to match the pin level of the multi-zone DToF proximity sensor with the pin level of the information processing fusion module.

[0057] The information processing fusion module includes a processor, a memory unit, a storage unit, a Gigabit Ethernet transceiver unit and a power management unit;

[0058] The processor is installed with an operating system, and is used to process the two-channel video stream signals, fuse the proximity sensor information and the visual information to obtain the working environment perception information, and communicate with the host computer in real time;

[0059] The memory unit is used to provide the memory required for the operation of the operating system and the memory required for processing image data;

[0060] The storage unit is used to store system data, image data and debugging algorithms. The system data is stored by reading the memory unit. At the same time, when the snake-like robot arm is working, the video signal of the working scene collected by the image sensor is stored.

[0061] The Gigabit Ethernet transceiver unit is used to transmit two video stream signals of the operation scene and the depth map obtained by the multi-zone DToF proximity sensor to the host computer through network communication, so as to realize real-time monitoring of the operation scene of the snake-like robot arm;

[0062] The power management unit is used to supply power to other parts of the information processing fusion module.

[0063] The present invention is further described in detail below with reference to specific embodiments.

[0064] Combination Figure 1The present invention provides a vision and proximity fusion perception system at the end of a serpentine robot arm, which is mainly composed of a vision and proximity perception PCB component and an information processing fusion PCB component. The two are connected by two socket strips with a 2.54mm pitch and fixed by copper pillars. They are easy to disassemble and assemble, and can be debugged or replaced and upgraded separately. The perception part PCB component completes the collection of environmental information through the arranged visual sensors and proximity sensors, and then uses the MIPI_CSI camera interface and I2C bus to send the information to the information processing part PCB component, and realizes the fusion of environmental information in the processor to obtain the image information and depth information of the working scene, and sends it to the host computer through Ethernet communication to realize the real-time monitoring of the working scene of the serpentine robot arm by the serpentine robot arm monitoring system.

[0065] The sensing part of the PCB assembly adopts a runway-shaped design, with a horizontal length of 90mm and a vertical length of 30mm. Two image sensors, two DToF proximity sensors, a level shifter, a voltage regulator chip, six LED lamp beads and two 18-pin socket strips are arranged. The two image sensors are arranged horizontally on both sides of the center line of the PCB assembly to form a binocular camera for collecting image information in the field of view of the end of the robot arm. The baseline distance of the binocular camera is 80mm. At the same time, a lens mount with a hole spacing of 20mm is installed on the image sensor, and a fixed-focus lens with a focal length of 4mm is used; the two DToF proximity sensors are arranged at the upper left and upper right corners of the PCB, the distance between the center points of the two sensors is 60mm, and the distance between the center of the sensor and the horizontal center line of the entire PCB assembly is 10mm, which can collect multi-target depth information within a 45°×45° field of view; the LED lamp beads are used for lighting when the snake-like robot enters a dark environment, providing conditions for the normal operation of the image sensor; the voltage regulator chip uses TLV76728DRVR to generate a 2.8V power supply to power the multi-zone DToF proximity sensor of the sensing part PCB assembly, and uses the XC6206P152MR low-voltage difference voltage regulator chip to generate a 1.5V power supply to cooperate with the 2.8V power supply to power the image sensor. The level shifter uses TXS0104ERGYR to match the pin level of the DToF proximity sensor with the pin level of the processor.

[0066] The information processing PCB component adopts a square design with a side length of 60mm. It is equipped with a high-performance application processor, LPDDR4 low-power memory, eMMC storage chip, Gigabit Ethernet transceiver and dedicated power management chip. The high-performance application processor needs to process two video stream signals and can install the corresponding Linux and Android operating systems. It supports the deployment of machine learning related algorithms to complete image processing, and has high-speed communication interfaces such as Gigabit Ethernet port, high-speed USB and CAN bus; the eMMC storage chip realizes the storage of system data, key image data and debugging algorithms, uses the standard MMC protocol to read the memory, can store system data, and can store the spatial video signal of the operating scene collected by the camera when the snake-like robot arm is working; the LPDDR4 low-power memory uses Micron Technology's 32-bit 16GB memory MT53D512M32D2DS-053, which ensures the memory requirements of the operating system while ensuring the memory requirements of the image data processing; the Gigabit Ethernet port transmits the two-way video stream signal and depth map of the operating scene to the host computer through network communication, realizing real-time monitoring of the snake-like robot arm operating scene; the Type_C interface ensures the system burning and algorithm code debugging functions.

[0067] The fusion perception method of the vision and proximity sense fusion perception system of the end of the snake-like robot arm of the present invention for the fusion perception of space is as follows Figure 6 As shown, the left and right cameras arranged at the end of the snake-like robot arm obtain image data. The left and right cameras are arranged on the same horizontal plane and use fixed-focus lenses with the same focal length. Therefore, the left and right cameras do not need to perform epipolar correction to obtain the left and right images. The image preprocessing obtains the grayscale image, calculates the absolute error and cost space, and obtains the filtered cost space after cost aggregation. The disparity map is obtained by selecting the minimum cost, and the left and right consistency detection is performed to eliminate singular data. At the same time, the low-resolution depth maps obtained by the left and right DToF proximity sensors are spliced, and the disparity map is inferred according to the triangulation of the binocular camera. The disparity map obtained by the binocular camera is then filled. Finally, the disparity map of the working scene at the end of the robot arm is obtained by using bilateral filtering, and then the depth map of the entire working scene is calculated.

[0068] Combination Figure 2 and Figure 3 The sensing part of the terminal sensing module of this embodiment adopts a runway-shaped design with a horizontal length of 90 mm and a vertical length of 30 mm. Two OV5640 image sensors 1 and 8, multi-area DToF proximity sensors 3 and 6, a voltage regulator chip 5, high-brightness LED patch lamp beads 2, 7, 9, 10, 14, 15, two in-line plug-in bars 4 and 12, and M3 copper pillars 11 and 13 are soldered on the front.

[0069] The image sensors 1 and 8 use the 1 / 4 inch CMOS QSXGA (2592*1944) image sensor OV5640 produced by OmniVision. The sensor is small in size and low in operating voltage, and provides all the functions of a single-chip QSXGA camera and image processor. Through SCCB bus control, various resolution 8 / 10-bit image data in the form of full frame, sub-sampling, scaling and windowing can be output. The QSXGA image of this product reaches up to 15 frames per second (1080P images can reach 30 frames, 720P images can reach 60 frames, and QVGA resolution can reach 120 frames). Users can fully control the image quality, data format and transmission method. All image processing functions including gamma curve, white balance, contrast, chromaticity, etc. can be programmed through the SCCB interface, supporting digital video DVP and MIPI interfaces. The MIPI interface has the characteristics of less wiring, fast speed and low power consumption, so the MIPI interface is used to transmit image sensor data. The image sensor uses the MIPI_CSI camera high-speed serial interface to connect to the socket strip, and the socket strip connects the image sensor to the MIPI interface of the i.MX 8M Plus processing chip through the socket with the mother socket. The two image sensors are arranged horizontally on both sides of the center line of the PCB circuit board to form a binocular camera with a baseline distance of 80mm. The image sensors 1 and 8 are symmetrically designed. For the sake of simplicity, only a single image sensor circuit connection diagram is given. Figure 7 The XCLK port of the image sensor 1 is connected to an active 24M crystal oscillator as the system clock of the image sensor, the STROBE port is connected to the gate to control the on and off of the field effect tube to control the LED light, the PWDN port is valid at a high level to control the power off, and the RESET port is valid at a low level to reset the sensor. The two interfaces are finally connected to the processor GPIO port through a socket strip; the SIO_C and SIO_D ports are the SCCB control interface clock line SIOC and data line SIOD, which are connected to the clock line and data line of the I2C1 of the processor through a socket strip; the Y5, Y4, Y8 and Y9 ports are the differential signal lines of the CSI interface of the OV5640 image sensor connected to the differential signal lines of the processor MIPI_CSI1_D0 and MIPI_CSI1_D1; the Y6 and Y7 ports are the differential clock lines of the CSI interface of the OV5640 image sensor connected to the differential clock lines of the processor MIPI_CSI1_CLK through a socket strip, and the order of connecting the socket strips is not described in detail here.

[0070] The multi-zone DToF proximity sensors 3 and 6 are VL53L5CX sensors. Due to the integration of single-photon avalanche diode receiving arrays (SPADs), physical infrared filters and diffraction optical components, they can provide 8×8 and up to 64 area target detection and distance measurement with a refresh rate of 15Hz within 4m under various lighting environments. The two multi-zone DToF proximity sensors are arranged at the upper left and upper right corners of the PCB. The distance between the center points of the two sensors is 60mm, and the distance between the center of the sensor and the horizontal center line of the entire PCB assembly is 10mm. It can collect multi-target depth information within a 45°×45° field of view, and can realize spatial perception and obstacle detection at the end of the snake-like robotic arm. The multi-zone DToF proximity sensors 3 and 6 adopt a symmetrical design, and only a single image sensor circuit connection is given for simple explanation. Combined with Fig. 9 , AVDD and IOVDD ports are both powered by 2.8V, and filter capacitors are placed next to the power supply pins. A1, A3, and A5 are the I2C communication reset, interrupt output, and I2C control ports in LP mode of LV53L5CX, respectively. The three ports are pulled up to 2.8V through 47KΩ pull-up resistors, and the two ports of the I2C interface are pulled up to 2.8V through 2.2KΩ pull-up resistors. To ensure the quality of the communication signal, the LP, SCL, SDA, and I2C_RST signals are adjusted to 3.3V through a level shifter, and then connected to the corresponding pins of the processor through a socket. The C3 and C4 ports are the data and clock lines of the I2C interface of VL53L5CX. The data and clock lines of the processor's I2C3 are connected through the socket to realize data transmission and communication between the i.MX 8M Plus processor and the VL53L5CX. The order of connecting the sockets is not detailed here.

[0071] The voltage stabilizing module 5 comprises two parts, Fig.10 , 3.3V voltage is input through TLV76728 low-dropout regulator input port 6 and enable port 4, and the output is converted. The output voltage is connected to the output voltage reference port 2 through a series 50Ω current-limiting resistor to make the output port 1 stabilized at 2.8V, and then a stable 1.5V voltage is obtained through XC6206P152 to power the sensing part PCB components.

[0072] Combination Figure 4 The information processing part of the terminal perception module of this embodiment adopts a square design with a length and width of 60 mm, ensuring a small overall size and suitable for installation at the end of the snake-like robot arm. Figure 4 , solder the eMMC storage chip 1-1, i.MX 8M Plus system-level processor 3-1, LPDDR4 low-power memory 4-1, power management IC 5-1 and two rows of female headers 2-1 and 6-1 for connecting the sensing part PCB on the front. Figure 5The information processing part has a Gigabit network receiving chip 8-1 soldered on the back, and a Type-C interface 7-1 and a Gigabit network interface 9-1 for communication and data transmission are placed at the same time.

[0073] The system-level processor 3-1 is NXP's i.MX 8M Plus, which integrates 4 Cortex-A53 and 1 Cortex-M7 processors. Its neural processing unit (NPU) provides 2.3TOPS computing power, which can complete machine learning tasks. It also has two MIPI_CSI camera interfaces, which can process two video signals. In addition, it can also realize real-time communication with the host computer through a gigabit network port, a high-speed USB or a CAN bus. As the core of the information processing part, it collects image information through the MIPI_CSI interface, processes binocular vision information, calculates disparity, and receives two multi-zone DToF proximity sensor information through the I2C bus at the same time, and fuses the proximity sensor information with the visual information to obtain accurate operating environment perception information. At the same time, through gigabit network communication, the video stream signal and depth information are fed back to the host computer, so that the operator of the snake-shaped robot arm can more intuitively understand the operating scene information and operate the robot arm at the same time.

[0074] The eMMC storage chip 1-1 can store system data, key image data and debugging algorithms. It uses Samsung's 16GB eMMC storage chip KLMAG2WEMB, adopts FBGA packaging, uses the standard MMC protocol to read the memory, and stores system data. It can store the video signal of the operation scene collected by the camera while the snake-like robot arm is working. Fig.11 The circuit connection of eMMC storage is specifically as follows: M5 is the CMD port of the storage chip, connected to the SD1_CMD port of the processor, for receiving processor commands and sending requests to the processor; M6 is the CLK synchronization clock port of the storage chip, connected to the SD1_CLK port of the processor, for providing a clock signal for data reading; DATA0~DATA7 are the data signal ports of the storage chip, connected to the SD1_DATA0~SD1_DATA7 ports of the processor, for realizing data transmission between the storage chip and the processor, and realizing data access.

[0075] The LPDDR4 low-power memory 4-1 uses Micron Technology's 32-bit 16GB memory MT53D512M32D2DS-053, which meets the memory requirements of real-time processing of two-way images while ensuring the memory requirements of the operating system. Its circuit connection diagram can be connected according to the DDR pin function list in the i.MX 8M Plus operation manual, and will not be described in detail here. The power management IC5-1 is PCA9450CHN produced by NXP. PCA9450CHN is a single-chip power management IC (PMIC) specially designed to support the i.MX 8M Plus processor, providing 6 high-efficiency buck regulators, 5 low-dropout linear regulators, and a 400mA load switch, which can meet the power supply of on-chip resources such as i.MX 8M Plus, LPDDR4 and eMMC memory.

[0076] The gigabit network receiving chip 8-1 is a MotorComm YT8511C three-speed 10 / 100 / 1000BASE_T gigabit Ethernet physical layer (PHY) transceiver, which is compatible with the 2.5v / 3.3v RGMII interface and supports 100-meter data transmission (4-wire). By using the gigabit network port to communicate through the network, the two-way video stream signal and depth map of the operation scene can be transmitted to the host computer, realizing real-time monitoring of the snake robot arm operation scene. Fig.12 , is the circuit connection diagram of YT8511, the Ethernet physical layer transceiver and the Gigabit network communication interface are connected through four combined differential signal lines RJ45_D1~RJ45_D4, 29, 28, 26 and 25 are the YT8511 transceiver signal sending ports, which are respectively connected to the processor's network signal receiving ports ENET_RD0~ENET_RD3; 34, 35, 36 and 37 are the YT8511 transceiver signal receiving ports, which are respectively connected to the processor's network signal sending ports ENET_TD0~ENET_TD4; 31 and 33 are the YT5811 receiving signal clock port and the sending signal clock port, which are respectively connected to the processor's ENET_TXC and ENET_RXC ports; 30 and 32 are the YT8511 receiving completion and sending enable ports, which are respectively connected to the processor's ENET_RX_CTL and ENET_TX_CTL ports; 4 and 5 are external passive 25MHz crystal oscillators.

[0077] The vision and proximity fusion perception system of the end of the serpentine robot arm of the present invention has a measurement range of 0.02m to 4m. Therefore, the fusion perception method of space is to use the parallax method when the viewing distance of the end of the serpentine robot arm is 10m to 4m to obtain the rough depth information of the scene; when the viewing distance is 4m to 0.5m, the DToF proximity sensor is used to obtain the depth information of 64 areas of 8×8, and the depth map calculated by the binocular camera is corrected to obtain more accurate environmental information; when approaching the operation point, due to limited vision, it is no longer suitable to use visual information, and the DToF proximity sensor can be used completely. At the same time, the minimum safety distance is set to 0.1m. When the distance between the end of the serpentine robot arm and the obstacle or the work object is less than 0.1m, the serpentine robot arm will stop and prompt the operator to perform manual operation to ensure that the end of the serpentine robot arm will not collide with the work object and obstacles, thereby ensuring safety.

[0078] The specific fusion method of vision and proximity perception is as follows Figure 6 As shown, the left and right cameras arranged at the end of the snake-like robot arm obtain image data. At the same time, the left and right cameras are arranged on the same horizontal plane and use fixed-focus lenses with the same focal length. Therefore, the left and right cameras do not need to perform epipolar correction to obtain the left and right images. The image preprocessing obtains the grayscale image, calculates the absolute error and cost space, and obtains the filtered cost space after cost aggregation. The disparity map is obtained by selecting the minimum cost, and the left and right consistency detection is performed to eliminate singular data. At the same time, the low-resolution depth maps obtained by the left and right DToF proximity sensors are spliced, and the disparity map is inferred according to the triangulation of the binocular camera. The disparity map obtained by the binocular camera is then filled. Finally, the disparity map of the working scene at the end of the robot arm is obtained by using bilateral filtering, and then the depth map of the entire working scene is calculated.

[0079] Combination Fig.13 The specific installation method of the fusion perception system of vision and proximity sense at the end of the serpentine robot arm is as follows: Currently, for a rope-driven serpentine robot arm, the rope-driven continuum robot arm consists of a drive box 1 and a serpentine robot arm body 2. The motor in the drive box controls the serpentine robot arm body by retracting and releasing the rope. For this serpentine robot arm, the specific installation method is as follows: first install the information processing part PCB assembly at the end of the serpentine robot arm, and then install the perception part PCB assembly through the connector and copper column. The information processing part PCB assembly is connected to the host computer through the network cable and Type_C interface. The perception part PCB assembly collects sensor data and sends it to the information processing part PCB assembly through the socket strip. The information processing part PCB sends the fused environmental information to the host computer through the network cable to realize the fusion perception of the environment at the end of the serpentine robot arm.

[0080] The present invention can solve the current market gap in a dedicated module for environmental perception at the end of a serpentine robotic arm, and has the characteristics of small size, light weight, easy installation, and easy replacement, and can meet various requirements for environmental perception at the end of a serpentine robotic arm.

[0081] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A vision and proximity fusion perception system at the end of a snake-like robotic arm, characterized in that: The system includes a perception module and an information processing fusion module; The perception module is used to collect the working scene environment information through the arranged visual sensors and proximity sensors, and send the environment information to the information processing and fusion module; The information processing fusion module is used to fuse the environmental information to obtain the image information and depth information of the working scene, and send it to the host computer to realize the real-time monitoring of the working scene of the snake-shaped robot arm by the snake-shaped robot arm monitoring system; The sensing module includes: an image sensor, a multi-zone DToF proximity sensor, an illumination module, a voltage stabilization module and a level shifter; The image sensor is used to obtain visual information from the end of the snake-like robotic arm and transmit the video stream signal to the information processing and fusion module; The multi-zone DToF proximity sensor is used to detect target objects in multiple zones and measure distances simultaneously; The lighting module is used to provide lighting when the end of the snake-like robotic arm enters a shielded space; The voltage stabilizing module is used to generate a 2.8V power supply to power the multi-zone DToF proximity sensor, and to generate a 1.5V power supply in conjunction with the 2.8V power supply to power the image sensor; The level shifter is used to match the pin level of the multi-zone DToF proximity sensor with the pin level of the information processing fusion module; The information processing fusion module includes a processor, a memory unit, a storage unit, a Gigabit Ethernet transceiver unit and a power management unit; The processor is installed with an operating system, and is used to process the two-channel video stream signals, fuse the proximity sensor information and the visual information to obtain the working environment perception information, and communicate with the host computer in real time; The memory unit is used to provide the memory required for the operation of the operating system and the memory required for processing image data; The storage unit is used to store system data, image data and debugging algorithms. The system data is stored by reading the memory unit. At the same time, when the snake-like robot arm is working, the video signal of the working scene collected by the image sensor is stored. The Gigabit Ethernet transceiver unit is used to transmit two video stream signals of the operation scene and the depth map obtained by the multi-zone DToF proximity sensor to the host computer through network communication, so as to realize real-time monitoring of the operation scene of the snake-like robot arm; The power management unit is used to supply power to other parts of the information processing fusion module; The system consists of two parts: a visual and proximity sensing PCB component and an information processing fusion PCB component. The two parts are connected by two socket strips with a 2.54mm pitch and fixed by copper pillars. The sensing part PCB component collects environmental information through the arranged visual sensors and proximity sensors, and then uses the MIPI_CSI camera interface and the I2C bus to send the information to the information processing part PCB component, and realizes the fusion of environmental information in the processor to obtain the image information and depth information of the working scene, and sends it to the host computer through Ethernet communication to realize the real-time monitoring of the working scene of the snake-like robot arm by the snake-like robot arm monitoring system. The sensing part of the PCB component adopts a runway-shaped design, with a horizontal length of 90mm and a vertical length of 30mm; The PCB components of the information processing part adopt a square design with a side length of 60mm.

2. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 1 is characterized in that: The number of the image sensors is two, and the OV5640 image sensor is used. The processor is i.MX 8M Plus; the XCLK port of the OV5640 image sensor is connected to an active 24M crystal oscillator as the system clock of the image sensor, the STROBE port is connected to the gate of the field effect tube to control the on and off of the field effect tube to control the lighting module, the PWDN port is valid at a high level and is used to control the power off, and the RESET port is valid at a low level and is used to reset the sensor. The two interfaces are finally connected to the GPIO port of the processor through a socket strip; the SIO_C and SIO_D ports are the SCCB control interface clock line SIOC and data line SIOD, and are connected to the I 2C1's clock line and data line; Y5, Y4, Y8 and Y9 ports are the differential signal lines of the OV5640 image sensor CSI interface, connecting the differential signal lines of the processor MIPI_CSI1_D0 and MIPI_CSI1_D1; Y6 and Y7 ports are the differential clock lines of the OV5640 image sensor CSI interface, connected to the differential clock line of the processor MIPI_CSI1_CLK through the socket strip; AVDD port, DOVDD port, AF-VCC port are powered by 2.8V, and DVDD port is powered by 1.5V.

3. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 2 is characterized in that: There are two multi-zone DToF proximity sensors, which use VL53L5CX sensors. Both AVDD and IOVDD ports are powered by 2.8V, and a filter capacitor is set next to the power supply pin. The SPI_I2C_N port, GPI01 port and LP port are respectively the I2C communication reset, interrupt output and I2C control port in LP mode of VL53L5CX, and the three ports are pulled up to 2.8V through 47KΩ pull-up resistors, and the SCL and SDA ports are pulled up to 2.8V through 2.2KΩ pull-up resistors. The four signals of the LP port, SCL port, SDA port and SPI_I2C_N port are adjusted to 3.3V through a level shifter and then connected to the corresponding pins of the processor through a socket strip. The SDA port and SCL port are the data line and clock line of the I2C interface of the VL53L5CX. The data line and clock line of the I2C3 of the processor are connected through the socket strip to realize data transmission and communication between the i.MX 8M Plus processor and the VL53L5CX.

4. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 3 is characterized in that: The voltage stabilizing module includes a TLV76728 chip and an XC6206P152 chip. The input port 6 and the enable port 4 of the TLV76728 chip input 3.3V, and the output is converted. The output voltage is connected to the output voltage reference port 2 through a series 50Ω current limiting resistor, so that the output port 1 is stabilized at 2.8V, and then a stable 1.5V voltage is obtained through the XC6206P152 chip.

5. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 1 is characterized in that: The storage unit adopts the eMMC storage chip KLMAG2WEMB, the CMD port of the chip is connected to the SD1_CMD port of the processor, and is used to receive commands from the processor and send requests to the processor; the CLK synchronization clock port is connected to the SD1_CLK port of the processor, and is used to provide a clock signal for data reading; the DATA0~DATA7 data signal ports are connected to the SD1_DATA0~SD1_DATA7 ports of the processor, and are used to realize data transmission between the storage chip and the processor, and realize data access.

6. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 5, characterized in that: The memory unit adopts LPDDR4 low-power memory MT53D512M32D2DS-053; the power management unit adopts PCA9450CHN chip.

7. The vision and proximity fusion perception system of the end of the snake-like robot arm according to claim 6 is characterized in that: The Gigabit Ethernet transceiver unit adopts the YT8511C transceiver, and the YT8511C transceiver and the Gigabit network communication interface are connected through four combined differential signal lines RJ45_D1~RJ45_D4. The four sending ports RXD0~RXD3 of the YT8511 transceiver are connected to the network signal receiving ports ENET_RD0~ENET_RD3 of the processor; the signal receiving ports TXD0~TXD3 of the YT8511 transceiver are connected to the network signal sending ports ENET_TD0~ENET_TD4 of the processor; the receiving signal clock port RX_CLK and the sending signal clock port of the YT5811 are connected to the ENET_TXC and ENET_RXC ports of the processor; the receiving completion port RX_DV and the sending enable port TX_EN of the YT8511 are connected to the ENET_RX_CTL and ENET_TX_CTL ports of the processor; XTL_IN and XTL_OUT are externally connected to a passive 25MHz crystal oscillator.

Images