A working method and device based on a multi-modal double-arm building inspection robot

Abstract

The application discloses a kind of based on multi-modal double-arm building inspection robot operation method and device, the inspection robot has double operation mechanical arm, and each operation mechanical arm end integrates end effector and wrist high-definition camera, the method includes: obtaining the preset inspection task path of at least one inspection point of containing multi-storey building, control robot along the preset path executes inspection;Multi-modal environment signal is collected during inspection process, and safety hazard detection is carried out;After reaching specified inspection point, the opening of target equipment door is completed by double operation mechanical arm, internal inspection and reset operation, realize the fire-fighting equipment inspection of high-rise office building, commercial complex, scientific research building, industrial park and the like scene, environmental safety monitoring, equipment state diagnosis, non-contact electric leakage hazard screening and the like all-weather unmanned safety inspection operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a working method and apparatus for a multimodal dual-arm building inspection robot. Background Technology

[0002] With the accelerating pace of urbanization, the number of multi-story buildings such as high-rise buildings, commercial complexes, research and office buildings, and industrial parks continues to grow. Building fire safety, electrical safety, and equipment operation safety have become core aspects of daily management. Traditional building inspections mainly rely on manual labor, which generally suffers from prominent problems such as numerous blind spots, high risks in hazardous areas, insufficient nighttime inspection capabilities, non-standard data recording, delayed discovery of hidden dangers, and a lack of professional testing capabilities. These issues can no longer meet the actual needs of modern building safety management.

[0003] In recent years, intelligent inspection robot technology has been gradually applied to the field of building security. However, most existing wheeled inspection robots adopt armless or single-arm structures, lacking the ability of dual-arm collaboration and force-controlled compliant operation. They cannot autonomously complete physical interaction tasks such as opening fire hydrant cabinet doors, operating fire doors, and inspecting equipment inside cabinets, and can only achieve external visual monitoring. At the same time, robots generally lack non-contact electric field detection modules, making it impossible to identify potential leakage hazards in distribution boxes, power lines, sockets, and equipment casings without contact, damage, or distance, resulting in significant blind spots in electrical safety monitoring. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a working method and apparatus based on a multimodal dual-arm building inspection robot.

[0005] On the one hand, this embodiment provides a working method for a multimodal dual-arm building inspection robot, including: Obtain a preset inspection task path that includes at least one inspection point in a multi-story building, and control the robot to perform inspections along the preset path; During the inspection, multimodal environmental signals are collected, and safety hazard detection is performed. After reaching the designated inspection point, acquire the target door image and extract its pose and contour features; The pose and contour features are input into the dual-arm force-position hybrid control model to generate a robotic arm control signal that includes the end gripper joint angle and force control quantity. The target motion trajectory is generated based on the robotic arm control signal. The data is sent to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

[0006] Furthermore, during the inspection process, multimodal environmental signals are collected, and safety hazard detection is performed, including: During the inspection process, the inspection robot uses its configured multimodal perception module to simultaneously collect five types of multimodal environmental data: visible light image data, infrared thermal image data, gas concentration data, electric field signal data, and audio feature data. The collected data from the five types of multimodal environmental data are then transmitted to the edge computing module. The edge computing module uses DS evidence theory to perform decision-level fusion processing, enabling comprehensive judgment of various types of safety hazards such as fire, smoke, gas leaks, equipment overheating, abnormal pipe noises, and electrical casing leakage.

[0007] Furthermore, the method for edge computing modules to detect potential safety hazards related to leakage current in electrical enclosures includes: The inspection robot collects the ambient electric field at a distance from the equipment and sends it to the edge computing module. The edge computing module filters and eliminates power frequency interference and electromagnetic noise to obtain the background electric field. When the inspection robot moves to the device under test, the chassis stops, the electric field probe is aligned with the detection area, and the robot scans the surface of the device at a constant speed to continuously collect electric field signals from multiple points. The signals are then sent to the edge computing module, which performs signal noise reduction and distortion rate calculation. Combined with the electric field strength, distortion rate, and abnormal duration, the module performs leakage current classification and outputs the judgment results. The location, electric field curve, and alarm information are simultaneously uploaded to the cloud.

[0008] Furthermore, the features are input into the dual-arm force-position hybrid control model, generating robotic arm control signals. Based on these robotic arm control signals, a target motion trajectory is generated, including: The pose and contour features are input into the dual-arm force-position hybrid control model in the edge computing module. The dual-arm force-position hybrid control model includes a visual pose calculation module, a kinematics inverse algorithm module, a force-position parameter generation module, and a control signal construction module. The visual pose calculation module converts the received pose features and contour features into robot three-dimensional spatial coordinates. The inverse kinematics algorithm module uses the door hinge as the rotation center, the door handle center as the clamping point, and the middle of the door panel as the auxiliary support point to calculate and solve the corresponding rotation angles of each joint of the left and right robotic arms, and outputs the clamping motion path of the left arm and the auxiliary support motion path of the right arm. The force-position parameter generation module combines the gate's self-weight parameters with the preset safety torque threshold to generate the force control target value and the position loop feedforward compensation amount in real time. The robot's three-dimensional spatial coordinates, the left arm gripping motion path and the right arm auxiliary support motion path, as well as the force control target value and the position loop feedforward compensation amount are integrated through the control signal construction module to form a robotic arm control signal that includes the gripper opening and closing displacement, the angular velocity of each joint, and the force control feedback gain parameters. Based on the robotic arm control signal, the target motion trajectory of the robotic arm is planned and generated.

[0009] Furthermore, the target movement trajectory includes: door opening trajectory, internal inspection trajectory, and door closing trajectory; The door opening trajectory is specifically as follows: with the door hinge of the target device door as the center and the distance from the center of the door handle to the door hinge as the radius of rotation, a counterclockwise or clockwise arc trajectory is generated around the door hinge; The internal inspection trajectory is as follows: when the door is opened to a preset fixed angle, the right arm carrying the wrist camera moves along a preset Z-shaped up-and-down scanning trajectory inside the cabinet. The closing trajectory is specifically as follows: both arms move synchronously in the opposite direction along the opening arc trajectory to form a closing arc return trajectory corresponding to the opening trajectory.

[0010] Furthermore, the inspection process inside the target equipment door includes: The left arm uses force control mode to keep the door at a constant opening angle, preventing the door from closing or shaking on its own, and providing a stable working space for internal inspection; the right arm uses position control mode to carry a wrist camera into the cabinet, collect images of key parts according to the preset internal inspection trajectory, and send them to the edge computing module to complete the status inspection of the fire-fighting equipment in the cabinet.

[0011] On the other hand, a working device based on a multimodal dual-arm building inspection robot is provided, including: Task path planning module: Obtains a preset inspection task path that includes at least one inspection point in a multi-story building, and controls the robot to perform inspections along the preset path; Safety hazard detection module: Collects multimodal environmental signals during inspections and performs safety hazard detection; Feature extraction module: After arriving at the designated inspection point, the target door image is acquired through the wrist high-definition camera, and the pose and contour features are extracted; Target motion trajectory generation module: Inputs the pose and contour features into the dual-arm force-position hybrid control model to generate robotic arm control signals including end-gripper joint angles and force control quantities; generates the target motion trajectory based on the robotic arm control signals; Operation control module: sends data to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

[0012] In another aspect, embodiments of this application also provide a robot, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, it implements the operation method of a multimodal dual-arm building inspection robot as described in any one of the claims.

[0013] In another aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the operation method of a multimodal dual-arm building inspection robot as described in any one of the claims.

[0014] In another aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the operation method of a multimodal dual-arm building inspection robot described in the above aspects.

[0015] The above technical solution has the following advantages or beneficial effects: This invention provides a multimodal dual-arm building inspection robot operation method and device. By acquiring a preset inspection task path, the robot is controlled to perform inspections along the preset path. During the inspection process, multimodal environmental signals are collected, and safety hazard detection is performed. Upon reaching a designated inspection point, an image of the target door is acquired, and its pose and contour features are extracted. The pose and contour features are input into a dual-arm force-position hybrid control model to generate robotic arm control signals. Based on the robotic arm control signals, a target motion trajectory is generated. The end effector completes the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory. The inspection robot of this invention possesses autonomous inspection, dual-arm collaborative operation, non-contact leakage detection, real-time edge decision-making, and anti-interference stable operation capabilities. In complex building environments such as high-rise office buildings, commercial complexes, research buildings, and industrial parks, it can ensure the stability, safety, and reliability of its inspection and operation, which is beneficial to ensuring the safe and stable operation of building fire protection, electricity, environment, and equipment. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a schematic diagram of a multimodal dual-arm building inspection robot as described in Embodiment 1.

[0018] Figure 2 This is a flowchart illustrating the operation of a multimodal dual-arm building inspection robot as described in Example 1. Detailed Implementation

[0019] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of this invention, "multiple" refers to two or more.

[0022] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0023] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0024] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0025] Example 1 This embodiment provides a multimodal dual-arm building inspection robot, such as... Figure 1 As shown, the inspection robot consists of a main body and an omnidirectional mobile chassis.

[0026] The inspection robot's main body is constructed with a lightweight, high-strength aluminum alloy frame, featuring a centrally located electric lifting mechanism with a lifting stroke of at least 60cm and continuously adjustable lifting speed. This mechanism adjusts the operating height of the dual robotic arms to accommodate inspections of objects at varying heights, such as high-level fire-fighting facilities, low-level power distribution interfaces, wall-mounted equipment, and pipe valves, thus expanding the sensing and operational coverage. The dual robotic arms are symmetrically mounted on either side of the electric lifting mechanism. Both are 6-DOF modular collaborative articulated arms with a single arm carrying a 3kg load and a repeatability accuracy of ±0.1mm. Each robotic arm integrates an end effector and a wrist-mounted high-definition camera. The end effector includes an electric two-finger gripper and a six-dimensional force / torque sensor.

[0027] The inspection robot's main body is equipped with a multimodal perception module, an edge computing module, a communication module, and an automatic charging module.

[0028] The multimodal sensing module is used to collect multi-dimensional environmental and equipment status signals to achieve full-area, blind-spot-free multi-dimensional sensing. It includes: a head sensing unit, including a visible light high-definition night vision camera, an infrared thermal imager, and a 360° omnidirectional gimbal; a mid-body sensing unit, including a combustible gas sensor, a toxic gas sensor, a CO2 sensor, a photoelectric smoke sensor, a temperature and humidity sensor, and other sensors; a bottom-body sensing unit, including a non-contact electric field probe; and a surrounding distributed sensing unit, including an ultrasonic sensor and a microphone.

[0029] The edge computing module is used to identify safety hazards from signals collected by the multimodal sensing module, control the motion planning of the dual robotic arms, and generate local real-time alarms. This edge computing module utilizes the NVIDIA Jetson AGX Orin embedded computing platform, running the Ubuntu Linux operating system and the ROS2 Humble robot middleware.

[0030] The communication module supports 5G, Wi-Fi 6, and Ethernet tri-mode redundant communication, enabling real-time data interaction between the robot and the cloud-based central management system. It can complete the uploading of inspection data, the reception of task instructions, the push of alarm information, and the transmission of video streams. When the network is interrupted, it automatically caches locally and resumes transmission after the interruption is restored.

[0031] The automatic charging module enables the inspection robot to autonomously locate charging stations, perform millimeter-level precise docking, and automatically charge without human intervention after completing the preset inspection tasks at all inspection points. It also allows for automatic return when the battery is low. The bottom of the main body integrates gold-plated charging plates, providing a battery life of no less than 4 hours under rated operating conditions, allowing it to continuously complete inspections at hundreds of points across multiple buildings.

[0032] The inspection robot's main body is fitted with an omnidirectional mobile chassis at its bottom for movement. This chassis features a four-wheel omnidirectional independent drive structure, with each wheel equipped with an independent drive motor and steering motor. It supports zero-radius in-situ turning, lateral translation, diagonal movement, and precise straight-line travel, allowing for flexible passage through complex building environments such as narrow corridors, elevator shafts, and equipment rooms. LiDAR and infrared collision avoidance sensors are installed at the front and rear of the chassis, enabling real-time SLAM mapping, autonomous localization, and dynamic obstacle avoidance. Its protection level is no lower than IP54, making it suitable for humid and dusty building environments.

[0033] This embodiment provides a working method based on a multimodal dual-arm building inspection robot, such as... Figure 2 As shown, the specific steps include: Obtain a preset inspection task path that includes at least one inspection point in a multi-story building, and control the robot to perform inspections along the preset path; During the inspection, multimodal environmental signals are collected, and safety hazard detection is performed. After reaching the designated inspection point, acquire the target door image and extract its pose and contour features; The pose and contour features are input into the dual-arm force-position hybrid control model to generate a robotic arm control signal that includes the end gripper joint angle and force control quantity. The target motion trajectory is generated based on the robotic arm control signal. The data is sent to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

[0034] S1: Obtain a preset inspection task path that includes at least one inspection point in a multi-story building, and control the robot to perform inspections along the preset path.

[0035] The preset inspection task path is a planned inspection route for the inspection robot based on the target equipment to be inspected in a specific multi-story building. The inspection robot will autonomously navigate to the elevator lobby according to the planned inspection task path, wirelessly call the elevator and autonomously select the floor. After arriving at the target floor, it will inspect the target equipment corresponding to each inspection point in sequence.

[0036] S2: Collect multimodal environmental signals during the inspection process and detect potential safety hazards.

[0037] In this embodiment, during the inspection process, the inspection robot uses its configured multimodal perception module to simultaneously collect five types of multimodal environmental data: visible light image data, infrared thermal image data, gas concentration data, electric field signal data, and audio feature data. Subsequently, the collected five types of multimodal environmental data are transmitted to the edge computing module, which uses DS evidence theory to perform decision-level fusion processing to achieve comprehensive judgment on various types of safety hazards such as fire, smoke, gas leaks, equipment overheating, abnormal pipe noises, and electrical leakage, effectively improving the comprehensiveness and accuracy of safety hazard identification.

[0038] Optionally, in this embodiment, the environmental image data collected by the head sensing unit in the multimodal perception module is obtained through the edge computing module, and the global environmental monitoring and status identification of various devices are completed based on the environmental image data, providing basic data support for the global safety hazard investigation of the inspection area.

[0039] Optionally, in this embodiment, the edge computing module acquires the sensor parameters collected by the sensing unit in the middle of the fuselage in the multimodal sensing module. Through real-time analysis and processing of the sensor parameters, it realizes real-time monitoring of various parameters of the inspection environment, and at the same time, it performs targeted detection and identification of potential gas leaks, ensuring timely capture of environmental anomalies and gas leak risks.

[0040] Optionally, in this embodiment, the edge computing module acquires spatial electric field distribution change data collected by the sensing unit at the bottom of the fuselage in the multimodal sensing module. Based on the spatial electric field distribution change data, it achieves non-contact, non-destructive, long-distance leakage hazard detection of the casings of electrical equipment such as distribution boxes, power supply lines, and sockets. The leakage hazard detection principle is as follows: Under normal energized conditions, the spatial electric field around electrical equipment exhibits a symmetrical and stable distribution. When leakage occurs or insulation performance deteriorates, stray electric fields and electric field distortions are generated around the equipment casing or lines. By collecting parameters such as spatial electric field strength, electric field gradient, and electric field distortion coefficient, the leakage risk of electrical equipment can be accurately determined.

[0041] Furthermore, the methods for edge computing modules to detect safety hazards in the casing of electrical equipment include: The inspection robot collects the environmental electric field at a distance of 50cm from the equipment using a non-contact electric field probe in the sensing unit at the bottom of the multimodal sensing module. The collected field is then sent to the edge computing module, which filters out power frequency interference and electromagnetic noise to obtain the background electric field. When the inspection robot moves to a position 5–30 cm in front of the equipment to be tested, such as the distribution box, power supply line, or socket, the chassis stops, the electric field probe is aimed at the detection area, and the robot scans horizontally and uniformly along the surface of the equipment to continuously collect electric field signals from multiple points. The signals are then sent to the edge computing module, which performs signal noise reduction and distortion rate calculation. Combined with the electric field strength, distortion rate, and abnormal duration, the robot performs leakage current classification.

[0042] Furthermore, leakage current can be determined based on the leakage current determination threshold: The maximum electric field strength is ≤1.2 times the background electric field, and the electric field distortion rate is ≤20%, which is considered to be in a normal state. When the maximum electric field strength is 1.2–1.5 times the background electric field, or the electric field distortion rate is 20%–50%, it is determined to be a warning state. If the maximum electric field strength is ≥1.5 times the background electric field, or the electric field distortion rate is ≥50%, and the abnormal duration is ≥500ms, it is determined to be a leakage current alarm state.

[0043] The edge computing module outputs the judgment result and synchronously uploads the location, electric field curve, and alarm information to the cloud. The edge computing module can complete the identification of safety hazards and alarm generation locally, achieving a response time in seconds without relying on the cloud, with an end-to-end alarm latency of ≤30 seconds. Even when the network is interrupted, the robot can still conduct independent inspections, and automatically synchronize data after the network is restored, ensuring uninterrupted inspections and no data loss.

[0044] This embodiment uses a non-contact electric field probe to detect safety hazards on the equipment casing, which can identify early and hidden electrical hazards. When fused with infrared thermal imaging and visual data, it will greatly improve the accuracy of electrical fire early warning.

[0045] Optionally, in this embodiment, the edge computing module acquires data from sensors and microphones collected by the distributed sensing units around the multimodal sensing module. By analyzing and processing the sensor data and microphone data, it can achieve near-distance obstacle avoidance and identification of abnormal equipment noises and pipeline leaks.

[0046] S3: After arriving at the designated inspection point, acquire the target door image and extract its pose and contour features; After the inspection robot described in this embodiment arrives at the designated inspection point, it acquires the target door image through the wrist camera, and then extracts the pose and contour features. The pose includes: the handle pose of the fire hydrant cabinet door / fire door, the door hinge position, and the opening direction.

[0047] S4: Input the features into the dual-arm force-position hybrid control model, generate the robotic arm control signal, and generate the target motion trajectory based on the robotic arm control signal.

[0048] The pose and contour features are input into the dual-arm force-position hybrid control model in the edge computing module. The dual-arm force-position hybrid control model includes a visual pose calculation module, a kinematics inverse algorithm module, a force-position parameter generation module, and a control signal construction module. The visual pose calculation module converts the received pose features and contour features into robot three-dimensional spatial coordinates. The inverse kinematics algorithm module uses the door hinge as the rotation center, the door handle center as the clamping point, and the middle of the door panel as the auxiliary support point to calculate and solve the corresponding rotation angles of each joint of the left and right robotic arms, and outputs the clamping motion path of the left arm and the auxiliary support motion path of the right arm. The force-position parameter generation module combines the gate's self-weight parameters with the preset safety torque threshold to generate the force control target value and the position loop feedforward compensation amount in real time. The robot's three-dimensional spatial coordinates, the left arm gripping motion path and the right arm auxiliary support motion path, as well as the force control target value and the position loop feedforward compensation amount are integrated through the control signal construction module to form a robotic arm control signal that includes the gripper opening and closing displacement, the angular velocity of each joint, and the force control feedback gain parameters. Based on the robotic arm control signal, the target motion trajectory of the robotic arm is planned and generated.

[0049] The robotic arm used in this embodiment has a parallel closed-loop architecture with a position loop as the main loop and a force loop as the secondary loop, and real-time feedback is based on a six-dimensional force / torque sensor.

[0050] S5: Send to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

[0051] The generated robotic arm control signal is sent to the end effector. The end effector, based on the target motion trajectory, sequentially completes the opening, internal inspection, and reset operations of the target equipment door. Specifically, the target motion trajectory includes an opening trajectory, an internal inspection trajectory, and a closing trajectory. The specific execution process of each trajectory is as follows: The door opening trajectory is specifically defined as follows: using the door hinge of the target equipment door as the center and the distance from the center of the door handle to the door hinge as the radius of rotation, a counterclockwise or clockwise arc trajectory is generated around the door hinge. During execution, the left arm smoothly pulls the door handle along the arc trajectory, while the right arm performs a follow-up auxiliary support movement along the plane of the door panel, achieving smooth opening of the target equipment door.

[0052] Specifically, the opening process of the target equipment door includes: The robotic arm generates an arc motion trajectory around the door hinge using control signals. The left arm quickly approaches the door handle using position control mode. Once the clamping mechanism is in place, it switches to force control mode to maintain a constant clamping force to prevent damage to the door handle. The right arm lightly touches the door panel using position control mode to maintain a zero-force contact state, providing auxiliary support and preventing the door from shaking.

[0053] The door opening direction employs a force control constraint mechanism, limiting the maximum opening pull force to no more than 30N to prevent damage to the door or handle due to excessive force. The vertical direction uses a position control mode to maintain the stability of the robotic arm and door posture. When the door encounters jamming during opening, the compliance coefficient is adjusted in real time, controlling the robotic arm to automatically decelerate, retract, and re-attempt opening. Simultaneously, based on force / torque information from six-dimensional force / torque sensors and the angle information of each joint of the robotic arm, the opening degree and posture of the door are estimated in real time, dynamically correcting the motion trajectory to effectively avoid overshoot and collisions between the door and the robotic arm.

[0054] The internal inspection trajectory is as follows: when the door is opened to a preset fixed angle, the right arm carrying the wrist camera moves along a preset Z-shaped up-and-down scanning trajectory inside the cabinet, sequentially aiming at key locations such as the fire extinguisher pressure gauge, fire hose, valve, and interface inside the cabinet, completing the image acquisition of each key part, and sending it to the edge computing module for safety inspection.

[0055] Specifically, the inspection process inside the target equipment door includes: The left arm uses force control mode to keep the door at a constant opening angle, preventing the door from closing or shaking on its own, and providing a stable working space for internal inspection; the right arm uses position control mode to carry a wrist camera into the cabinet, collect images of key parts according to the preset internal inspection trajectory, and send them to the edge computing module to complete the status inspection of the fire-fighting equipment in the cabinet.

[0056] In this embodiment, the specific steps for the edge computing module to perform status checks on fire-fighting equipment are as follows: The right arm carries a wrist camera that is inserted into the cabinet and takes pictures of each area inside the cabinet in order from top to bottom to obtain overall image information of the cabinet. The specific location of the fire extinguisher inside the cabinet is located by image recognition algorithm, and the range of the pointer of the fire extinguisher pressure gauge is read to determine whether the fire extinguisher pressure status is normal, under-pressure or over-pressure. Image recognition technology is used to identify the coiling status of fire hoses, determine whether they are coiled neatly, and detect whether the hoses are damaged or the joints are intact. Check the opening and closing status of the fire valves inside the cabinet to confirm that they are in the preset normal opening and closing positions. At the same time, check whether there is rust on the valve surface and whether there is leakage at the valve connection. Check the working status of the alarm buttons and indicator lights inside the cabinet, as well as the connection status of the pipe interfaces, to confirm that all components are in normal working condition; All the key inspection areas mentioned above are photographed and stored. The collected image data is transmitted to the edge computing module, which analyzes and judges the status of the fire-fighting equipment and generates corresponding inspection conclusions. If the edge computing module determines that there is a safety hazard in the fire-fighting equipment, it sends an alarm signal through the alarm module of the edge computing module. At the same time, the specific location of the safety hazard and the alarm information are uploaded to the cloud server so that the staff can handle it in a timely manner.

[0057] The closing trajectory is specifically as follows: both arms move synchronously in the opposite direction along the opening arc trajectory to form a closing arc return trajectory corresponding to the opening trajectory. During execution, after the door is fully closed, the right arm gently pushes the door panel to confirm that the door lock is engaged. After confirming that it is engaged, the left arm slowly releases the gripper, and both arms return to the initial posture along the preset safety avoidance trajectory.

[0058] Specifically, the closing process of the target equipment door includes: The robotic arm control signals control both arms to apply forces synchronously and in opposite directions, slowly closing the door with a closing force not exceeding 20N to avoid damage to the door, lock, or equipment inside the cabinet due to excessive closing force. After the door closes, a visual recognition module confirms whether the lock is locked. If the lock is not locked, the robotic arm automatically performs a second closing operation until the lock is locked. After the lock is locked, the left arm gripper slowly releases, and both arms move along a preset safety avoidance trajectory, returning to the initial working posture. Simultaneously, the force / position data and operation results throughout the entire operation are recorded, providing data support for subsequent troubleshooting and process optimization.

[0059] This embodiment uses a dual-arm force-position hybrid compliant control system to achieve composite working modes such as one-arm fixation, one-arm operation, dual-arm synchronous clamping, and coordinated push-pull. It can autonomously complete the smooth opening of fire hydrant cabinet doors and normally closed fire doors, check the status of internal equipment, and perform door closing and reset operations. It has high operational accuracy, strong anti-interference ability, and does not damage the equipment.

[0060] This embodiment describes a working method based on a multimodal dual-arm building inspection robot. It supports autonomous navigation, autonomous elevator access, autonomous inspection, and autonomous recharging in multi-story buildings, forming a fully automated closed-loop inspection system capable of 24-hour unmanned operation. The overall system exhibits strong environmental adaptability and stable, reliable operation, effectively replacing manual labor in building safety inspections and significantly improving building fire safety and electrical safety management.

[0061] Example 2 This embodiment provides a working device based on a multimodal dual-arm building inspection robot, applied to the inspection robot, including: Task path planning module: Obtains a preset inspection task path that includes at least one inspection point in a multi-story building, and controls the robot to perform inspections along the preset path; Safety hazard detection module: Collects multimodal environmental signals during inspections and performs safety hazard detection; Feature extraction module: After reaching the designated inspection point, acquire the target door image and extract pose and contour features; Target motion trajectory generation module: Inputs the pose and contour features into the dual-arm force-position hybrid control model to generate robotic arm control signals including end-gripper joint angles and force control quantities; generates the target motion trajectory based on the robotic arm control signals; Operation control module: Sends data to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

[0062] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0063] The proposed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative, and the division of the modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0064] Example 3 The robot provided in this application embodiment may also include a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described operation method embodiment based on the multimodal dual-arm building inspection robot and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0065] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0067] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0068] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0069] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0070] Example 4 This embodiment also provides a computer storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0071] Example 5 This embodiment also provides a computer program product, including a computer program that, when run on one or more processors, implements the method described in Embodiment 1.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for operating a multimodal dual-arm building inspection robot, characterized in that, include: Obtain a preset inspection task path that includes at least one inspection point in a multi-story building, and control the robot to perform inspections along the preset path; During the inspection, multimodal environmental signals are collected, and safety hazard detection is performed. After reaching the designated inspection point, acquire the target door image and extract its pose and contour features; The pose and contour features are input into the dual-arm force-position hybrid control model to generate a robotic arm control signal that includes the end gripper joint angle and force control quantity. The target motion trajectory is generated based on the robotic arm control signal. The data is sent to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

2. The operation method of a multimodal dual-arm building inspection robot according to claim 1, characterized in that, During the inspection, multimodal environmental signals are collected, and safety hazard detection is performed, including: During the inspection process, the inspection robot uses its configured multimodal perception module to simultaneously collect five types of multimodal environmental data: visible light image data, infrared thermal image data, gas concentration data, electric field signal data, and audio feature data. The collected data from the five types of multimodal environmental data are then transmitted to the edge computing module. The edge computing module uses DS evidence theory to perform decision-level fusion processing, enabling comprehensive judgment of various types of safety hazards such as fire, smoke, gas leaks, equipment overheating, abnormal pipe noises, and electrical casing leakage.

3. The operation method of a multimodal dual-arm building inspection robot according to claim 2, characterized in that, The method for edge computing modules to detect safety hazards related to leakage current in electrical enclosures includes: The inspection robot collects the ambient electric field at a distance from the equipment and sends it to the edge computing module. The edge computing module filters and eliminates power frequency interference and electromagnetic noise to obtain the background electric field. When the inspection robot moves to the device under test, the chassis stops, the electric field probe is aligned with the detection area, and the robot scans the surface of the device at a constant speed to continuously collect electric field signals from multiple points. The signals are then sent to the edge computing module, which performs signal noise reduction and distortion rate calculation. Combined with the electric field strength, distortion rate, and abnormal duration, the module performs leakage current classification and outputs the judgment results. The location, electric field curve, and alarm information are simultaneously uploaded to the cloud.

4. The operation method of a multimodal dual-arm building inspection robot according to claim 1, characterized in that, The features are input into the dual-arm force-position hybrid control model, generating robotic arm control signals. Based on these signals, the target motion trajectory is generated, including: The pose and contour features are input into the dual-arm force-position hybrid control model in the edge computing module. The dual-arm force-position hybrid control model includes a visual pose calculation module, a kinematics inverse algorithm module, a force-position parameter generation module, and a control signal construction module. The visual pose calculation module converts the received pose features and contour features into robot three-dimensional spatial coordinates. The inverse kinematics algorithm module uses the door hinge as the rotation center, the door handle center as the clamping point, and the middle of the door panel as the auxiliary support point to calculate and solve the corresponding rotation angles of each joint of the left and right robotic arms, and outputs the clamping motion path of the left arm and the auxiliary support motion path of the right arm. The force-position parameter generation module combines the gate's self-weight parameters with the preset safety torque threshold to generate the force control target value and the position loop feedforward compensation amount in real time. The robot's three-dimensional spatial coordinates, the left arm gripping motion path and the right arm auxiliary support motion path, as well as the force control target value and the position loop feedforward compensation amount are integrated through the control signal construction module to form a robotic arm control signal that includes the gripper opening and closing displacement, the angular velocity of each joint, and the force control feedback gain parameters. Based on the robotic arm control signal, the target motion trajectory of the robotic arm is planned and generated.

5. The operation method of a multimodal dual-arm building inspection robot according to claim 1, characterized in that, The target movement trajectory includes: door opening trajectory, internal inspection trajectory, and door closing trajectory; The door opening trajectory is specifically as follows: with the door hinge of the target device door as the center and the distance from the center of the door handle to the door hinge as the radius of rotation, a counterclockwise or clockwise arc trajectory is generated around the door hinge; The internal inspection trajectory is as follows: when the door is opened to a preset fixed angle, the right arm carrying the wrist camera moves along a preset Z-shaped up-and-down scanning trajectory inside the cabinet. The closing trajectory is specifically as follows: both arms move synchronously in the opposite direction along the opening arc trajectory to form a closing arc return trajectory corresponding to the opening trajectory.

6. The operation method of a multimodal dual-arm building inspection robot according to claim 1, characterized in that, The inspection process inside the target equipment door includes: The left arm uses force control mode to keep the door at a constant opening angle, preventing the door from closing or shaking on its own, and providing a stable working space for internal inspection; the right arm uses position control mode to carry a wrist camera into the cabinet, collect images of key parts according to the preset internal inspection trajectory, and send them to the edge computing module to complete the status inspection of the fire-fighting equipment in the cabinet.

7. A working device based on a multimodal dual-arm building inspection robot, characterized in that, include: Task path planning module: Obtains a preset inspection task path that includes at least one inspection point in a multi-story building, and controls the robot to perform inspections along the preset path; Safety hazard detection module: Collects multimodal environmental signals during inspections and performs safety hazard detection; Feature extraction module: After arriving at the designated inspection point, the target door image is acquired through the wrist high-definition camera, and the pose and contour features are extracted; Target motion trajectory generation module: Inputs the pose and contour features into the dual-arm force-position hybrid control model to generate robotic arm control signals including end-gripper joint angles and force control quantities; generates the target motion trajectory based on the robotic arm control signals; Operation control module: sends data to the end effector, which then performs the opening, internal inspection, and reset operations of the target equipment door according to the target motion trajectory.

8. A robot, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the operation method of a multimodal dual-arm building inspection robot according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the operation method of a multimodal dual-arm building inspection robot as described in any one of claims 1-6.

10. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the operation method of a multimodal dual-arm building inspection robot as described in any one of claims 1-6.

Images