A multi-modal intelligent inspection robot system for chemical laboratories

Abstract

This invention discloses a multimodal intelligent inspection robot system for chemical laboratories, comprising: an autonomous mobile robot platform, a multimodal intelligent sensing unit, an edge intelligent processing unit, and a robot motion control unit. The autonomous mobile robot platform includes an AGV chassis, an industrial control cabinet, a human-machine interface, a lifting device, and a gimbal mechanism. The multimodal intelligent sensing unit collects visual image data and environmental parameter data. The edge intelligent processing unit synchronizes data and sends anomaly detection requests to a cloud-based multimodal large-scale model. The robot motion control unit extracts anomaly type labels and bounding box coordinates from the anomaly detection results, calculates the coordinates of the anomaly area, drives the robot to move, and simultaneously queries an emergency response mapping table to execute corresponding actions. This invention aims to solve the safety supervision problem in unattended chemical laboratories, achieving cross-modal joint reasoning and accurate identification of safety hazards.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of intelligent robots and safety production monitoring, specifically relating to a multimodal intelligent inspection robot system suitable for chemical laboratory environments, used to achieve automated perception, intelligent identification and autonomous response to safety risks in experimental sites. Background Technology

[0002] Chemical laboratories commonly face safety risks such as toxic gas leaks, uncontrolled high-temperature reactions, and abnormal equipment operation. Traditional manual inspection methods suffer from problems such as delayed response, large blind spots, and difficulties in nighttime monitoring, making it difficult to meet the continuous safety monitoring needs of high-risk chemical experiments.

[0003] In existing technologies, some laboratories deploy fixed sensors or monitoring cameras, but due to their fixed installation locations and limited sensing modalities, they cannot effectively track and locate mobile anomalies. In recent years, although there have been attempts to apply inspection robots to industrial scenarios, most existing inspection robots only possess single-modal visual perception capabilities, lacking the ability to fuse and analyze multimodal information such as gas leaks and abnormal noise. Furthermore, their identification of safety hazards in complex scenarios relies on preset rules or traditional visual algorithms, resulting in low accuracy and poor generalization ability. In addition, existing solutions, after detecting anomalies, typically only issue alarm signals, lacking the ability to autonomously move to the anomaly source location for close inspection and coordinated handling of laboratory equipment, making it difficult to form a complete closed loop of "perception-analysis-decision-execution."

[0004] Therefore, there is an urgent need for a multimodal intelligent inspection robot system that can integrate multimodal perception information, utilize large-scale intelligent analysis capabilities, and possess autonomous mobility and emergency response functions to address the practical needs of chemical laboratory safety supervision. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal intelligent inspection robot system for chemical laboratories. It aims to solve the safety supervision problem in unattended chemical laboratories and realize cross-modal joint reasoning and accurate identification of safety hazards such as reagent leakage, abnormal equipment noise, overheating reaction, and excessive gas, thereby effectively supporting the unattended operation mode of high-risk chemical experiments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a multimodal intelligent inspection robot system for chemical laboratories, characterized by comprising: an autonomous mobile robot platform, a multimodal intelligent sensing unit, an edge intelligent processing unit, and a robot motion control unit; wherein, The autonomous mobile robot platform includes: an AGV chassis, an industrial control cabinet, a human-machine interface, a lifting device, and a gimbal mechanism. The AGV chassis is located at the bottom of the platform and drives the robot's movement; a positioning and navigation module is integrated within the chassis. The industrial control cabinet is located above the AGV chassis and houses the edge intelligent processing unit and the robot motion control unit. A human-machine interface is located on the front of the cabinet, displaying inspection results and supporting user interaction control. A lifting device is located on the top of the cabinet, and a gimbal mechanism with horizontal and pitch degrees of freedom is mounted on the lifting device. The multimodal intelligent sensing unit includes a visual sensor and an environmental sensor; the visual sensor is installed on the rotating end of the gimbal mechanism and is used to collect visual image data of the laboratory scene; the environmental sensor is installed on the side of the industrial control chassis and is used to collect environmental parameter data of the laboratory. The edge intelligent processing unit is used to receive the visual image data and the environmental parameter data, and synchronize the data through a multi-threaded architecture; then, it concatenates the synchronized data with a pre-stored strong constraint prompt word template to generate an anomaly detection request, and then sends the anomaly detection request to an external cloud multimodal large model through wireless communication; at the same time, it receives and parses the structured anomaly detection results returned by the cloud multimodal large model, and sends the structured anomaly detection results to the robot motion control unit; After receiving the structured anomaly detection result, the robot motion control unit extracts the anomaly type label, risk level information, and normalized bounding box coordinates of the anomaly region in the image coordinate system. Combined with the current attitude angle fed back by the gimbal mechanism and the current global physical coordinates fed back by the positioning and navigation module, it calculates the target physical coordinates of the anomaly region in the laboratory global physical coordinate system through a coordinate projection transformation algorithm. This drives the AGV chassis to move to a preset range of the target position. Using the anomaly type label as an index, it queries a pre-stored accident emergency response mapping table to obtain an emergency response instruction sequence and executes the corresponding response strategy.

[0007] The multimodal intelligent inspection robot system for chemical laboratories described in this invention is also characterized by the multi-threaded architecture in the edge intelligent processing unit, which includes: a visible light video acquisition thread, an infrared video acquisition thread, and an environmental data acquisition thread; after communicating with each thread through an asynchronous message queue, the visual image data and the environmental parameter data are aligned based on timestamps to obtain synchronized data; the edge intelligent processing unit constructs a unified time base, thereby using a sliding time window to cache and resample the synchronized data.

[0008] Furthermore, the edge intelligent processing unit's parsing of the structured anomaly detection results includes: structure verification and anomaly correction processing. The structure verification includes: field integrity verification, data type verification, and boundary value validity verification. The anomaly correction processing includes: completing missing fields based on preset rules, trimming and correcting abnormal coordinates, reconstructing and parsing response data that does not conform to the format, and triggering a retry mechanism when multiple verifications fail.

[0009] Furthermore, the edge intelligence processing unit also includes a post-processing and visualization module; The post-processing and visualization module is used to convert the normalized bounding box coordinates into pixel coordinates in the original image; Based on the anomaly type labels and risk level information in the structured anomaly detection results, the normalized bounding boxes are semantically enhanced and annotated. The semantically enhanced annotation includes at least: color coding, text labels, and risk level indicators. Temporal correlation analysis is performed on abnormal regions in multiple consecutive frames of images to calculate the stability index of the abnormal regions; and the confidence of the anomaly detection results is corrected based on the stability index to generate dynamic visualization results containing information on the anomaly change trend, which are then displayed on the human-computer interaction interface.

[0010] Furthermore, the coordinate projection transformation algorithm in the robot motion control unit performs the following calculation logic: Obtain the intrinsic parameter matrix of the vision sensor and extrinsic parameter matrix ,in, For rotation matrix, The translation vector is used; the extrinsic parameter matrix is ​​determined by the current attitude angle fed back by the gimbal mechanism and the robot's current global physical coordinates fed back by the positioning and navigation module. Extract the center point of the bounding box of the abnormal region as the target point, and mark the coordinates of the target point in the image as follows: ; Based on the pinhole camera model, calculate the normalized ray direction vector. And construct a parameterized spatial ray expression for the target point in the global physical coordinate system of the laboratory scene: ,in, The scale parameter along the ray is used to determine the specific location of the target point on the ray; If the vision sensor uses a depth camera to acquire depth information, then the coordinates of the target point will be compared with those of the depth sensor. Corresponding depth value The scale parameter is used to calculate the three-dimensional physical coordinates of the target point; If the visual sensor uses a visible light camera and does not acquire depth information, the intersection point of the ray and the reference plane is solved by a preset reference plane to determine the scale parameter and calculate the two-dimensional physical coordinates of the target point.

[0011] Furthermore, the positioning and navigation module is configured to perform adaptive path planning and dynamic obstacle avoidance, specifically including: performing global path planning based on a pre-built topology map of the laboratory scene, and continuously checking environmental perception data during the inspection process; when environmental changes or dynamic obstacles are detected that obstruct the current task, local path replanning is triggered, the original task is canceled, and a new travel sequence is generated from the currently reachable nodes; at the same time, the access order of the inspection path is dynamically weighted and adjusted according to the risk level of the abnormal area, so that high-risk abnormal areas get higher priority in the path queue, thereby realizing risk-driven adaptive inspection scheduling.

[0012] Furthermore, the robot motion control unit also includes a gimbal control module; The gimbal control module dynamically adjusts the target attitude angle of the gimbal mechanism based on the deviation between the center coordinates of the bounding box of the abnormal region and the center coordinates of the image, so that the optical axis of the vision sensor is aligned with the physical spatial position corresponding to the abnormal region. At the same time, the gimbal control module constructs a closed-loop control loop based on the center point position of the bounding box of the abnormal region in the image fed back by the vision sensor, and continuously tracks the abnormal target in the abnormal region, so that the abnormal region is always kept in the center area of ​​the field of view of the vision sensor.

[0013] Furthermore, the accident emergency response mapping table in the robot motion control unit is a mapping relationship table between anomaly types and multi-level response strategies. The robot motion control unit selects the corresponding response strategy according to the anomaly type label and risk level information. The response strategies include: equipment linkage control, alarm triggering, and remote notification. Among them, the equipment linkage control includes: control of the ventilation system, gas cut-off device, and safety protection equipment.

[0014] The present invention provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to execute the computer program to implement the multimodal intelligent inspection robot system for chemical laboratories.

[0015] The present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the multimodal intelligent inspection robot system for chemical laboratories.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs an inspection system that integrates an autonomous mobile robot platform, a multimodal intelligent sensing unit, and an edge intelligent processing unit, thereby achieving automated and unmanned inspection of the chemical laboratory environment, reducing the cost of manual inspection, and improving the safety and continuity of the inspection process.

[0017] 2. This invention integrates visual sensors and environmental sensors, and combines multi-threaded data synchronization, caching and resampling mechanisms to achieve collaborative perception and efficient fusion of multimodal data, effectively solving the problem of inconsistent time sequence of multi-source data and significantly improving the accuracy and robustness of anomaly detection.

[0018] 3. This invention introduces strongly constrained prompt word templates and combines them with cloud-based multimodal large models to generate structured anomaly detection results. At the same time, it performs structural verification and anomaly correction on the model output, thereby improving the system's semantic understanding of complex scenarios and its fault tolerance for unstable output, thus enhancing the overall stability and engineering usability of the system.

[0019] 4. This invention maps abnormal regions in image space to the laboratory's global physical coordinate system using a coordinate projection transformation algorithm. Combined with adaptive path planning and dynamic obstacle avoidance mechanisms, it achieves closed-loop control from anomaly detection to target localization and autonomous navigation. Simultaneously, it continuously tracks abnormal targets through a gimbal-based closed-loop tracking mechanism and executes multi-level automated handling based on an emergency response mapping strategy. Combined with a visualization module for dynamic display, this improves the accuracy and efficiency of anomaly handling and enhances the system's real-time interactive capabilities and laboratory safety assurance level. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system structure of the multimodal intelligent inspection robot in an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal data processing and collaborative control method in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the visualization of anomaly detection results in an embodiment of the present invention. Detailed Implementation

[0021] In this embodiment, a multimodal intelligent inspection robot system for chemical laboratories includes: an autonomous mobile robot platform, a multimodal intelligent sensing unit, an edge intelligent processing unit, and a robot motion control unit. These units are interconnected via wired or wireless communication to form a closed-loop control system. The overall system configuration forms a closed-loop link of "perception—analysis—decision-execution—feedback," enabling the system to not only detect anomalies but also proactively respond and handle them, thus distinguishing it from traditional inspection equipment that only has monitoring functions.

[0022] The overall structure of the autonomous mobile robot platform is as follows: Figure 1 As shown, it includes: an AGV chassis, an industrial control cabinet, a human-machine interface, a lifting device, and a gimbal mechanism; the AGV chassis is located at the bottom of the autonomous mobile robot platform and integrates a positioning and navigation module to drive the robot's movement; the industrial control cabinet is located above the AGV chassis and houses an edge intelligence processing unit and a robot motion control unit; the human-machine interface is located on the front of the industrial control cabinet and is used to display inspection results and support user interaction control; the lifting device is located on the top of the industrial control cabinet; the gimbal mechanism is located above the lifting device and has horizontal rotation and pitch rotation degrees of freedom.

[0023] The aforementioned vertically layered layout of "chassis support + central computing + top sensing" places the vehicle's center of gravity within the chassis height range. Compared to the conventional approach of placing sensors on top, this significantly improves the robot's anti-tipping ability during sudden stops, turns, and traversing uneven ground, ensuring its safety in narrow laboratory passageways. In practical implementation, the AGV chassis preferably adopts a differential drive structure. The positioning and navigation module can achieve centimeter-level positioning accuracy based on laser SLAM and supports autonomous path planning and dynamic obstacle avoidance. The lifting device is used to adjust the sensing height, with a travel range of 0.3m to 1.5m to cover different height areas such as experimental tables and fume hoods. The gimbal mechanism provides two-dimensional degrees of freedom, with a horizontal rotation angle range covering a 360-degree panoramic view and a pitch rotation angle range covering 90 degrees vertically, enabling the vision sensor to actively align with the target area, reducing the frequency of vehicle movement and improving response efficiency.

[0024] The multimodal intelligent sensing unit includes a visual sensor and an environmental sensor. The visual sensor is mounted on the rotating end of the gimbal mechanism to collect visual image data of the laboratory scene. The environmental sensor is mounted on the side of the industrial control chassis to collect environmental parameter data of the laboratory. In specific implementations, the visual sensor preferably includes an RGB camera and an infrared thermal imaging camera. The RGB camera can have a resolution of 1920×1080 and a frame rate of 30fps, mainly used to acquire texture and structural information. The infrared camera has a temperature resolution preferably ≤0.1℃, used to detect temperature anomalies in equipment or reagents. Furthermore, the visual sensor can also be equipped with a depth camera to directly acquire spatial distance information, thereby providing scale parameters for subsequent coordinate calculations. The environmental sensor preferably includes an H2S sensor, a CO sensor, a temperature sensor, and a noise sensor. The outputs of various sensors reflect the trend of environmental changes in the form of a time series. This arrangement fixes the environmental sensor to the side of the chassis, which is closer to the gas diffusion path and avoids sampling disturbances caused by the movement of the gimbal, thereby improving detection stability and achieving complementary fusion of visual and environmental information.

[0025] like Figure 2As shown, this processing flow constitutes the core data path of the system, including the following steps: Starting visible light video acquisition threads, infrared video acquisition threads, and environmental data acquisition threads to acquire RGB images, infrared thermal images, and sensor values ​​respectively; aligning the data of each thread based on timestamps, and using a sliding time window to eliminate timing deviations caused by sampling frequency differences; encoding the synchronized image into a Base64 string, and concatenating it with a pre-stored strong constraint prompt template to generate an API request; sending the request to the cloud-based multimodal large model service via a wireless communication module and receiving the returned response data; parsing the structured anomaly detection results from the response data, including anomaly type labels, normalized bounding box coordinates, and risk level information; converting the normalized coordinates to pixel coordinates, and color-coding the bounding box according to the risk level to generate a visual marker image; combining the gimbal attitude angle and the global physical coordinates fed back by the positioning and navigation module to calculate the target physical coordinates of the anomaly area; driving the AGV chassis to move to the anomaly position, while simultaneously adjusting the gimbal to align the optical axis of the vision sensor with the anomaly area; querying the accident emergency response mapping table using the anomaly type label and executing the corresponding response instructions.

[0026] The edge intelligent processing unit receives visual image data and environmental parameter data, and synchronizes the data through a multi-threaded architecture. It then concatenates the synchronized data with pre-stored strong constraint cue word templates to generate an anomaly detection request, which is sent wirelessly to an external cloud-based multimodal large-scale model. Finally, it receives and parses the structured anomaly detection results returned by the cloud-based multimodal large-scale model, and sends these results to the robot motion control unit. In this configuration, the edge unit is responsible for data preprocessing and scheduling, while the cloud-based multimodal large-scale model handles complex semantic understanding and anomaly determination, thus achieving high-precision detection under resource-constrained conditions.

[0027] The multi-threaded architecture of the edge intelligent processing unit includes: visible light video acquisition threads, infrared video acquisition threads, and environmental data acquisition threads. These threads communicate via asynchronous message queues and align multi-source data based on timestamps. The edge intelligent processing unit also constructs a unified time reference, using a sliding time window to cache and resample multimodal data to eliminate timing discrepancies caused by differences in sampling frequencies from different sensors. Through this mechanism, high-frequency video streams and low-frequency sensor data can be uniformly mapped to the same time slice, ensuring the consistency of multimodal inputs and avoiding false detections or missed detections due to time misalignment.

[0028] The edge intelligent processing unit stores strong constraint prompt word templates to generate model response data that conforms to a predefined data structure. The edge intelligent processing unit is also used to perform structure verification and anomaly correction on the response data. Structure verification includes field integrity verification, data type verification, and boundary value validity verification. Anomaly correction includes completing missing fields based on preset rules, pruning and correcting abnormal coordinates, reconstructing and parsing response data that does not conform to the format, and triggering a retry mechanism when multiple verifications fail.

[0029] For example, when the abnormal data returned from the cloud lacks a description field, the system automatically completes it using the abnormal type as the default description; when the returned normalized coordinates exceed the range of 0 to 1, the system prunes them to the boundary values; when the returned data format does not conform to the JSON specification, the system extracts key information using regular expressions and reconstructs it into a standard structure. If multiple consecutive validations fail, the system triggers a retry mechanism to resend the request, and if the retry still fails, it enters a security degradation mode, where local pre-defined rules take over the abnormal judgment, and a service abnormality alarm is sent to the upper-level platform. This mechanism can constrain the output of the multimodal large model in the cloud to standardized structured data, thereby improving the stability and availability of the system engineering.

[0030] The edge intelligence processing unit also includes a post-processing and visualization module. This module is used to: convert normalized bounding box coordinates into pixel coordinates in the original image; perform semantic enhancement annotation on the bounding boxes based on anomaly type labels and risk level information in the structured anomaly detection results, with the semantic enhancement annotation including at least color coding, text labels, and risk level indicators; perform temporal correlation analysis on anomaly regions in multiple consecutive frames of images to calculate the stability index of the anomaly regions; and correct the confidence level of the anomaly detection results based on the stability index, generating a dynamic visualization result containing anomaly change trend information, which is then displayed on the human-computer interaction interface.

[0031] like Figure 3 As shown, by overlaying color coding and text labels, abnormal areas and their risk levels can be visually displayed. High-risk anomalies are marked with prominent colors, facilitating quick identification and decision-making by operators. Figure 3 As shown in section a, during the detection process in a localized area of ​​the workbench, the system can identify improper handling of graduated cylinders stored in beakers and the abnormal state of graduated cylinders not being capped. The former may lead to reagent spillage and container breakage, while the latter may cause reagent evaporation, contamination, or accidental contact risks. Figure 3As shown in part b, during the overall inspection of the workbench area, the system further identified several typical violations, such as lack of protection for the robotic arm, improper cable management, and disorderly placement of items. Among these, the lack of protective devices on the robotic arm may lead to the risk of pinching or collision, disorderly stacking of cables may cause tripping or electrical faults, and disorderly placement of items will increase interference and safety hazards during the experimental operation.

[0032] The robot motion control unit receives structured anomaly detection results and extracts anomaly type labels, risk level information, and normalized bounding box coordinates of the anomaly region in the image coordinate system. Combining this with the current attitude angle fed back by the gimbal mechanism and the current global physical coordinates fed back by the positioning and navigation module, it calculates the target physical coordinates of the anomaly region in the laboratory's global physical coordinate system using a coordinate projection transformation algorithm. This drives the AGV chassis to move to a preset range of the target position. Furthermore, using the anomaly type label as an index, it queries a pre-stored emergency response mapping table to obtain an emergency response instruction sequence and executes the corresponding actions. This process achieves a mapping from image space to physical space, enabling the robot to proactively respond to detected anomalies.

[0033] The coordinate projection transformation algorithm in the robot motion control unit specifically executes the following calculation logic: Obtain the intrinsic parameter matrix of the vision sensor and extrinsic parameter matrix ,in, For rotation matrix, The translation vector is used; the extrinsic parameter matrix is ​​determined by the current attitude angle fed back by the gimbal mechanism and the robot's current global physical coordinates fed back by the positioning and navigation module.

[0034] Extract the center point of the bounding box of the abnormal region as the target point, and mark the coordinates of the target point in the image as follows: ; Based on the pinhole camera model, calculate the normalized ray direction vector. And construct a parameterized spatial ray expression for the target point in the laboratory's global physical coordinate system: ,in, The scale parameter along the ray is used to determine the specific location of the target point on the ray.

[0035] If the vision sensor uses a depth camera to acquire depth information, then it will be compared with the coordinates of the target point. Corresponding depth value As a scale parameter, the three-dimensional physical coordinates of the target point are calculated; If the vision sensor uses a visible light camera and does not acquire depth information, the intersection point of the ray and the reference plane is solved by a preset reference plane to determine the scale parameters and calculate the two-dimensional physical coordinates of the target point.

[0036] The above method establishes a mapping relationship between the image coordinate system and the laboratory's global physical coordinate system based on a pinhole imaging model. When a depth camera is used, the three-dimensional spatial position of the target point can be directly obtained, thereby improving positioning accuracy. When depth information is lacking, the spatial position is approximated through reference plane constraints, thus ensuring system availability while being compatible with different types of vision sensor configurations.

[0037] The positioning and navigation module is configured to perform adaptive path planning and dynamic obstacle avoidance. Specifically, it includes: performing global path planning based on a pre-built laboratory topology map and continuously checking environmental perception data during the inspection process; when environmental changes or dynamic obstacles are detected that obstruct the current task, local path replanning is triggered, the original task is canceled, and a new travel sequence is generated from the currently reachable nodes; at the same time, the access order of the inspection path is dynamically weighted according to the risk level of abnormal areas, so that high-risk abnormal areas receive higher priority in the path queue, thereby realizing risk-driven adaptive inspection scheduling.

[0038] In implementation, the system first loads a pre-built laboratory topology map, which includes node coordinates and graph topology. Nodes correspond to key locations such as lab benches, fume hoods, and reagent cabinets, while edges represent walkable paths. Figure 1 Taking the laboratory layout shown as an example, the topology diagram may include node A (east side lab bench), node B (fume hood), node C (west side reagent cabinet), etc. The connection relationship between each node is pre-marked according to the actual passage layout.

[0039] The positioning and navigation module uses LiDAR-based SLAM technology to achieve centimeter-level positioning and communicates with an external robot scheduling system (such as Hikvision RCS-2000) via HTTP protocol. In practice, the system sends a path point sequence to the scheduling system through a task publishing interface, setting the task type parameter to "ZKD" (autonomous inspection) and the location code type to "00" (normal path point). After the scheduling system returns the task code, the system periodically queries the task status at a frequency of 1Hz and parses the returned status code: status code "9" indicates the task is completed normally, while status code "1" or "2" indicates the task is obstructed (e.g., the path is blocked by an obstacle). When the number of consecutive obstructions exceeds a preset threshold (e.g., 25 times), the system determines that the current path is impassable, calls the task cancellation interface to forcibly terminate the current task, skips the obstructed node, and regenerates the travel sequence from the currently reachable node.

[0040] For risk-driven adaptive inspection scheduling, the system maintains a path priority queue, where each node is associated with a risk weight coefficient. During normal inspections, all nodes have a weight coefficient of 1. When an area is identified as having a high-risk anomaly (such as an H2S gas leak), the system adjusts the weight coefficient of the corresponding node in that area to 10, significantly increasing its priority in the path queue. The path planning algorithm uses a weighted Dijkstra's algorithm, comprehensively considering path length and node weights to calculate the optimal access order. For example, if a high-risk anomaly is detected in the fume hood on the east side, even if the node is not on the original inspection path, the system will insert it at the head of the queue for priority handling.

[0041] Through the above mechanism, the system can sense environmental changes in real time and dynamically adjust the path, effectively deal with temporary obstacles in the laboratory (such as mobile carts and people walking around), and ensure that high-risk abnormal areas receive priority response, thereby significantly improving inspection efficiency and emergency response capabilities.

[0042] The robot motion control unit also includes a gimbal control module. This module dynamically adjusts the target attitude angle of the gimbal mechanism based on the deviation between the center coordinates of the bounding box of the abnormal region and the center coordinates of the image, ensuring that the optical axis of the vision sensor is aligned with the physical spatial position corresponding to the abnormal region. Simultaneously, based on the center point of the bounding box of the abnormal region in the image fed back by the vision sensor, the gimbal control module constructs a closed-loop control circuit to continuously track the abnormal target, keeping the abnormal region always in the center of the vision sensor's field of view. This closed-loop control mechanism continuously corrects deviations to achieve stable target tracking, thereby improving detection and positioning accuracy.

[0043] The emergency response mapping table in the robot motion control unit includes a mapping relationship between anomaly types and multi-level response strategies. The robot motion control unit also selects the corresponding response strategy based on the anomaly type label and risk level information. Response strategies include equipment linkage control, alarm triggering, and remote notification. Equipment linkage control includes the control of ventilation systems, gas shut-off devices, and safety protection equipment. By constructing a mapping relationship between anomaly types and response strategies, automated hierarchical response can be achieved. For example, high-risk anomalies trigger linkage control and alarms, while low-risk anomalies only record and provide notifications, thus ensuring safety while avoiding over-response.

[0044] Experimental results show that the system of this invention can stably identify and rapidly respond to multiple types of anomalies in a real chemical laboratory environment, demonstrating good environmental adaptability and engineering application value. In the deployment verification on two automated chemical synthesis platforms, the dataset contained 5000 laboratory scene images covering 15 types of anomalies. The overall anomaly identification accuracy of the system was 92.7%, the false alarm rate was 5.3%, the false negative rate was 2.0%, the average detection time per point was 2.8 seconds, the response time for toxic and harmful gas detection was 4.5 seconds, and the inspection coverage rate was 93%. The system accumulated over 100 hours of trouble-free operation, successfully issuing warnings and assisting in the handling of multiple safety hazards. This embodiment is only a preferred implementation method; any equivalent substitutions or modifications based on the technical solution of this invention should fall within the protection scope of this invention.

Claims

1. A multimodal intelligent inspection robot system for chemical laboratories, characterized in that, include: Autonomous mobile robot platform, multimodal intelligent sensing unit, edge intelligent processing unit, and robot motion control unit; among which, The autonomous mobile robot platform includes: an AGV chassis, an industrial control cabinet, a human-machine interface, a lifting device, and a gimbal mechanism. The AGV chassis is located at the bottom of the platform and drives the robot's movement; a positioning and navigation module is integrated within the chassis. The industrial control cabinet is located above the AGV chassis and houses the edge intelligent processing unit and the robot motion control unit. A human-machine interface is located on the front of the cabinet, displaying inspection results and supporting user interaction control. A lifting device is located on the top of the cabinet, and a gimbal mechanism with horizontal and pitch degrees of freedom is mounted on the lifting device. The multimodal intelligent sensing unit includes a visual sensor and an environmental sensor; the visual sensor is installed on the rotating end of the gimbal mechanism and is used to collect visual image data of the laboratory scene; the environmental sensor is installed on the side of the industrial control chassis and is used to collect environmental parameter data of the laboratory. The edge intelligent processing unit is used to receive the visual image data and the environmental parameter data, and synchronize the data through a multi-threaded architecture; then, it concatenates the synchronized data with a pre-stored strong constraint prompt word template to generate an anomaly detection request, and then sends the anomaly detection request to an external cloud multimodal large model through wireless communication; at the same time, it receives and parses the structured anomaly detection results returned by the cloud multimodal large model, and sends the structured anomaly detection results to the robot motion control unit; After receiving the structured anomaly detection result, the robot motion control unit extracts the anomaly type label, risk level information, and normalized bounding box coordinates of the anomaly region in the image coordinate system. Combined with the current attitude angle fed back by the gimbal mechanism and the current global physical coordinates fed back by the positioning and navigation module, it calculates the target physical coordinates of the anomaly region in the laboratory global physical coordinate system through a coordinate projection transformation algorithm. This drives the AGV chassis to move to a preset range of the target position. Using the anomaly type label as an index, it queries a pre-stored accident emergency response mapping table to obtain an emergency response instruction sequence and executes the corresponding response strategy.

2. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The multi-threaded architecture of the edge intelligent processing unit includes: a visible light video acquisition thread, an infrared video acquisition thread, and an environmental data acquisition thread; after communicating with each thread through an asynchronous message queue, the visual image data and the environmental parameter data are aligned based on timestamps to obtain synchronized data; the edge intelligent processing unit constructs a unified time base, thereby using a sliding time window to cache and resample the synchronized data.

3. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The edge intelligent processing unit's parsing of the structured anomaly detection results includes: structure verification and anomaly correction processing. The structure verification includes: field integrity verification, data type verification, and boundary value validity verification. The anomaly correction processing includes: completing missing fields based on preset rules, trimming and correcting abnormal coordinates, reconstructing and parsing response data that does not conform to the format, and triggering a retry mechanism when multiple verifications fail.

4. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The edge intelligent processing unit also includes a post-processing and visualization module; The post-processing and visualization module is used to convert the normalized bounding box coordinates into pixel coordinates in the original image; Based on the anomaly type labels and risk level information in the structured anomaly detection results, the normalized bounding boxes are semantically enhanced and annotated. The semantically enhanced annotation includes at least: color coding, text labels, and risk level indicators. Temporal correlation analysis is performed on abnormal regions in multiple consecutive frames of images to calculate the stability index of the abnormal regions; and the confidence of the anomaly detection results is corrected based on the stability index to generate dynamic visualization results containing information on the anomaly change trend, which are then displayed on the human-computer interaction interface.

5. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The coordinate projection transformation algorithm in the robot motion control unit performs the following calculation logic: Obtain the intrinsic parameter matrix of the vision sensor and extrinsic parameter matrix ,in, For rotation matrix, The translation vector is used; the extrinsic parameter matrix is ​​determined by the current attitude angle fed back by the gimbal mechanism and the robot's current global physical coordinates fed back by the positioning and navigation module. Extract the center point of the bounding box of the abnormal region as the target point, and mark the coordinates of the target point in the image as follows: ; Based on the pinhole camera model, calculate the normalized ray direction vector. And construct a parameterized spatial ray expression for the target point in the global physical coordinate system of the laboratory scene: ,in, The scale parameter along the ray is used to determine the specific location of the target point on the ray; If the vision sensor uses a depth camera to acquire depth information, then the coordinates of the target point will be compared with those of the depth sensor. Corresponding depth value The scale parameter is used to calculate the three-dimensional physical coordinates of the target point; If the visual sensor uses a visible light camera and does not acquire depth information, the intersection point of the ray and the reference plane is solved by a preset reference plane to determine the scale parameter and calculate the two-dimensional physical coordinates of the target point.

6. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The positioning and navigation module is configured to perform adaptive path planning and dynamic obstacle avoidance, specifically including: performing global path planning based on a pre-built topology map of the laboratory scene, and continuously checking environmental perception data during the inspection process; when environmental changes or dynamic obstacles are detected that obstruct the current task, local path replanning is triggered, the original task is canceled, and a new travel sequence is generated from the currently reachable nodes; at the same time, the access order of the inspection path is dynamically weighted and adjusted according to the risk level of the abnormal area, so that high-risk abnormal areas get higher priority in the path queue, thereby realizing risk-driven adaptive inspection scheduling.

7. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The robot motion control unit also includes a gimbal control module; The gimbal control module dynamically adjusts the target attitude angle of the gimbal mechanism based on the deviation between the center coordinates of the bounding box of the abnormal region and the center coordinates of the image, so that the optical axis of the vision sensor is aligned with the physical spatial position corresponding to the abnormal region. At the same time, the gimbal control module constructs a closed-loop control loop based on the center point position of the bounding box of the abnormal region in the image fed back by the vision sensor, and continuously tracks the abnormal target in the abnormal region, so that the abnormal region is always kept in the center area of ​​the field of view of the vision sensor.

8. The multimodal intelligent inspection robot system for chemical laboratories according to claim 1, characterized in that, The accident emergency response mapping table in the robot motion control unit is a mapping relationship table between anomaly types and multi-level response strategies. The robot motion control unit selects the corresponding response strategy according to the anomaly type label and risk level information. The handling strategy includes: equipment linkage control, alarm triggering, and remote notification. The equipment linkage control includes: control of the ventilation system, gas cut-off device, and safety protection equipment.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the computer program to implement the multimodal intelligent inspection robot system for chemical laboratories as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal intelligent inspection robot system for chemical laboratories as described in any one of claims 1 to 8.

Images