Intelligent robot anomaly detection method and system based on visual language model

Abstract

The application provides an intelligent robot anomaly detection method and system based on a visual language model. The method comprises: acquiring visual image data and laser radar point cloud data of an intelligent robot driving in an industrial scene; inputting the visual image data into a pre-trained visual language model for feature encoding to obtain an image visual feature vector, the visual language model being obtained by fine-tuning a general visual language model based on a visual question and answer data set; identifying industrial anomalies based on fused perception features to obtain an anomaly detection result; judging the feasibility of passing based on the anomaly detection result and the state information of the intelligent robot to generate a passing decision instruction; and controlling the intelligent robot to perform an obstacle avoidance action or continue driving along the original navigation path based on the passing decision instruction and a preset path planning strategy. The application improves the accuracy of anomaly detection and the rationality of passing decisions in complex industrial scenes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot perception and obstacle avoidance technology, specifically relating to an anomaly detection method and system for intelligent robots based on visual language models. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, industrial intelligent robots (such as Automated Guided Vehicles (AGVs), handling robots, and inspection robots) are increasingly widely used in warehousing, logistics, and manufacturing. One of the core capabilities of industrial intelligent robots is their ability to navigate autonomously and avoid obstacles in complex industrial environments, which directly affects the safety and efficiency of industrial production.

[0003] In existing technologies, obstacle avoidance methods for industrial intelligent robots mainly rely on single sensors or simple combinations of sensing solutions. Common sensing methods include: This solution uses LiDAR to acquire environmental point cloud data and detect obstacles ahead. While it can obtain accurate spatial distance information, it struggles to identify the semantic category of obstacles, failing to distinguish between "suspended obstacles" and "ground-based obstacles," or to identify anomalies with specific passage significance, such as "holes in barbed wire." When the LiDAR point cloud forms a hole in a barbed wire area, traditional solutions easily misinterpret it as a solid wall, causing unnecessary obstacle stops for the robot and impacting operational efficiency.

[0004] This approach captures environmental images using cameras and identifies obstacles based on image processing or deep learning-based object detection methods. While it can acquire rich visual semantic information, its detection accuracy is unstable due to interference from complex industrial environments such as varying lighting conditions, glass reflections, and dust obstruction. Particularly in low-light industrial scenarios with reflective materials, the visual solution exhibits high false positive and false negative rates.

[0005] Simply stitching or weighting the data collected by LiDAR and visual cameras together is insufficient; this fusion method remains at a basic level, only at the data or feature level, failing to achieve cross-modal semantic alignment and reasoning. Due to a lack of semantic understanding of abnormal obstacles in industrial scenarios, this type of solution cannot achieve cognitive-level reasoning from "detecting an anomaly" to "understanding whether the anomaly affects passage."

[0006] In summary, existing technologies have the following technical problems: traditional solutions can only detect the presence of obstacles, but cannot understand the semantic category of obstacles and their actual impact on passage; they are difficult to accurately identify special industrial anomalies such as wire mesh cutouts, glass reflections, suspended obstacles, reflective material interference, low-light occlusion, and dynamic occlusion misjudgments; the fusion of vision and LiDAR remains at the feature stitching level and has not achieved semantic-level cross-modal alignment and reasoning; and it is impossible to determine whether passage is possible based on the relative relationship between the obstacle's height above the ground and the robot's cargo height.

[0007] Therefore, the technical problem to be solved by the present invention is to provide an intelligent robot anomaly detection method and system based on a visual language model, so as to solve the technical problems of insufficient semantic understanding ability, high industrial anomaly misjudgment rate, insufficient cross-modal fusion depth and lack of feasibility judgment ability in the prior art. Summary of the Invention

[0008] The purpose of this application is to provide an intelligent robot anomaly detection method and system based on visual language models, in order to solve the technical problems in the existing technology, which are that due to the complex background of industrial scenes and the limitations of multimodal perception, intelligent robot anomaly recognition methods that rely on traditional geometric detection are difficult to distinguish between special obstacles such as hollows and reflections and non-target objects, resulting in high false alarm rate and high false negative rate of anomaly detection, and lack of semantic level passage decision-making ability.

[0009] To achieve the above objectives, a specific embodiment of the present invention provides an anomaly detection method for intelligent robots based on a visual language model. The method includes: acquiring visual image data and lidar point cloud data of an intelligent robot moving in an industrial scene. The visual image data is input into a pre-trained visual language model, and the input image is feature-encoded based on the visual language model to obtain an image visual feature vector. The visual language model is obtained by performing domain adaptation fine-tuning on a general visual language model based on a visual question-and-answer dataset. The visual question-and-answer dataset contains multiple sets of questions and answers, images, and abnormal label samples. Based on the image visual feature vector, the geometric features extracted from the lidar point cloud data are fused, and cross-modal feature alignment is performed in the multimodal fusion module to generate fused perception features. Based on the fused perception features, the anomaly detection module identifies industrial abnormal obstacles in the current driving environment and obtains anomaly detection results. The industrial abnormal obstacles include at least one of the following: wire mesh perforation, glass reflection, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment. Based on the anomaly detection results, and combined with the intelligent robot's own state information, a feasibility judgment is made for passage, and a passage decision instruction is generated. The passage decision instruction includes a stop decision instruction, a deceleration decision instruction, or a forward decision instruction. Based on the passage decision command and the preset path planning strategy, the obstacle avoidance execution module controls the intelligent robot to perform obstacle avoidance actions or continue to travel along the original navigation path.

[0010] This invention acquires visual image data and LiDAR point cloud data, inputs them into a domain-adapted, fine-tuned visual language model for feature encoding, and then performs multimodal feature fusion, anomaly recognition, feasibility assessment, and obstacle avoidance, forming a complete "perception-understanding-decision-execution" closed loop. Compared with existing technologies that rely solely on LiDAR geometric detection or single visual detection, this method introduces the cross-modal semantic understanding capability of the visual language model into the field of anomaly detection for intelligent robots. This allows the robot not only to "see" obstacles but also to "understand" the category attributes of the obstacles and their actual impact on passage. This effectively solves the problems of missed detection and misjudgment caused by special anomalies such as wire mesh perforations and glass reflections in existing technologies, significantly improving the accuracy of anomaly detection and the rationality of passage decisions in complex industrial scenarios.

[0011] In one or more embodiments of the present invention, the visual language model is the LLaVA-7B base model, the visual question answering dataset is the self-built AED-VQA dataset, which contains more than 3,000 sets of question-answer, image and anomaly label samples, and the visual language model is obtained by performing domain adaptation fine-tuning on the AED-VQA dataset.

[0012] This invention provides a clear technical path for domain-specific fine-tuning of visual language models by limiting the general visual language model to the LLaVA-7B base model and the visual question-answering dataset to a self-built dataset containing more than 3,000 sets of question-answer, image, and anomaly label samples. LLaVA-7B, as a mature general visual language model, has a solid foundation in cross-modal understanding; while the self-built high-quality visual question-answering dataset specifically covers and semantically annotates high-frequency anomalies in industrial scenarios. This allows the fine-tuned model to accurately learn the mapping relationship between the visual features of industrial anomalies and common semantics, thereby significantly improving the recognition accuracy of specific industrial anomalies while maintaining general understanding capabilities.

[0013] In one or more embodiments of the present invention, the training strategy of the visual language model includes the following steps: General vision-language pre-training is performed based on a general vision-language dataset. The feature distance between matching problems and image pairs is narrowed through image-text comparison learning. Masked language modeling is used to enhance the model's understanding of keywords in industrial scenarios. The consistency of cross-modal feature space is constrained by the loss function. Domain-adaptive pre-training is performed based on a subset of the visual question-answering dataset. The model is trained to focus on abnormal regions through bounding box regression and region feature extraction. Cross-modal completion training is used to strengthen the mapping relationship between visual features and abnormal semantics. Industrial noise such as dust blur, uneven lighting, and reflective spots is injected into the training images to improve anti-interference ability. End-to-end fine-tuning was performed based on the full dataset of the visual question answering dataset. A multi-task joint loss function was used to balance the anomaly type classification loss and the anomaly key parameter regression loss. The anomaly type classification used weighted cross-entropy loss, and the anomaly parameter regression used Smooth L1 loss.

[0014] This invention employs a three-stage progressive training strategy to fine-tune a visual language model. The first stage involves general pre-training to build basic cross-modal association capabilities. The second stage involves domain-adaptive pre-training, injecting industrial anomaly recognition knowledge and noise to enhance robustness. The third stage involves full end-to-end fine-tuning to optimize recognition accuracy, forming a progressive optimization path from coarse to fine, and from general to specific. Compared to single-stage fine-tuning, this strategy avoids the catastrophic forgetting of general knowledge, enhances the model's resistance to interference in real-world complex environments through industrial noise injection, and balances classification accuracy and parametric regression accuracy through a multi-task joint loss function, ultimately achieving a significant improvement in anomaly detection accuracy.

[0015] In one or more embodiments of the present invention, when a suspended obstacle is detected, the following step is further included to determine the feasibility of passage: Obtain the current cargo height of the intelligent robot; Obtain the camera's intrinsic parameter matrix and the camera's rotation matrix and translation vector relative to the world coordinate system, and establish an imaging geometric model from spatial points to the image plane based on perspective projection relationships; Calculate the pixel height difference based on the bottom and top pixel coordinates of the suspended obstacle in the image; The distance to the ground intersection line is estimated based on the camera's installation height and pitch angle, and then the height of the suspended obstacle above the ground is estimated. The height of the suspended obstacle above the ground is compared with the current cargo height of the intelligent robot: when the height of the suspended obstacle above the ground is greater than the current cargo height, it is determined to be passable, and a forward decision command is output; when the height of the suspended obstacle above the ground is less than or equal to the current cargo height, it is determined to be impassable, and a stop decision command is output.

[0016] This invention addresses the problem in existing technologies where AGVs cannot autonomously determine whether suspended obstacles impede passage by designing a dedicated feasibility assessment step. Specifically, it estimates the obstacle's height above the ground based on a camera imaging geometric model and compares this estimate with the robot's current cargo height to generate a decision to proceed or stop. This method eliminates the need for expensive depth sensors or complex LiDAR modeling, achieving accurate height estimation solely through the perspective projection relationship of a monocular camera. It features low computational load, high real-time performance, and fully considers the dynamic changes in the robot's cargo state, enabling the robot to make accurate passage decisions when encountering height restriction bars, roller shutters, suspended cables, etc., avoiding unnecessary stopping or collision risks.

[0017] In one or more embodiments of the present invention, when generating a passage decision instruction based on the anomaly detection result, the following steps are included: The spatial position relationship of detected abnormal obstacles is determined to distinguish whether the abnormal obstacles are on the main forward navigation channel of the intelligent robot; When an abnormal obstacle is determined to be located in the main passage, the abnormality is marked as a forward obstruction, and a stop decision command or a deceleration decision command is generated based on the passage feasibility assessment results. When an abnormal obstacle is determined to be located in a lateral or non-main passage area, the abnormality is marked as a lateral disturbance, a forward decision instruction is generated, and the intelligent robot is allowed to continue to pass along the main passage. When an abnormal obstacle is determined to be located on the main forward passage and is deemed impassable, local path replanning is initiated to generate a sequence of detour path points.

[0018] This invention introduces a "main channel priority" spatial positional relationship determination when generating passage decision instructions. It distinguishes abnormal obstacles into frontal obstructions and lateral interference, and adopts different strategies for stopping / decelerating around or continuing forward, respectively. This avoids the inefficiency of traditional methods that "alternate around every obstacle." This strategy fully utilizes the abnormal location information provided by multimodal fusion perception, combined with the robot's navigation path for intelligent judgment. This enables the robot to distinguish between anomalies that truly obstruct the main channel and lateral interference that does not interfere with passage, thereby significantly reducing unnecessary detour lengths and turning times, and improving the continuity and overall operational efficiency of logistics transportation.

[0019] In one or more embodiments of the present invention, the cross-modal completion training in the domain adaptation pre-training specifically includes: Randomly mask out some semantic elements in the question text to obtain an incomplete question text; The incomplete problem text and corresponding image features are input into the visual language model. Based on the cross-entropy loss constraint model, the accuracy of the anomaly type is improved according to the image features, thereby strengthening the mapping relationship between visual features and anomaly semantics. In the end-to-end fine-tuning, the multi-task joint loss function is specifically as follows: L total = λ 1 L cls + λ 2 L reg ; in, L cls Weighted cross-entropy loss for anomaly type classification, λ 1 = 0.7;L reg The regression loss component for regressing outlier key parameters. λ 2=0.3; To address the imbalanced sample problem, higher weights are assigned to outlier categories with fewer samples for training optimization.

[0020] This invention significantly improves the model's accurate identification of anomaly categories by limiting the specific operations of cross-modal completion training and the weight configuration of the multi-task joint loss function. Specifically, it randomly masks semantic elements of the problem, requiring the model to complete the anomaly type. Furthermore, it sets the classification loss weight λ1=0.7 and the regression loss weight λ2=0.3, assigning higher weights to classes with fewer samples. Cross-modal completion training strengthens the mapping relationship between visual features and anomaly semantics, enabling the model to accurately infer anomaly types even with only image input. The weighted loss function effectively addresses the problem of insufficient training on low-frequency samples such as suspended obstacles, ensuring the model's balanced recognition ability across all six types of industrial anomalies, thereby comprehensively reducing the false negative rate of anomaly detection.

[0021] In one or more embodiments of the present invention, the process of acquiring lidar point cloud data and extracting geometric features includes the following steps: Preprocess the point cloud data collected by lidar to remove outliers and elevation noise; Extract geometric feature parameters from point cloud data, including reflectance features, spatial density distribution features, and three-dimensional coordinate position features; The extracted geometric feature parameters are concatenated with the image visual feature vector in a unified dimensional space, and the multimodal features are weighted and fused through an attention mechanism to generate fused perceptual features.

[0022] This invention specifically defines the geometric feature extraction steps for LiDAR point cloud data, including removing outliers and elevation noise, extracting reflectivity features and spatial density distribution features, etc. It then uses an attention mechanism to weightedly fuse geometric features with image visual features, enabling multimodal fusion perception features to retain rich texture information from the image while supplementing the precise geometric structure information of the point cloud. The attention mechanism can adaptively adjust the fusion weights of different modal features. For different abnormal scenarios such as wire mesh cutouts and glass reflections, the model can dynamically focus on more reliable modal features, thereby improving the expressive power and robustness of the fused features.

[0023] In one or more embodiments of the present invention, the anomaly detection module further includes a semantic question-answering reasoning unit, which performs cognitive-level semantic reasoning on industrial anomaly scenarios based on question-answering format: It receives text queries containing descriptions of abnormal scenarios and performs cross-modal reasoning based on fused perceptual features and text semantics; Output the anomaly type judgment result and the corresponding semantic label of passage meaning, realizing cognitive-level reasoning from detecting the existence of an anomaly to understanding whether the anomaly affects passage.

[0024] In one or more embodiments of the present invention, identifying abnormal wire mesh perforations includes the following steps: Analyze the visual texture features of the wire mesh cutout area in the image to identify the regular grid dot matrix structure; By combining the characteristics of regular holes and reflectivity below a preset threshold in the hollowed-out area of ​​the lidar point cloud, multimodal feature joint judgment is performed. Exclude cases where point cloud voids caused by wire mesh perforations are mistakenly identified as solid walls, output a wire mesh perforation anomaly type label, and determine that the anomaly does not block the main passage.

[0025] In another aspect of the present invention, a visual language model-based intelligent robot anomaly detection system is provided for implementing the steps of the visual language model-based intelligent robot anomaly detection method described in the claims, wherein the visual language model-based intelligent robot anomaly detection system includes: The data acquisition module is used to perform the steps of acquiring visual image data and lidar point cloud data when the intelligent robot is moving in an industrial scene; The feature encoding module is used to perform the steps of inputting the visual image data into a pre-trained visual language model, and encoding the features of the input image based on the visual language model to obtain the image visual feature vector, wherein the visual language model is obtained by performing domain adaptation fine-tuning on a general visual language model based on a visual question answering dataset; The multimodal fusion module is used to perform the steps of fusing geometric features extracted from the lidar point cloud data based on the image visual feature vector, aligning cross-modal features in the multimodal fusion module, and generating fused perception features. Anomaly detection module is used to perform the step of identifying industrial abnormal obstacles in the current driving environment based on the fused perception features and obtaining anomaly detection results. The industrial abnormal obstacles include at least one of wire mesh perforation, glass reflection, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment. The passage decision module is used to execute the step of judging the feasibility of passage based on the anomaly detection result and the intelligent robot's own state information, and generating a passage decision instruction. The passage decision instruction includes a stop decision instruction, a deceleration decision instruction, or a forward decision instruction. The obstacle avoidance execution module is used to execute the steps of the intelligent robot to perform obstacle avoidance actions or continue to travel along the original navigation path based on the passage decision command and the preset path planning strategy. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an anomaly detection method for intelligent robots based on a visual language model according to one embodiment of the present invention; Figure 2 This is a graph showing the change of the loss function with the number of iterations in one embodiment of the present invention; Figure 3 This is a heatmap showing the detection accuracy in one embodiment of the present invention; Figure 4 This is a block diagram of an intelligent robot anomaly detection device based on a visual language model according to one embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0029] Please see the appendix Figure 1 This application provides an anomaly detection method for intelligent robots based on a visual language model. The method is executed by an intelligent robot anomaly detection system based on a visual language model, and specifically includes the following steps: Step S1: Acquire visual image data and lidar point cloud data of the intelligent robot as it moves in the industrial scene.

[0030] Specifically, taking Automated Guided Vehicles (AGVs) as an example, the intelligent robot is equipped with visual sensors and LiDAR. The visual sensor uses a monocular camera. When operating in industrial settings, including warehousing and production lines, the camera collects environmental image data, while the LiDAR collects point cloud data. The visual image data is used to extract texture and color information, while the LiDAR point cloud data is used to obtain geometric structure and reflectivity information. The two sensors complement each other, providing a data foundation for subsequent multimodal fusion. The intelligent robot performs transportation tasks according to the production rhythm, collecting data at a frame rate of 30fps, ensuring that the pixel size of abnormal targets in the image is sufficient to support high-precision detection.

[0031] Table 1 below shows the results of the relevant scenario survey in this embodiment, demonstrating example scenarios and perception failures of six types of anomalies: wire mesh cutouts, glass reflections, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment.

[0032] Table 1. Summary of Scenario Survey Results

[0033] Step S2: Input the visual image data into the pre-trained visual language model, and encode the features of the input image based on the visual language model to obtain the image visual feature vector.

[0034] The visual language model is obtained by fine-tuning a general visual language model based on a visual question-answering dataset for domain adaptation. The visual question-answering dataset contains multiple sets of question-answer, image, and anomaly label samples. In this embodiment, the general visual language model uses the LLaVA-7B base model. The visual question-answering dataset is a self-built AED-VQA dataset, constructed as follows: First, through industrial site surveys and fault log analysis, 3000 AGV abnormal shutdown event records from 2022 to 2024 were collected. Root cause analysis revealed that environmental perception failure accounted for 72% of the event causes, involving warehousing scenarios, production line scenarios, and special environments. Then, text mining was performed on the 3000 event records. Text mining was based on the BERT model to extract key features, including keywords such as metal mesh, glass reflection, suspension, and low light. These features were combined with the fault modes of the AGV perception module, including point cloud emptiness. Holes, image overexposure, and depth jumps were identified as the six most frequent abnormal shutdown scenarios, including wire mesh cutouts, glass reflections, suspended obstacles, reflective material interference, low-light occlusion, and dynamic occlusion misjudgments. These scenarios account for more than 85% of industrial abnormal shutdown events. Finally, environmental images and point cloud data under these scenarios were collected and multi-level annotated. The annotations included bounding boxes or region segmentation annotations for target localization, as well as scene type labels and risk level labels for semantic reasoning. Based on the annotated data, a visual question-answering dataset in the form of question-image-answer triples was constructed, containing more than 3,000 sets of question-answer-image-abnormal label samples.

[0035] The visual language model referred to in this application as the anomaly detection visual language model is fine-tuned through a three-stage progressive training strategy.

[0036] The first phase is general pre-training: based on a large-scale general vision-language dataset, pre-training is performed using LLaVA-7B as the base model. The large-scale general vision-language dataset includes 3.3 million image-text pairs from Conceptual Captions, LAION-5B, and an industry-related subset of COCO Captions. Training tasks include image-text contrast learning, masked language modeling, and feature alignment constraints. Image-text contrast learning uses the InfoNCE loss function to shorten the feature distance between matching problem-image pairs and widen non-matching pairs, improving the model's understanding of the semantic requirements and the association between visual data. Masked language modeling randomly masks industry terms in the problem at a 15% masking rate. Industry terms include aperture and reflectivity. Masked words are predicted through contextual features, strengthening the model's understanding of AGV scene keywords. Feature alignment constraints minimize the distance between the problem's semantic features and the image's visual features using the L2 loss function, ensuring consistency across modal feature spaces. This phase has 10 training cycles with a learning rate of 5e-5 and does not involve anomaly recognition tasks; it only lays the foundation for subsequent domain adaptation.

[0037] The second stage is domain-adaptive pre-training: based on a 20% subset of the AED-VQA dataset, AGV scene anomaly recognition knowledge is injected. This subset covers all six anomaly categories, with approximately 100 sets of question-image-anomaly label samples for each category. Training tasks include anomaly feature localization pre-training, cross-modal completion training, and industrial noise injection. Anomaly feature localization pre-training takes question-image pairs as input and trains the model's ability to focus on anomaly regions through bounding box regression and region feature extraction. Bounding box regression uses IoU loss to reduce missed detections due to inaccurate region localization. Cross-modal completion training randomly masks some semantic elements in the question, for example, masking "Does the image contain wire mesh cutouts?" as "Does the image contain...". The model is required to complete the anomaly type based on image features, using cross-entropy loss to constrain completion accuracy and strengthen the mapping relationship between visual features and anomaly semantics. Industrial noise injection adds typical industrial scene noise to the training images, including dust blurring, uneven lighting, and reflective spots, simulating a real operating environment and improving the model's anti-interference ability. This phase involves 8 training cycles with a learning rate of 2e-5. The first 10 layers of the text encoder and image encoder of the base model are frozen, and only the subsequent fusion and output layers are fine-tuned to avoid degradation of general visual-language capabilities.

[0038] The third stage is full-scale fine-tuning: The model is fine-tuned end-to-end using the full AED-VQA dataset, focusing on improving anomaly detection accuracy. The AED-VQA full dataset contains 3000 sets of question-image-anomaly label samples, covering diverse scenarios across 6 anomaly categories, including wire mesh with different apertures and glass reflections at different angles. The multi-task joint loss function is designed as follows: L total = λ 1 L cls + λ 2 L reg ; in, L cls Weighted cross-entropy loss for anomaly type classification, λ 1 = 0.7; L reg The regression loss components for the abnormal key parameters, λ2=0.3; To address the imbalanced sample problem, higher weights are assigned to anomaly categories with fewer samples for training optimization. This regression loss component uses squared form to calculate the loss when the absolute value of the prediction error is less than a preset threshold, and absolute value form when the absolute value of the prediction error is greater than or equal to the preset threshold. Regression errors for parameters such as aperture and reflective area are constrained to ensure that the parameters match the actual scene. After training, the model achieves an anomaly detection accuracy of 92.7% and an inference latency of only 50 milliseconds, representing a significant improvement compared to traditional methods and general visual language models.

[0039] Please see the appendix Figure 2 During model training, the loss function first rises rapidly to a peak as the number of iterations increases, reaching its peak around the 45th epoch, and then gradually decreases. This indicates that the model explores the optimal parameters in the early stages of training and then gradually converges, with the fitting effect continuously improving.

[0040] During feature encoding, the model scales the input image to 640×640 pixels, extracts multi-scale feature maps using a visual encoder employing ViT, and maps the feature maps to a semantic feature space to obtain the image visual feature vector. After the above three-stage training, the anomaly detection visual language model described in this application is obtained, hereinafter referred to as the AbnormalVL-AGV model.

[0041] Step S3: Based on the image visual feature vector, fuse the geometric features extracted from the LiDAR point cloud data, perform cross-modal feature alignment in the multimodal fusion module, and generate fused perception features.

[0042] Specifically, the point cloud data acquired by the LiDAR is preprocessed, including the removal of outliers and elevation noise. Geometric feature parameters are extracted from the point cloud data, including reflectance features, spatial density distribution features, and three-dimensional coordinate position features. Then, the extracted geometric feature parameters are concatenated with the image visual feature vector in a unified dimensional space. A weighted fusion of multimodal features is performed using an attention mechanism, employing multi-head attention. Specifically, the image visual feature vector is used as the query, and the point cloud geometric features are used as the key and value. After calculating the attention weights, an enhanced fused feature is output. This fused perceptual feature simultaneously contains environmental texture information and geometric structure information, providing a more comprehensive representation for anomaly detection.

[0043] Step S4: Based on the fused perception features, identify industrial abnormal obstacles in the current driving environment through the anomaly detection module to obtain the anomaly detection results.

[0044] The industrial anomalies include at least one of the following: wire mesh perforations, glass reflections, suspended obstacles, reflective material interference, low-light occlusion, and false positives due to dynamic occlusion. The anomaly detection module employs a lightweight detection head, which is based on a classification and regression network using a convolutional neural network or a Transformer. It inputs fused perceptual features and outputs the anomaly's category information, confidence level information, and bounding box position information in the image coordinate system.

[0045] Taking wire mesh perforations as an example, the recognition process includes the following sub-steps. First, the visual texture features of the wire mesh perforation area in the image are analyzed to identify the regular grid dot matrix structure. Then, combined with the feature that the LiDAR point cloud shows regular holes in the perforation area and the reflectivity is lower than a preset threshold, multi-modal feature joint judgment is performed, with the preset threshold being 15%. Finally, the case where the point cloud holes caused by the wire mesh perforation are mistakenly judged as solid walls is excluded, a wire mesh perforation anomaly type label is output, and it is determined that the anomaly does not block the main passage. For glass reflection anomalies, the model makes a judgment by identifying glare areas in the image and combining the feature of weak LiDAR reflection signals, where weak reflection signals refer to reflectivity below 10%. For low-light occlusion, the model uses the image brightness histogram and point cloud density distribution for comprehensive judgment.

[0046] Step S5: Based on the anomaly detection results and the intelligent robot's own state information, determine the feasibility of passage and generate a passage decision instruction.

[0047] The vehicle's own status information includes current cargo height, current driving speed, current navigation route, and vehicle dimensions. Feasibility assessment is differentiated based on different anomaly types. For suspended obstacles, feasibility assessment includes the following steps: Obtain the current cargo height H of the intelligent robot load Obtain the camera's intrinsic parameter matrix K and the camera's rotation matrix R and translation vector t relative to the world coordinate system. K Including focal length f x , f y and principal point coordinates c x , c y Wherein, the intrinsic parameter matrix of the camera is assumed to be: K= ; Furthermore, based on perspective projection relationships, an imaging geometric model is established for spatial points to the image plane, where spatial point P(X,Y,Z) is located in the image plane. The projection relationship in the image is represented as follows: s =K[R|t] ; Where s is the scale factor; based on the bottom pixel ordinate of the suspended obstacle in the image. v 0 and top pixel y-coordinate v t Calculate the pixel height difference Δ v = v 0- v t According to the camera installation height H c and pitch angle θ The distance between ground intersection lines can be approximated using imaging geometry: D = ; and then estimate the height of the suspended obstacle above the ground: =D ).Will h 0 and current cargo height H load Compare: When h 0 greater than H load When it is determined that the path is passable, a forward decision instruction is output, denoted as G; when h 0 less than or equal to H load When the passage is deemed impassable, a stop decision command is output, denoted as S.

[0048] For non-physical obstacles such as wire mesh cutouts and glass reflections, the model determines that they do not block the main passage and directly outputs a forward decision command. For dynamic occlusion misjudgments, including pedestrians temporarily passing by, the model combines the timing information to determine that they are brief interferences and outputs a deceleration decision command, which is denoted as S_D.

[0049] Furthermore, the anomaly detection module also includes a semantic question-answering reasoning unit, which performs cognitive-level semantic reasoning on industrial anomaly scenarios based on a question-and-answer format. Specifically, it receives text queries containing descriptions of anomaly scenarios, such as "Can the AGV safely pass through in the current scenario?", and performs cross-modal reasoning based on fused perceptual features and text semantics; it outputs anomaly type judgment results and corresponding passage significance semantic tags, including passable, need to slow down, and impassable, realizing cognitive-level reasoning from detecting the existence of an anomaly to understanding whether the anomaly affects passage.

[0050] Step S6: Based on the passage decision command and the preset path planning strategy, the obstacle avoidance execution module controls the intelligent robot to perform obstacle avoidance actions or continue to travel along the original navigation path.

[0051] The preset path planning strategy adopts a main channel priority strategy. Specifically, it includes the following sub-steps: First, the spatial position relationship of detected abnormal obstacles is determined, distinguishing whether the abnormal obstacle is on the robot's forward navigation main channel. The forward navigation main channel refers to whether the obstacle is located within a certain width range on both sides of the centerline of the preset navigation path. When the abnormal obstacle is determined to be on the forward main channel, it is marked as a forward obstruction, and a stop decision command or a deceleration decision command is generated based on the feasibility judgment result. When the abnormal obstacle is determined to be on the side or in a non-main channel area, it is marked as a lateral interference, and a forward decision command is generated, allowing the robot to continue along the main channel. When the abnormal obstacle is determined to be on the forward main channel and is deemed impassable (i.e., the height of the suspended obstacle above the ground is lower than the cargo height), local path replanning is initiated, generating a detour path point sequence, and returning to the original navigation path after detour.

[0052] Finally, the obstacle avoidance module executes the corresponding motion commands. The obstacle avoidance module uses a chassis controller to control the intelligent robot to move forward, decelerate, stop, or detour.

[0053] Edge computing deployment: The aforementioned visual language model, multimodal fusion module, and anomaly detection module are all deployed on the edge computing device within the intelligent robot itself. The edge computing device utilizes an NVIDIA Jetson AGX Orin or similar high-performance embedded computing platform. This edge computing device connects to the robot's sensors and control system via a local area network, receiving visual image data and LiDAR point cloud data in real time, performing local inference, and outputting decision commands to the actuators. Compared to traditional cloud processing solutions, the edge computing solution reduces data transmission from the entire video stream to only the detection results. The entire video stream is approximately 1.5MB per frame, while the detection results, in JSON format, are approximately 1KB per frame. This results in a 99.8% reduction in communication traffic, decreasing annual costs from 45,000 yuan to 93 yuan, and optimizing end-to-end latency from 4 seconds to less than 500 milliseconds, a reduction of 87.5%.

[0054] Experimental verification results: Table 2 is a comparison table of the responses of the general VL model and AbnormalVL-AGV to the same abnormal scenario; Appendix Figure 3 The heatmap shows a comparison of the detection accuracy of general VL models (Qwen3-VL, Hunyuan-VL, GPT-4o-mini, GPT-5-mini) and AbnormalVL-AGV in three scenarios: wire mesh cutouts, glass reflections, and suspended obstacles.

[0055] Table 2. Example Results Comparison Table

[0056] Using the above method, the model was validated on a test set containing six types of anomalies: wire mesh perforations, glass reflections, suspended obstacles, reflective material interference, low-light occlusion, and dynamic occlusion misjudgments. Table 3 compares the comprehensive performance of the proposed model with the mean values ​​of traditional schemes, general visual language models, and the model proposed in this application. This table demonstrates the significant improvement of the proposed anomaly detection visual language model over existing technologies (traditional schemes and general VL models) in several key indicators. The anomaly detection accuracy of this application reaches 92.7%, the false negative rate is 7.2%, the false positive rate is 3.5%, the inference latency is 50ms, the mAP decreases by 10.1% after robust noise, and the small sample generalization ability achieves an mAP of 83.9% with 100 training samples. Compared with existing technologies, the anomaly detection accuracy is improved by 24.5%, the false negative rate is reduced by 74.7%, the false positive rate is reduced by 86.2%, and the inference latency is reduced by 84.4%, achieving significant technological progress.

[0057] Table 3. Comprehensive Performance Evaluation Comparison Table

[0058] like Figure 4 As shown, a system for implementing an anomaly detection method for intelligent robots based on a visual language model according to a specific embodiment of the present invention is described. The system includes: The data acquisition module is used to perform the steps of acquiring visual image data and lidar point cloud data when the intelligent robot is moving in an industrial scene; The feature encoding module is used to perform the steps of inputting the visual image data into a pre-trained visual language model, and encoding the features of the input image based on the visual language model to obtain the image visual feature vector, wherein the visual language model is obtained by performing domain adaptation fine-tuning on a general visual language model based on a visual question answering dataset; The multimodal fusion module is used to perform the steps of fusing geometric features extracted from the lidar point cloud data based on the image visual feature vector, aligning cross-modal features in the multimodal fusion module, and generating fused perception features. Anomaly detection module is used to perform the step of identifying industrial abnormal obstacles in the current driving environment based on the fused perception features and obtaining anomaly detection results. The industrial abnormal obstacles include at least one of wire mesh perforation, glass reflection, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment. The passage decision module is used to execute the step of judging the feasibility of passage based on the anomaly detection result and the intelligent robot's own state information, and generating a passage decision instruction. The passage decision instruction includes a stop decision instruction, a deceleration decision instruction, or a forward decision instruction. The obstacle avoidance execution module is used to execute the steps of the intelligent robot to perform obstacle avoidance actions or continue to travel along the original navigation path based on the passage decision command and the preset path planning strategy.

[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0063] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0064] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An anomaly detection method for intelligent robots based on a visual language model, characterized in that, Includes the following steps: Acquire visual image data and lidar point cloud data of intelligent robots operating in industrial scenarios; The visual image data is input into a pre-trained visual language model, and the input image is feature-encoded based on the visual language model to obtain an image visual feature vector. The visual language model is obtained by performing domain adaptation fine-tuning on a general visual language model based on a visual question-answering dataset. The visual question-answering dataset contains multiple sets of question-answer-image-anomaly label samples. Based on the image visual feature vector, the geometric features extracted from the lidar point cloud data are fused, and cross-modal feature alignment is performed in the multimodal fusion module to generate fused perception features. Based on the fused perception features, the anomaly detection module identifies industrial abnormal obstacles in the current driving environment and obtains anomaly detection results. The industrial abnormal obstacles include at least one of the following: wire mesh perforation, glass reflection, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment. When a suspended obstacle is detected, the following steps are also included to determine the feasibility of passage: Obtain the current cargo height of the intelligent robot; Obtain the camera's intrinsic parameter matrix and the camera's rotation matrix and translation vector relative to the world coordinate system, and establish an imaging geometric model from spatial points to the image plane based on perspective projection relationships; Calculate the pixel height difference based on the bottom and top pixel coordinates of the suspended obstacle in the image; The distance to the ground intersection line is estimated based on the camera's installation height and pitch angle, and then the height of the suspended obstacle above the ground is estimated. The height of the suspended obstacle above the ground is compared with the current cargo height of the intelligent robot: when the height of the suspended obstacle above the ground is greater than the current cargo height, it is determined that it is passable and a forward decision command is output; when the height of the suspended obstacle above the ground is less than or equal to the current cargo height, it is determined that it is impassable and a stop decision command is output. Based on the anomaly detection results, and combined with the intelligent robot's own state information, a feasibility judgment is made for passage, and a passage decision instruction is generated. The passage decision instruction includes a stop decision instruction, a deceleration decision instruction, or a forward decision instruction. When generating a passage decision instruction based on the anomaly detection results, the following steps are included: The spatial position relationship of detected abnormal obstacles is determined to distinguish whether the abnormal obstacles are on the main forward navigation channel of the intelligent robot; When an abnormal obstacle is determined to be located in the main passage, the abnormality is marked as a forward obstruction, and a stop decision command or a deceleration decision command is generated based on the passage feasibility assessment results. When an abnormal obstacle is determined to be located in a lateral or non-main passage area, the abnormality is marked as a lateral disturbance, a forward decision instruction is generated, and the intelligent robot is allowed to continue to pass along the main passage. When an abnormal obstacle is determined to be located on the main passage and is deemed impassable, local path replanning is initiated to generate a sequence of detour path points. Based on the passage decision command and the preset path planning strategy, the obstacle avoidance execution module controls the intelligent robot to perform obstacle avoidance actions or continue along the original navigation path; the visual language model is the LLaVA-7B basic model, and the visual question-answering dataset is the self-built AED-VQA dataset, which contains more than 3,000 sets of question-answer, image, and anomaly label samples. The visual language model is obtained after domain adaptation fine-tuning using the AED-VQA dataset; the training strategy for the visual language model includes the following steps: General vision-language pre-training is performed based on a general vision-language dataset. The feature distance between matching problems and image pairs is narrowed through image-text comparison learning. Masked language modeling is used to enhance the model's understanding of keywords in industrial scenarios. The consistency of cross-modal feature space is constrained by the loss function. Domain-adaptive pre-training is performed based on a subset of the visual question-answering dataset. The model is trained to focus on abnormal regions through bounding box regression and region feature extraction. Cross-modal completion training is used to strengthen the mapping relationship between visual features and abnormal semantics. Industrial noise such as dust blur, uneven lighting, and reflective spots is injected into the training images to improve anti-interference ability. End-to-end fine-tuning was performed based on the full dataset of the visual question answering dataset. A multi-task joint loss function was used to balance the anomaly type classification loss and the anomaly key parameter regression loss. The anomaly type classification used weighted cross-entropy loss, and the anomaly parameter regression used Smooth L1 loss.

2. The intelligent robot anomaly detection method based on a visual language model according to claim 1, characterized in that, In the domain adaptation pre-training, the cross-modal completion training specifically includes: Randomly mask out some semantic elements in the question text to obtain an incomplete question text; The incomplete problem text and corresponding image features are input into the visual language model. Based on the cross-entropy loss constraint model, the accuracy of the anomaly type is improved according to the image features, thereby strengthening the mapping relationship between visual features and anomaly semantics. In the third-stage end-to-end fine-tuning, the multi-task joint loss function is specifically as follows: L total = λ 1 L cls + λ 2 L reg ； in, L cls Weighted cross-entropy loss for anomaly type classification, λ 1 = 0.7; L reg The regression loss components for the abnormal key parameters, λ 2=0.3; To address the imbalanced sample problem, higher weights are assigned to outlier categories with fewer samples for training optimization.

3. The intelligent robot anomaly detection method based on a visual language model according to claim 1, characterized in that, The process of acquiring lidar point cloud data and extracting geometric features includes the following steps: Preprocess the point cloud data collected by lidar to remove outliers and elevation noise; Extract geometric feature parameters from point cloud data, including reflectance features, spatial density distribution features, and three-dimensional coordinate position features; The extracted geometric feature parameters are concatenated with the image visual feature vector in a unified dimensional space, and the multimodal features are weighted and fused through an attention mechanism to generate fused perceptual features.

4. The intelligent robot anomaly detection method based on a visual language model according to claim 1, characterized in that, The anomaly detection module also includes a semantic question-answering reasoning unit, which performs cognitive-level semantic reasoning on industrial anomaly scenarios based on a question-answering format. It receives text queries containing descriptions of abnormal scenarios and performs cross-modal reasoning based on fused perceptual features and text semantics; Output the anomaly type judgment result and the corresponding semantic label of passage meaning, realizing cognitive-level reasoning from detecting the existence of an anomaly to understanding whether the anomaly affects passage.

5. The intelligent robot anomaly detection method based on a visual language model according to claim 1, characterized in that, When identifying abnormalities in wire mesh perforations, the following steps are included: Analyze the visual texture features of the wire mesh cutout area in the image to identify the regular grid dot matrix structure; By combining the characteristics of regular holes and reflectivity below a preset threshold in the hollowed-out area of ​​the lidar point cloud, multimodal feature joint judgment is performed. Exclude cases where point cloud voids caused by wire mesh perforations are mistakenly identified as solid walls, output a wire mesh perforation anomaly type label, and determine that the anomaly does not block the main passage.

6. An intelligent robot anomaly detection system based on a visual language model, characterized in that, The steps for implementing the intelligent robot anomaly detection method based on a visual language model according to any one of claims 1 to 5, wherein the intelligent robot anomaly detection system based on a visual language model comprises: The data acquisition module is used to perform the steps of acquiring visual image data and lidar point cloud data when the intelligent robot is moving in an industrial scene; The feature encoding module is used to perform the steps of inputting the visual image data into a pre-trained visual language model, and encoding the features of the input image based on the visual language model to obtain the image visual feature vector, wherein the visual language model is obtained by performing domain adaptation fine-tuning on a general visual language model based on a visual question answering dataset; The multimodal fusion module is used to perform the steps of fusing geometric features extracted from the lidar point cloud data based on the image visual feature vector, aligning cross-modal features in the multimodal fusion module, and generating fused perception features. Anomaly detection module is used to perform the step of identifying industrial abnormal obstacles in the current driving environment based on the fused perception features and obtaining anomaly detection results. The industrial abnormal obstacles include at least one of wire mesh perforation, glass reflection, suspended obstacles, reflective material interference, low light occlusion, and dynamic occlusion misjudgment. The passage decision module is used to execute the step of judging the feasibility of passage based on the anomaly detection result and the intelligent robot's own state information, and generating a passage decision instruction. The passage decision instruction includes a stop decision instruction, a deceleration decision instruction, or a forward decision instruction. The obstacle avoidance execution module is used to execute the steps of the intelligent robot to perform obstacle avoidance actions or continue to travel along the original navigation path based on the passage decision command and the preset path planning strategy.

Images