Navigation method and system for intelligent robot in complex plant environment
By constructing a prior GIS geospatial database of the factory area and multimodal fusion with a large language model, the safety and robustness issues of intelligent robots navigating in complex factory environments were solved, achieving improvements in safety and efficiency, as well as interpretability of decisions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU YUANZI INTELLIGENT TECH CO LTD
- Filing Date
- 2026-05-06
- Publication Date
- 2026-06-02
AI Technical Summary
Existing intelligent robots lack global spatial prior constraints, have insufficient semantic reasoning ability in abnormal scenarios, and have uninterpretable decision-making logic in complex factory environments, resulting in insufficient navigation safety and decision robustness.
A priori GIS geospatial database for the factory area is constructed. By combining a visual perception module and a large language model, LoRA parameters are efficiently fine-tuned to achieve multimodal fusion and cognitive-level deep semantic collaboration for obstacle detection and navigation decision-making.
It improves the navigation safety and decision-making robustness of intelligent robots in complex factory environments, reduces misjudgments and omissions, achieves synergistic optimization of safety and transportation efficiency, and has interpretable decision-making logic.
Smart Images
Figure CN122131776A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent robot navigation technology, specifically relating to a navigation method and system for intelligent robots in complex factory environments. Background Technology
[0002] With the rapid development of industrial automation and intelligent manufacturing, intelligent robots such as Automated Guided Vehicles (AGVs) are increasingly widely used in factory logistics and transportation. Complex factory environments are characterized by complex road topologies, diverse obstacle types, frequent personnel flow, and significant differences in road surface physical conditions, which place extremely high demands on the navigation safety, stability, and decision-making robustness of intelligent robots.
[0003] Existing intelligent robot navigation methods primarily rely on local sensors such as LiDAR and depth cameras for obstacle detection and path planning. However, existing technologies have the following shortcomings: Lacking global spatial prior constraints, lidar and vision sensors can only perceive local environmental information around the robot and cannot obtain global geospatial prior knowledge such as global road topology, road surface physical properties and spatial risk distribution in the factory area. As a result, intelligent robots lack sufficient decision-making basis in special scenarios such as complex intersections, sharp bends or low-friction road surfaces.
[0004] The ability to perform semantic reasoning in a scene is insufficient. Traditional deep learning-based obstacle detection methods (such as the YOLO series of object detection models) can only achieve binary classification of obstacles as "obstacle / non-obstacle" and cannot perform semantic association reasoning on obstacles. For example, they cannot determine whether stationary goods on the road are blocking adjacent lanes or whether people on the road are in an emergency evacuation state, etc.
[0005] Multimodal fusion is shallow. Existing vision-LiDAR fusion solutions usually remain at the data-level or feature-level fusion level, lacking cognitive-level deep semantic collaboration and failing to fully utilize the structured spatial knowledge in Geographic Information Systems (GIS) for decision-making and reasoning.
[0006] There is a lack of dedicated datasets to support decision-making for industrial intelligent robots in abnormal scenarios in the factory area (such as personnel intrusion, goods scattering, road damage, etc.). There is also a lack of high-quality structured multimodal training datasets to support the domain adaptation training of large language models.
[0007] In summary, existing technologies suffer from technical problems such as a lack of global spatial prior constraints, insufficient semantic reasoning ability in abnormal scenarios, and unexplainable decision-making logic in intelligent robots. There is an urgent need to propose a new intelligent robot navigation method that can achieve cognitive-level deep fusion of global prior knowledge from GIS geospatial data and local visual perception, and realize semantic reasoning decision-making in complex scenarios through a large language model. This would significantly improve the navigation safety, decision robustness, and interpretability of intelligent robots in complex factory environments. Summary of the Invention
[0008] The purpose of this invention is to solve the technical problems of existing intelligent robots, such as lack of global spatial prior constraints, insufficient semantic reasoning ability in abnormal scenarios, and unexplainable decision-making logic, and to provide a navigation method and system for intelligent robots in complex factory environments.
[0009] To achieve the above objectives, a specific embodiment of the present invention provides a navigation method for intelligent robots in complex factory environments. The method includes: constructing a priori GIS geospatial database for the factory area; extracting data on the topology of factory roads, ground physical attributes, and spatial risk levels based on a geographic information system; and quantifying the data on the topology of the roads, ground physical attributes, and spatial risk levels to obtain priori global geospatial data for the factory area. The visual perception module installed on the intelligent robot collects local images of the factory area in real time. Based on the deep learning obstacle detection model, abnormal obstacles are detected in the local images of the factory area to obtain obstacle category information and spatial position information of obstacles relative to the intelligent robot. The spatial semantic text conversion operator is used to convert the GIS vector data in the factory area GIS geospatial prior database into natural language text description, and the spatial semantic Prompt corresponding to the current visual perception area is obtained. The abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt are used as multimodal inputs and sent to the large language model cognitive reasoning module, which has been efficiently fine-tuned by LoRA parameters. The large language model cognitive reasoning module is based on the thought chain reasoning mechanism and outputs reasoning results that include intermediate reasoning steps and final navigation decision instructions. The navigation decision instructions generate motion control signals for the intelligent robot, driving the intelligent robot to perform safe navigation actions.
[0010] In one or more embodiments of the present invention, the construction of the factory area GIS geospatial prior database includes: By connecting the QGIS geographic information system software with the factory's digital twin platform, the vector map data of the factory area is read and parsed, and the topological data of the factory's roads are extracted, including road boundary lines, lane center lines, road intersections and road curvature radii. Based on the vector map data of the factory area, extract the physical attribute data of the factory area ground, including the ground friction coefficient and road slope value of each road section; Based on the factory area road topology and ground physical property data, and combined with the intelligent robot's braking safety distance formula, the safe braking distance of each spatial location in the factory area is quantitatively calculated. The formula for the braking safety distance of intelligent robots includes: ,in This represents the current real-time speed of the intelligent robot. The coefficient of friction of the ground is given. It is the acceleration due to gravity. For system response time, This is a slope correction term; Based on the safe braking distances of various spatial locations within the plant area, the heat map of historical accidents in the plant area, and the marking of densely populated areas, a quantitative model of the spatial risk level of each area of the plant area is established.
[0011] In one or more embodiments of the present invention, the multimodal fusion process includes: Based on the real-time positioning coordinates of the intelligent robot, the current GPS positioning coordinates are transformed from the WGS84 coordinate system to the local plane coordinate system through a multi-level coordinate system transformation model. Then, the coordinates of the local plane coordinate system are mapped to the visual image pixel coordinate system to establish a two-way mapping relationship from spatial location to image pixels. Based on the bidirectional mapping relationship from spatial location to image pixels, the spatial location coordinates of the obstacles in the abnormal obstacle detection results are synchronously mapped to the prior GIS geospatial database of the factory area, and the GIS spatial attribute data corresponding to the obstacle location is retrieved and obtained, including the road width, ground friction coefficient and spatial risk level of the road segment where the obstacle is located. The GIS spatial attribute data corresponding to the obstacle location is used as a spatial constraint condition. Together with the abnormal obstacle category information and obstacle spatial location information output by the visual perception module, the data is processed through multimodal feature alignment and fusion, and then input into the large language model cognitive reasoning module.
[0012] In one or more embodiments of the present invention, the method for constructing the spatial semantic textification transformation operator includes: Preprocess the GIS vector map data of the factory area and extract the lane width, road curvature, ground type, spatial risk level and historical abnormal event records of each road section in the factory area as spatial semantic feature attributes; Based on a predefined spatial semantic Prompt template, the spatial semantic feature attributes are organized into structured natural language text according to a three-level structure of current road segment physical constraint information, adjacent road segment risk warning information, and factory area global path planning information. The converted structured natural language text is scored for quality, low-quality semantic descriptions with scores below a preset threshold are filtered out, and a spatial semantic prompt that matches the current visual perception area is output. The spatial range matching is a distance threshold filtering with the current position of the intelligent robot as the center and the radius as the visual perception range.
[0013] In one or more embodiments of the present invention, the navigation decision reasoning process based on the thought chain reasoning mechanism includes: Based on a predefined thought chain Prompt template, the spatial semantic Prompt, the abnormal obstacle detection results, and the current intelligent robot motion state parameters are combined into a complete input Prompt for the large language model according to the Prompt template format. The large language model is guided to output reasoning results according to the following reasoning steps: analyze the visual perception information of the current local environment of the factory area; retrieve the GIS global geospatial constraints of the current road segment; assess the threat level of the current obstacle by combining the braking safety distance and spatial risk level; perform safety reasoning by integrating local perception information and global spatial constraints; and generate navigation decision instructions. The navigation decision instructions output by the large language model are subject to security verification. The verification includes the consistency between the instruction action and the current GIS spatial constraints, and the prediction verification of whether the instruction will exceed the braking safety distance after execution. If the security verification fails, the instruction rejection mechanism is triggered and the instruction is regenerated.
[0014] In one or more embodiments of the present invention, the step of: A pre-built multimodal dataset for abnormal scenarios of industrial intelligent robots, IndAGV-4K, is constructed, including image frame data extracted from factory monitoring videos according to a preset sampling interval, as well as the corresponding GIS geographic coordinate attribute data; The extracted image frames are labeled with abnormal events, including personnel intrusion events, cargo spillage events, road damage events, and abnormal obstacle events. By associating GIS coordinates, GIS spatial attribute data corresponding to each image frame is exported from the prior GIS geospatial database of the factory area, including the ground friction coefficient, spatial risk level and road curvature radius of the area corresponding to the image frame. The expert decision-making process records collected by the annotation tool are processed into structured text to generate expert decision samples corresponding to each abnormal event; The abnormal event annotations, GIS spatial attribute data, expert decision samples, and factory environment attribute annotations are packaged into JSON format to form an industrial intelligent robot abnormal scenario multimodal dataset IndAGV-4K containing no less than 4,000 samples.
[0015] In one or more embodiments of the present invention, the efficient LoRA parameter fine-tuning method includes: Design a dedicated prompt word template for navigation scenarios of industrial intelligent robots. The dedicated prompt word template includes a description of the factory space constraint format, a reference example for decision-making in abnormal scenarios, and a description of safety verification rules. Based on the IndAGV-4K dataset, we constructed multimodal fine-tuning training samples that include visual perception description, GIS spatial semantic prompt, and expert decision-making process. The LoRA algorithm is used to perform low-rank decomposition and adaptation on the attention weight matrix of the large language model. The weight matrices of the Query layer and Value layer of the large language model are selected as the LoRA adaptation targets, and the LoRA rank parameter and scaling factor are set. The multimodal fine-tuning training samples are used to perform efficient parameter fine-tuning training on the graphics processing unit, and the weights of the LoRA adaptation matrix are updated so that the navigation decision accuracy of the fine-tuned large language model in abnormal scenarios of industrial intelligent robots is not lower than a preset threshold.
[0016] In one or more embodiments of the present invention, the abnormal obstacle detection further includes: The visual perception module acquires local image sequences of the factory area in real time, and preprocesses the image sequences, including image denoising and brightness equalization. The preprocessed local images of the factory area are input into a target detection model based on an improved YOLOv11 convolutional neural network to obtain obstacle bounding boxes, obstacle categories, and detection confidence scores. Based on the bidirectional mapping relationship from spatial location to image pixels, the obstacle bounding box is mapped from the pixel coordinate system to the local planar coordinate system of the factory area GIS geospatial prior database to obtain the actual physical size and spatial location coordinates of the obstacle.
[0017] In one or more embodiments of the present invention, the security reasoning in the reasoning step sequence further includes: Based on the current real-time speed of the intelligent robot, the ground friction coefficient, and the spatial risk level, the safe following distance threshold under the current state is calculated using the braking safety distance formula. Based on the spatial position information and relative motion speed of the abnormal obstacle relative to the intelligent robot, calculate the time collision (TTC) value with the abnormal obstacle; When the TTC value is less than the safety TTC benchmark value corresponding to the safety following distance threshold, the safety decision weight is adjusted based on the spatial risk level, and the output includes deceleration, stopping, or detour decision instructions.
[0018] In another aspect of the invention, a navigation system for an intelligent robot in a complex factory environment is provided, comprising: The GIS spatial database construction module is used to construct a prior GIS geospatial database for the factory area. Based on the geographic information system, it extracts data on the topology of the factory area roads, ground physical attributes, and spatial risk levels, and quantifies the data on the topology of the roads, ground physical attributes, and spatial risk levels to obtain the prior global geospatial data of the factory area. The visual perception and detection module is used to acquire local images of the factory area in real time through the visual perception module installed on the intelligent robot, and to detect abnormal obstacles in the local images of the factory area based on a deep learning obstacle detection model, thereby obtaining obstacle category information and spatial position information of the obstacles relative to the intelligent robot. The spatial semantic conversion module is used to convert the GIS vector data in the prior GIS geospatial database of the factory area into natural language text descriptions through spatial semantic text conversion operators, so as to obtain the spatial semantic Prompt corresponding to the current visual perception area. The cognitive reasoning module is used to send the abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt as multimodal inputs to the large language model cognitive reasoning module, which has been efficiently fine-tuned by LoRA parameters; the large language model cognitive reasoning module is based on the thought chain reasoning mechanism and outputs reasoning results including intermediate reasoning steps and final navigation decision instructions. The execution control module is used to generate motion control signals for the intelligent robot based on the navigation decision instructions, and drive the intelligent robot to perform safe navigation actions.
[0019] Beneficial effects: Compared with the prior art, the present invention has the following significant advancements: This invention integrates GIS geospatial priors (road width, friction coefficient, risk level), visual semantics, and a large language model, enabling intelligent robots to perform physical feasibility reasoning based on global spatial constraints when facing unstructured abnormal scenarios such as road oil stains, water accumulation, fallen goods, and temporary construction, rather than relying solely on instantaneous observations from local sensors. This effectively avoids misjudgments and omissions caused by changes in lighting, missing visual features, or sensor noise in traditional solutions, fundamentally improving the robustness and safety of decision-making.
[0020] This application utilizes the reasoning capabilities of a large language model to comprehensively determine "detour," "slow down," or "stop" based on road width, friction coefficient, and obstacle type provided by GIS. This minimizes unnecessary emergency stops while ensuring safety, achieving synergistic optimization of safety and transportation efficiency.
[0021] This application designs a Prompt template containing a thought chain to guide a large language model to output intermediate reasoning results according to a fixed sequence of steps: "visual analysis, GIS retrieval, braking distance assessment, comprehensive reasoning, and decision generation." This ensures that decision instructions possess a complete logical chain and interpretability. This feature facilitates fault tracing, safety auditing, and system optimization in industrial settings.
[0022] This invention employs LoRA parameter efficient fine-tuning technology, updating only the low-rank matrix in the attention layer of a large language model while freezing most of the weights in the pre-trained model. This enables efficient model adaptation even with limited industrial samples, significantly reducing computational requirements and deployment costs. Simultaneously, it avoids the catastrophic forgetting problem that easily occurs with full parameter fine-tuning, allowing the large language model to accurately learn expert decision-making knowledge for industrial scenarios while retaining its general semantic understanding capabilities.
[0023] This invention constructs a structured multimodal dataset containing visual images, synchronized GIS attributes, and expert-level thought chain decision-making instructions, providing the first data benchmark in the field of industrial intelligent robots that integrates geospatial priors and expert decision-making. This dataset can support subsequent research and verification of more extensive intelligent decision-making models for intelligent robots, and has good technical scalability and industrial application value. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a flowchart of a navigation method for an intelligent robot in a complex factory environment according to one embodiment of the present invention; Figure 2 This is a block diagram of a navigation device for an intelligent robot in a complex factory environment according to one embodiment of the present invention. Detailed Implementation
[0026] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0027] Please see the appendix Figure 1 This application provides a navigation method for intelligent robots in complex factory environments. The navigation method is executed by a navigation system for intelligent robots in complex factory environments, and specifically includes the following steps: Step S1: Construct a prior geospatial database for the factory area. Extract road topology, ground physical attributes, and spatial risk level data based on the geographic information system. Quantify the road topology, ground physical attributes, and spatial risk level data to obtain global prior geospatial data for the factory area.
[0028] Furthermore, this embodiment also includes the following steps: Connecting the QGIS geographic information system software with the factory's digital twin platform to read and parse the factory's vector map data, extracting the factory's road topology data, including road boundary lines, lane center lines, road intersections, and road curvature radii; based on the factory's vector map data, extracting the factory's ground physical attribute data, including the ground friction coefficient and road slope values for each road segment; according to the factory's road topology and ground physical attribute data, combined with the intelligent robot's braking safety distance formula, quantitatively calculating the safe braking distance for each spatial location within the factory; and establishing a quantitative model of the spatial risk level for each area of the factory based on the safe braking distance for each spatial location, the factory's historical accident heat map, and the marking of densely populated areas.
[0029] Specifically, the method for constructing the prior GIS geospatial database for the factory area includes the following steps: The first step is to establish a data connection between the QGIS 3.28 geographic information system software and the factory's digital twin platform. QGIS supports reading and processing various vector map formats. The factory's digital twin platform provides vector map data services for the factory area through a unified GIS data interface. The QGIS software calls this GIS data interface, covering the entire factory area according to a preset spatial query range, to read and parse the factory's vector map data. The data format is Shapefile or GeoJSON. After parsing, the factory's road topology data is extracted, specifically including: Road boundary lines are stored as line segment geometry types, recording the coordinate sequence of the left and right boundary lines of each road; Lane centerlines are stored as line segment geometry types, recording the coordinate sequence of each lane centerline. Road intersections are stored as point geometry types, identifying the locations of the intersections of various roads; The road curvature radius is stored as a numerical attribute field. It calculates the radius of the circle corresponding to the maximum curvature of the centerline for each road segment, in meters.
[0030] Based on the aforementioned vector map data of the factory area, ground physical attribute data is extracted. This ground physical attribute data is linked to the road topology data via road numbers. Specifically, the extracted ground physical attribute data includes: the ground friction coefficient for each road segment (a dimensionless value: 0.6 to 0.8 for concrete pavement, 0.4 to 0.6 for epoxy flooring, 0.3 to 0.5 for metal mesh, and 0.2 to 0.4 for wet / slippery pavement); and the road slope value for each road segment, stored as angle values, with positive values indicating uphill and negative values indicating downhill. The ground friction coefficient is collected using a friction coefficient tester, such as a lateral force coefficient tester, at representative locations along each road segment. Three measurement points are selected for each segment, and the arithmetic mean is taken as the ground friction coefficient for that segment. The road slope value is extracted from the topographic elevation data of the factory area's digital twin platform, calculated by the ratio of the elevation difference to the horizontal distance between adjacent elevation control points.
[0031] Based on the factory area's road topology and ground physical properties data, and combined with the intelligent robot's braking safety distance formula, the safe braking distance for each spatial location within the factory area is quantitatively calculated. The braking safety distance refers to the minimum safe distance that the intelligent robot must maintain from detecting an obstacle to coming to a complete stop at its current speed. The formula for the intelligent robot's braking safety distance is: ; in, The current real-time speed of the intelligent robot is obtained in real time by the vehicle-mounted speed sensor, and the unit is meters per second. The coefficient of friction of the ground; The acceleration due to gravity is taken as 9.8 meters per second squared. The slope correction factor is equal to , The road slope angle; The system response time is 0.3 seconds in this embodiment.
[0032] In the specific calculation, each road segment in the prior GIS geospatial database of the factory area is used as a calculation unit, and the friction coefficient corresponding to that road segment is calculated. and slope value Substituting the values into the braking safety distance formula and considering the rated maximum speed of the intelligent robot, for example, 3.0 m / s, calculate the braking safety distance of this road segment at its maximum speed. .
[0033] Based on the safe braking distances of various spatial locations within the plant area, historical accident heat maps, and markings of densely populated areas, a spatial risk level quantification model is established for each area of the plant. This model divides each spatial location within the plant area into five risk levels, from 1 to 5, with level 1 being the lowest risk and level 5 the highest. The specific classification method is as follows: using historical accident heat maps stored in the plant's safety management information system, a density heat map is generated based on the coordinates of accident locations. Locations with high accident incidence and heat values greater than 0.7 are automatically marked as level 4 or 5 risk areas. In densely populated areas such as loading / unloading platforms and workshop entrances / exits, risk levels are determined through personnel flow sensor data or manual marking and marked as level 3 or higher risk areas. High-risk road sections with safe braking distances greater than 5 meters (i.e., sharp bends with a road surface friction coefficient less than 0.3 or a road curvature radius less than 5 meters) are marked as level 4 risk areas. Low-risk areas on main roads with straight sections, open visibility, and a ground friction coefficient greater than 0.6 are marked as level 1 or 2 risk areas. Among them, the comprehensive geographical risk score The calculation formula is: ; in, For the first The base coefficient for each risk point is 1.0 for high-risk points, 0.6 for medium-risk points, and 0.3 for low-risk points. The Euclidean distance between the intelligent robot and the risk center; The preset influence radius for GIS; The coefficient of friction is the ground friction factor. This score is used to quantitatively assess the comprehensive geographical risks faced by the intelligent robot at its current location. Finally, the spatial risk level quantification model, in the form of rasterized spatial data, is stored together with the braking safety distance quantification data in the GIS geospatial prior database, forming complete global geospatial prior data for the entire factory area.
[0034] Step S2: Real-time acquisition of local images of the factory area through the visual perception module installed on the intelligent robot, and abnormal obstacle detection of the local images of the factory area based on the deep learning obstacle detection model to obtain obstacle category information and spatial position information of the obstacle relative to the intelligent robot.
[0035] Specifically, the intelligent robot is equipped with a forward-facing industrial camera with a resolution of 1280 x 720 pixels, a frame rate of 15 frames per second, and a field of view of 120 degrees horizontally and 75 degrees vertically. The camera is mounted on a support at the front of the intelligent robot, 1.2 meters above the ground, and tilted forward at a 15-degree angle to cover the typical obstacle detection area in front of the robot. The visual perception module is connected to the main control system of the intelligent robot via a gigabit Ethernet interface, acquiring local images of the factory area in real time and transmitting the image data to the image processing unit of the main control system.
[0036] Furthermore, this embodiment also includes the following steps: acquiring local image sequences of the factory area in real time through the visual perception module, preprocessing the image sequences, including image denoising and brightness equalization; inputting the preprocessed local images of the factory area into a target detection model based on an improved YOLOv11 convolutional neural network to obtain obstacle bounding boxes, obstacle categories, and detection confidence; based on the bidirectional mapping relationship from spatial location to image pixels, mapping the obstacle bounding boxes from the pixel coordinate system to the local planar coordinate system of the factory area GIS geospatial prior database to obtain the actual physical size and spatial location coordinates of the obstacles.
[0037] Specifically, image preprocessing includes the following steps: First, the original image is subjected to Gaussian filtering for noise reduction, with a convolution kernel size of 5x5 and a standard deviation of [missing information]. σ The Clip Limit is set to 1.2 to suppress image noise caused by complex lighting conditions in the factory area. Then, an adaptive histogram equalization method, namely the CLAHE method, is used with the Clip Limit set to 2.0 to perform brightness equalization processing on the image to cope with scenes with large differences in indoor and outdoor lighting in the factory area, such as driving from a bright loading and unloading platform into a dark workshop. Finally, the preprocessed image is scaled to 640 by 640 pixels to adapt to the input size of the object detection model.
[0038] The target detection model employs an improved YOLOv11 convolutional neural network. In this embodiment, YOLOv11 is adapted for obstacle detection scenarios in industrial intelligent robots. The main improvements include replacing the original C3k2 modules with FasterNeXt modules in the P4 / 16 and P5 / 32 feature extraction layers of the model backbone network. The FasterNeXt module reduces computational cost and memory usage, and improves feature extraction speed by optimizing the convolutional kernel structure and channel allocation. A REG module (Refinement and Enhancement Gate) is introduced into the feature fusion network. The REG module uses multi-branch convolutional operations, combining 3×3 and 5×5 convolutional kernels to capture texture information under different receptive fields, and enhances the weight of obstacle boundary region features through a feature weighting mechanism, resulting in higher accuracy in the output obstacle bounding box localization. The detection head retains the original YOLOv11 detection layer architecture and outputs four coordinate values of the obstacle bounding box: x1, y1, x2, y2, corresponding to the pixel coordinates of the upper left and lower right corners of the bounding box. It also outputs the obstacle category, including four categories: "workers", "scattered goods", "construction equipment" and "other obstacles". Finally, it outputs the detection confidence score, which is a floating-point value between 0 and 1.
[0039] After inference by the object detection model, the pixel coordinates of the output obstacle bounding box are converted into the pixel coordinates of the obstacle center point. The calculation formula is as follows: , Simultaneously, the approximate width and height of the obstacle in pixel coordinates are calculated using the following formula: , These data serve as one of the input data for subsequent multi-level coordinate system transformations.
[0040] Let a point in the QGIS global coordinate system be... Its projection onto the pixel coordinates of the local visual image of the intelligent robot The transformation equation is: ; in, This is the real-time transformation matrix from the world to the robot's base, calculated based on EKF fusion GPS / IMU data. and K These are the camera's extrinsic and intrinsic parameter matrices, respectively. This transformation maps the center pixel coordinates of the detected obstacle's bounding box back to a local Cartesian coordinate system, yielding the obstacle's actual physical coordinates. Subsequently, attribute data such as road width, ground friction coefficient, and spatial risk level corresponding to that location are retrieved from the GIS database.
[0041] Step S3: Convert the GIS vector data in the factory area GIS geospatial prior database into natural language text descriptions using the spatial semantic text conversion operator to obtain the spatial semantic Prompt corresponding to the current visual perception area.
[0042] First, the GIS vector map data of the factory area is preprocessed. Taking the current GPS coordinates of the intelligent robot as the center and the visual perception range R=30 meters as the radius, the semantic feature attributes of each road segment within the window are extracted, including lane width (2.0~8.0 meters), road curvature (the reciprocal of the radius of curvature), ground type (concrete, epoxy flooring, etc.), spatial risk level (level 1~5) and historical abnormal event records.
[0043] Then, based on the predefined spatial semantic Prompt template, the above feature attributes are organized into structured natural language text according to a three-level structure: Level-1 (Current Road Segment Physical Constraint Information): Road segment number, width, friction coefficient, risk level, braking safety distance, radius of curvature; Level-2 (Adjacent Road Segment Risk Warning Information): Adjacent road segment number, width, risk level, and statistics of abnormal events within a 50-meter radius in the past 30 days; Level-3 (Plant Area Global Path Planning Information): Optimal path sequence from the current location to the target point, total length, estimated travel time, and maximum risk level along the way.
[0044] Finally, the converted text is scored for quality, with scoring dimensions including completeness (checking whether each field is filled in effectively) and consistency (checking...). The criteria for determining the success rate are: lane width (≤) and timeliness (whether historical event records exceed 90 days). If the overall score is below a preset threshold (e.g., 0.6), the query radius is automatically expanded to 50 meters and the query is re-converted; if the score still does not meet the requirements, a Prompt with a "data unreliable" warning is output and logged. The final output is a spatial semantic Prompt that matches the current visual perception area within the spatial range.
[0045] Step S4: The abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt are used as multimodal inputs and sent to the large language model cognitive reasoning module, which has been efficiently fine-tuned by the LoRA (Low-Rank Adaptation) parameter; the large language model cognitive reasoning module outputs a reasoning result containing intermediate reasoning steps and final navigation decision instructions based on the thought chain reasoning mechanism.
[0046] Cross-modal feature alignment is achieved through a self-attention mechanism: ; Among them, query vector Derived from visual anomaly features, key vector K Sum value vector V It incorporates the physical constraints of GIS.
[0047] This embodiment employs LoRA (Low-Rank Adaptation) technology for efficient parameter fine-tuning. The weight matrices of the Query and Value layers of a large language model are selected as the adaptation targets, and a low-rank decomposition matrix is introduced. and The forward propagation formula is: ; The rank parameter r=8 and the scaling factor α=16. Fine-tuning training was performed on an NVIDIA RTX 4090 GPU using the IndAGV-4K dataset containing 4000 "image-GIS-decision" samples, with a batch size of 16 and a learning rate of 2×10⁻⁶. 5 The training run consists of 5 epochs. The loss function used is cross-entropy loss. ; in, A token for real decision-making by experts. To increase the probability prediction of the model, a weighting coefficient is added to high-risk samples. To enhance safety recall rates.
[0048] The navigation decision-making reasoning process based on the thought chain reasoning mechanism is as follows: First, based on a predefined thought chain Prompt template, the spatial semantic Prompt, abnormal obstacle detection results (obstacle type, relative distance, relative speed), and current intelligent robot motion state parameters (velocity, heading angle) are combined into a complete input Prompt. An example of the template format is as follows: Scenario Description: You are the decision-making system of an industrial intelligent robot, performing material transportation tasks within the factory area.
[0049] Your location information: Current GPS coordinates are (latitude, longitude), located at road segment number X, speed... meters per second, heading angle α Spend.
[0050] Visual perception information: An obstacle of [obstacle category] has been detected, located in front. d At a distance of meters, the relative velocity meters per second, confidence level conf.
[0051] GIS spatial constraint information: Current road segment width W meters, friction coefficient μ, risk level R, braking safety distance. D safe rice.
[0052] Please output the following reasoning steps: Step 1: Analyze visual information to identify obstacle types and threat levels; Step 2: Retrieve global constraints in the GIS; Step 3: Calculate the current safe braking distance based on the braking safety distance formula, and assess whether the threat exceeds the safety threshold; Step 4: Perform security reasoning by integrating local perception and global constraints; Step 5: Generate navigation decision instructions (decelerate / stop / detour) and parameters.
[0053] Output format: First, output the reasoning steps one by one, and finally output the JSON format instructions in the [Decision Instruction] field.
[0054] The large language model is guided to output intermediate inference results and final decision instructions according to the above inference step sequence. Further, after generating the decision instructions, a safety check is performed on the instructions, including: consistency check—determining whether the instruction action (e.g., "decelerate to 0.5 m / s") is consistent with the maximum speed allowed on the current road segment; braking distance prediction check—determining whether the expected position in the next cycle after the instruction is executed is still within the safe braking distance range. If any check fails, an instruction rejection mechanism is triggered, the reason for rejection is fed back to the large language model, and the instruction is regenerated, up to a maximum of 3 times; if it still fails after 3 times, the default safe action is forcibly executed.
[0055] Furthermore, the security reasoning in the reasoning step also includes based on TTC Dynamic decision-making. Specifically: based on the current real-time speed. Ground friction coefficient Based on the spatial risk level, the safe following distance threshold is calculated using the braking safety distance formula. and calculate TTC value: ; in, d The relative distance between the obstacle and the robot. Relative velocity (when there is a stationary obstacle) = ).
[0056] Safety TTC Baseline value: .when At that time, the comprehensive space risk level Rrisk Adjust the weights for security decisions, and determine the weighting coefficients according to the following rules: Rrisk ≤2: Weight 1.0; 2 < Rrisk ≤ 3: Weight 1.2; 3 < Rrisk ≤ 4: Weight 1.5; Rrisk > 4: Weight 2.0; The final decision instruction is determined according to the ratio: decelerate to 50% of the current speed when > 1.5; decelerate to 20% of the current speed when 1.0 - 1.5; stop when ≤ 1.0; if Rrisk ≥ 4 and there is an adjacent available lane, output a detour instruction.
[0057] Step S5: Generate a motion control signal for the intelligent robot according to the navigation decision instruction that passes the safety verification. For a differential drive type intelligent robot, the kinematic model is: ; where and are the commanded linear velocity (m / s) and angular velocity (rad / s) respectively, is the wheelbase (0.8 m), is the radius of the drive wheel (0.1 m). The calculated motor speed command is sent to the drive controller via the CAN bus to drive the intelligent robot to perform safe navigation actions.
[0058] This embodiment also includes the step of pre - constructing the multi - modal dataset IndAGV - 4K for abnormal scenarios of industrial intelligent robots. The specific method is: extract image frames from the factory area monitoring video at a preset sampling interval, and synchronously record the GPS coordinates and heading angles; use the CVAT annotation tool to annotate the abnormal events in the image frames, and the annotation content includes the obstacle bounding box, event type and severity level; through GIS coordinate association, export the spatial attribute data corresponding to each image frame from the factory area GIS geospatial prior database; structure the expert decision description text filled in during the annotation process to generate expert decision samples; finally, package the abnormal event annotation, GIS attribute data, expert decision samples and factory area environment attribute annotation into a JSON format to form the IndAGV - 4K dataset containing no less than 4000 samples.
[0059] Based on this dataset, use the aforementioned LoRA parameter efficient fine - tuning method to fine - tune the large - language model so that the navigation decision accuracy of the fine - tuned model in the abnormal scenarios of industrial AGVs is not lower than the preset threshold.
[0060] This embodiment also provides a navigation system for intelligent robots in a complex factory area environment. The system includes: GIS Spatial Database Construction Module: Used to construct a priori GIS geospatial database for the factory area. Based on the geographic information system, it extracts data on the topology of the factory area's roads, ground physical attributes, and spatial risk levels, and quantifies the data to obtain priori global geospatial data for the factory area.
[0061] Visual perception and detection module: This module is used to acquire local images of the factory area in real time through a visual perception module installed on the intelligent robot, and to detect abnormal obstacles in the local images of the factory area based on a deep learning obstacle detection model, thereby obtaining obstacle category information and spatial position information of the obstacles relative to the intelligent robot.
[0062] Spatial semantic conversion module: used to convert GIS vector data in the prior GIS geospatial database of the factory area into natural language text descriptions through spatial semantic text conversion operators, and obtain spatial semantic Prompt corresponding to the current visual perception area.
[0063] Cognitive reasoning module: It is used to send the abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt as multimodal inputs to the large language model cognitive reasoning module with efficient fine-tuning of LoRA parameters; the large language model cognitive reasoning module is based on the thought chain reasoning mechanism and outputs the reasoning result containing intermediate reasoning steps and final navigation decision instructions.
[0064] Execution control module: used to generate motion control signals for the intelligent robot based on the navigation decision instructions, and drive the intelligent robot to perform safe navigation actions.
[0065] The specific implementation methods, parameter settings, and workflows of each module correspond one-to-one with the method steps described in Example 1, and will not be repeated here.
[0066] This embodiment was experimentally validated on the IndAGV-4K dataset. Four sets of comparison models were set up: Method A (CNN-Only, traditional visual obstacle avoidance based on YOLOv8), Method B (VLA-Vanilla, multimodal model without GIS prior), Method C (GIS-Enhanced, rule-based decision-making with GIS but without LLM inference), and the method of this embodiment (GIS+Vision+LLM).
[0067] Evaluation indicators include the safety index:
[0068] in, This represents the number of samples in which the model outputs actions that strictly comply with industrial safety regulations (such as accurately executing "emergency stop" or "prohibition of operation" when faced with personnel intrusion). Efficiency Index: The smaller the value, the more flexible the model's decision-making, and the more effectively it can distinguish between "emergency avoidance" and "deceleration and detour".
[0069] Experimental results show that the safety index of the method in this embodiment reaches 0.91, and the efficiency index is 0.22. Compared with Method B, the safety index is improved by 21.3%, and the efficiency index is reduced by 42.1%. A typical case is as follows: When the intelligent robot encounters scattered cardboard boxes ahead, the system retrieves the GIS and finds that the current road width is only 2.0 meters (the robot itself is 1.2 meters wide), and determines that it cannot detour. The LLM combines the semantics of "narrow road" and "falling cargo" to output emergency braking and "prohibit automatic resumption of operation" instructions, and the comprehensive safety index reaches 0.98.
[0070] This embodiment also included an ablation experiment to verify the performance-enhancing effects of GIS priors and LLM inference.
[0071] In summary, the navigation method for intelligent robots in complex factory environments provided in this embodiment significantly improves decision robustness and safety in abnormal scenarios by integrating GIS geospatial priors, visual detection based on improved YOLOv11, spatial semantic text conversion, and thought chain reasoning of a large language model fine-tuned by LoRA. It solves the technical problems of insufficient decision robustness, lack of global spatial prior constraints, and poor semantic reasoning ability of intelligent robots in abnormal scenarios such as oil stains on the road, temporary construction, and fallen goods in the prior art.
[0072] The above description of the disclosed embodiments enables those skilled in the art to implement or use these embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of these embodiments. Therefore, these embodiments are not to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0073] Obviously, those skilled in the art can make several improvements and modifications to this embodiment without departing from the principle of this embodiment, and these improvements and modifications also fall within the protection scope of this embodiment.
[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.
[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0078] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0079] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A navigation method for an intelligent robot in a complex factory environment, characterized in that, Includes the following steps: A priori geospatial database of the factory area is constructed. Based on the geographic information system, data on the topology of the factory road, the physical attributes of the ground, and the spatial risk level are extracted. The data on the topology of the road, the physical attributes of the ground, and the spatial risk level are then quantified to obtain the priori geospatial data of the factory area. The visual perception module installed on the intelligent robot collects local images of the factory area in real time. Based on the deep learning obstacle detection model, abnormal obstacles are detected in the local images of the factory area to obtain obstacle category information and spatial position information of obstacles relative to the intelligent robot. The spatial semantic text conversion operator is used to convert the GIS vector data in the factory area GIS geospatial prior database into natural language text description, and the spatial semantic Prompt corresponding to the current visual perception area is obtained. The abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt are used as multimodal inputs and sent to the large language model cognitive reasoning module, which has been efficiently fine-tuned by LoRA parameters. The large language model cognitive reasoning module is based on the thought chain reasoning mechanism and outputs reasoning results that include intermediate reasoning steps and final navigation decision instructions. The navigation decision instructions generate motion control signals for the intelligent robot, driving the intelligent robot to perform safe navigation actions.
2. The navigation method for intelligent robots in complex factory environments according to claim 1, characterized in that, The construction of the factory area GIS geospatial prior database includes: By connecting the QGIS geographic information system software with the factory's digital twin platform, the vector map data of the factory area is read and parsed, and the topological data of the factory's roads are extracted, including road boundary lines, lane center lines, road intersections and road curvature radii. Based on the vector map data of the factory area, extract the physical attribute data of the factory area ground, including the ground friction coefficient and road slope value of each road section; Based on the factory area road topology and ground physical property data, and combined with the intelligent robot's braking safety distance formula, the safe braking distance of each spatial location in the factory area is quantitatively calculated. The formula for the braking safety distance of intelligent robots includes: ,in This represents the current real-time speed of the intelligent robot. The coefficient of friction of the ground is given. It is the acceleration due to gravity. For system response time, This is a slope correction term; Based on the safe braking distances of various spatial locations within the plant area, the heat map of historical accidents in the plant area, and the marking of densely populated areas, a quantitative model of the spatial risk level of each area of the plant area is established.
3. The navigation method for an intelligent robot in a complex factory environment as described in claim 1, characterized in that, The multimodal fusion process includes: Based on the real-time positioning coordinates of the intelligent robot, the current GPS positioning coordinates are transformed from the WGS84 coordinate system to the local plane coordinate system through a multi-level coordinate system transformation model. Then, the coordinates of the local plane coordinate system are mapped to the visual image pixel coordinate system to establish a two-way mapping relationship from spatial location to image pixels. Based on the bidirectional mapping relationship from spatial location to image pixels, the spatial location coordinates of the obstacles in the abnormal obstacle detection results are synchronously mapped to the prior GIS geospatial database of the factory area, and the GIS spatial attribute data corresponding to the obstacle location is retrieved and obtained, including the road width, ground friction coefficient and spatial risk level of the road segment where the obstacle is located. The GIS spatial attribute data corresponding to the obstacle location is used as a spatial constraint condition. Together with the abnormal obstacle category information and obstacle spatial location information output by the visual perception module, the data is processed through multimodal feature alignment and fusion, and then input into the large language model cognitive reasoning module.
4. The navigation method for an intelligent robot in a complex factory environment as described in claim 1, characterized in that, The method for constructing the spatial semantic textification transformation operator includes: Preprocess the GIS vector map data of the factory area and extract the lane width, road curvature, ground type, spatial risk level and historical abnormal event records of each road section in the factory area as spatial semantic feature attributes; Based on a predefined spatial semantic Prompt template, the spatial semantic feature attributes are organized into structured natural language text according to a three-level structure of current road segment physical constraint information, adjacent road segment risk warning information, and factory area global path planning information. The converted structured natural language text is scored for quality, low-quality semantic descriptions with scores below a preset threshold are filtered out, and a spatial semantic prompt that matches the current visual perception area is output. The spatial range matching is a distance threshold filtering with the current position of the intelligent robot as the center and the radius as the visual perception range.
5. The navigation method for an intelligent robot in a complex factory environment as described in claim 3, characterized in that, The navigation decision-making reasoning process based on the thought chain reasoning mechanism includes: Based on a predefined thought chain Prompt template, the spatial semantic Prompt, the abnormal obstacle detection results, and the current intelligent robot motion state parameters are combined into a complete input Prompt for the large language model according to the Prompt template format. The large language model is guided to output reasoning results according to the following reasoning steps: analyze the visual perception information of the current local environment of the factory area; retrieve the GIS global geospatial constraints of the current road segment; assess the threat level of the current obstacle by combining the braking safety distance and spatial risk level; perform safety reasoning by integrating local perception information and global spatial constraints; and generate navigation decision instructions. The navigation decision instructions output by the large language model are subject to security verification. The verification includes the consistency between the instruction action and the current GIS spatial constraints, and the prediction verification of whether the instruction will exceed the braking safety distance after execution. If the security verification fails, the instruction rejection mechanism is triggered and the instruction is regenerated.
6. The navigation method for an intelligent robot in a complex factory environment as described in claim 1, characterized in that, It also includes the following steps: A pre-built multimodal dataset for abnormal scenarios of industrial intelligent robots, IndAGV-4K, is constructed, including image frame data extracted from factory monitoring videos according to a preset sampling interval, as well as the corresponding GIS geographic coordinate attribute data; The extracted image frames are labeled with abnormal events, including personnel intrusion events, cargo spillage events, road damage events, and abnormal obstacle events. By associating GIS coordinates, GIS spatial attribute data corresponding to each image frame is exported from the prior GIS geospatial database of the factory area, including the ground friction coefficient, spatial risk level and road curvature radius of the area corresponding to the image frame. The expert decision-making process records collected by the annotation tool are processed into structured text to generate expert decision samples corresponding to each abnormal event; The abnormal event annotations, GIS spatial attribute data, expert decision samples, and factory environment attribute annotations are packaged into JSON format to form an industrial intelligent robot abnormal scenario multimodal dataset IndAGV-4K containing no less than 4,000 samples.
7. The navigation method for an intelligent robot in a complex factory environment as described in claim 6, characterized in that, The efficient fine-tuning method for LoRA parameters includes: Design a dedicated prompt word template for navigation scenarios of industrial intelligent robots. The dedicated prompt word template includes a description of the factory space constraint format, a reference example for decision-making in abnormal scenarios, and a description of safety verification rules. Based on the IndAGV-4K dataset, we constructed multimodal fine-tuning training samples that include visual perception description, GIS spatial semantic prompt, and expert decision-making process. The LoRA algorithm is used to perform low-rank decomposition and adaptation on the attention weight matrix of the large language model. The weight matrices of the Query layer and Value layer of the large language model are selected as the LoRA adaptation targets, and the LoRA rank parameter and scaling factor are set. The multimodal fine-tuning training samples are used to perform efficient parameter fine-tuning training on the graphics processing unit, and the weights of the LoRA adaptation matrix are updated so that the navigation decision accuracy of the fine-tuned large language model in abnormal scenarios of industrial intelligent robots is not lower than a preset threshold.
8. The navigation method for an intelligent robot in a complex factory environment as described in claim 3, characterized in that, The abnormal obstacle detection also includes: The visual perception module acquires local image sequences of the factory area in real time, and preprocesses the image sequences, including image denoising and brightness equalization. The preprocessed local images of the factory area are input into a target detection model based on an improved YOLOv11 convolutional neural network to obtain obstacle bounding boxes, obstacle categories, and detection confidence scores. Based on the bidirectional mapping relationship from spatial location to image pixels, the obstacle bounding box is mapped from the pixel coordinate system to the local planar coordinate system of the factory area GIS geospatial prior database to obtain the actual physical size and spatial location coordinates of the obstacle.
9. The navigation method for an intelligent robot in a complex factory environment as described in claim 5, characterized in that, The security reasoning in the inference step sequence also includes: Based on the current real-time speed of the intelligent robot, the ground friction coefficient, and the spatial risk level, the safe following distance threshold under the current state is calculated using the braking safety distance formula. Based on the spatial position information and relative motion speed of the abnormal obstacle relative to the intelligent robot, calculate the time collision (TTC) value with the abnormal obstacle; When the TTC value is less than the safety TTC benchmark value corresponding to the safety following distance threshold, the safety decision weight is adjusted based on the spatial risk level, and the output includes deceleration, stopping, or detour decision instructions.
10. A navigation system for intelligent robots in complex factory environments, characterized in that: The steps for implementing the navigation method of the intelligent robot in a complex factory environment according to any one of claims 1 to 9, wherein the navigation system of the intelligent robot in the complex factory environment comprises: The GIS spatial database construction module is used to construct a prior GIS geospatial database for the factory area. Based on the geographic information system, it extracts data on the topology of the factory area roads, ground physical attributes, and spatial risk levels, and quantifies the data on the topology of the roads, ground physical attributes, and spatial risk levels to obtain the prior global geospatial data of the factory area. The visual perception and detection module is used to acquire local images of the factory area in real time through the visual perception module installed on the intelligent robot, and to detect abnormal obstacles in the local images of the factory area based on a deep learning obstacle detection model, thereby obtaining obstacle category information and spatial position information of the obstacles relative to the intelligent robot. The spatial semantic conversion module is used to convert the GIS vector data in the prior GIS geospatial database of the factory area into natural language text descriptions through spatial semantic text conversion operators, so as to obtain the spatial semantic Prompt corresponding to the current visual perception area. The cognitive reasoning module is used to send the abnormal obstacle detection results obtained by the visual perception module and the spatial semantic prompt as multimodal inputs to the large language model cognitive reasoning module, which has been efficiently fine-tuned by LoRA parameters; the large language model cognitive reasoning module is based on the thought chain reasoning mechanism and outputs reasoning results including intermediate reasoning steps and final navigation decision instructions. The execution control module is used to generate motion control signals for the intelligent robot based on the navigation decision instructions, and drive the intelligent robot to perform safe navigation actions.