A method, device and equipment for path planning of an inspection robot and a storage medium

Abstract

The application discloses a kind of inspection robot path planning method, device, equipment and storage medium, it is related to intelligent control technical field, including: obtaining target instruction issued by user, target instruction is analyzed, to determine the task to be executed corresponding to target instruction;Corresponding query vector is generated based on the task to be executed, and corresponding environmental entity information is inquired from target vector knowledge base according to query vector;Target vector knowledge base stores the semantic environment model of target scene corresponding to target instruction, environmental entity information includes object in target scene and the stability weight of object, and the stability weight is used to indicate the immobility of object;Using preset path planner, the path of inspection robot is planned according to environmental entity information, to generate corresponding path instruction, and control inspection robot to execute the task to be executed according to path instruction.The application can realize the low-cost, high-adaptive path planning of inspection robot in complex environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a path planning method, apparatus, device, and storage medium for an inspection robot. Background Technology

[0002] Inspection robots, as an important branch of special-purpose robots, have been widely used in high-risk or highly repetitive fields such as power, petrochemical, and rail transportation. Their core value lies in reducing human risks and improving operational efficiency and accuracy. Traditional manual inspections face problems such as threats to personal safety, low efficiency, and strong data subjectivity. In contrast, intelligent inspection robots, by integrating technologies such as environmental perception, autonomous path planning, and intelligent recognition and decision-making, achieve a closed-loop operation of the entire process of "perception-processing-decision-execution," becoming a key automated solution to replace manual labor.

[0003] From a technological evolution perspective, inspection robots have developed from early pre-set tracks or remote control operations to highly autonomous third-generation intelligent systems. Their intelligence primarily relies on the collaborative efforts of core technologies such as autonomous path planning and SLAM (Simultaneous Localization and Mapping), intelligent recognition and edge computing, multi-sensor fusion, and collaborative operation and maintenance. Lithium-based SLAM and visual SLAM provide robots with localization and mapping capabilities, deep learning models assist in fault diagnosis and anomaly identification, and multi-sensor fusion enhances the completeness and reliability of environmental perception.

[0004] However, existing technologies still have significant shortcomings: First, they lack environmental adaptability, and SLAM technology is prone to positioning drift under sudden changes in lighting, electromagnetic interference, or dynamic obstacles; second, intelligent decision-making has limitations, relying heavily on preset rules or single-point recognition, lacking a deep understanding of environmental semantics, and making it difficult to perform complex tasks that require common-sense reasoning; third, cost and deployment efficiency are prominent issues, with high-performance sensors driving up system costs, and requiring professionals to remap and debug when the environment changes, resulting in poor versatility and ease of maintenance.

[0005] In summary, how to achieve low-cost and highly adaptive path planning for inspection robots in complex environments is a pressing technical problem that needs to be solved. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a path planning method, apparatus, device, and storage medium for inspection robots, enabling low-cost and highly adaptive path planning for inspection robots in complex environments. The specific solution is as follows: Firstly, this application provides a path planning method for an inspection robot, including: Obtain the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction; Based on the task to be executed, a corresponding query vector is generated, and the corresponding environmental entity information is queried from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, and the stability weights are used to represent the immobility of the objects; Using a preset path planner, the inspection robot's path is planned based on the environmental entity information to generate corresponding path instructions, and the inspection robot is controlled to execute the task to be performed according to the path instructions.

[0007] Optionally, the inspection robot path planning method further includes: The system acquires image data and laser point cloud data of the target scene, and determines the objects in the target scene based on the image data and the laser point cloud data. Label the category and stability weight of the object in the image data to obtain the corresponding labeled image; The initial semantic segmentation model is trained and fine-tuned using the labeled images to obtain the corresponding target semantic segmentation model; the initial semantic segmentation model is a model based on the Transformer architecture.

[0008] Optionally, the process of labeling the category and stability weight of the object in the image data includes: Based on the category corresponding to the object, the stability weight of the object is matched from a preset stability weight setting rule library; the preset stability weight setting rule library is used to store the stability weights corresponding to the categories of the object; Alternatively, the historical trajectory of the object can be obtained, and the historical movement frequency and historical movement amplitude of the object can be determined based on the historical trajectory. Then, the stability weight of the object can be generated based on the historical movement frequency and historical movement amplitude using a preset time series prediction model.

[0009] Optionally, the inspection robot path planning method further includes: The inspection robot is used to perform SLAM mapping on the target scene to generate a point cloud map corresponding to the target scene, and key frame images generated during the SLAM mapping process are obtained. The target semantic segmentation model is used to perform semantic segmentation on the keyframe image, and the semantic segmentation result is associated with the point cloud map to determine the text description of the object in the point cloud map based on the association result; the text description includes the object's category, stability weight, and geometric position information; The semantic environment model of the target scene is constructed based on the textual description of the object, and the textual description of the object in the semantic environment model is encoded into a first vector through a preset sentence embedding model, and the first vector is stored in the target vector knowledge base.

[0010] Optionally, parsing the target instruction to determine the task to be executed corresponding to the target instruction includes: The target instruction is parsed using natural language processing technology, and the user intent and motion endpoint information are determined based on the parsing results. The task to be executed corresponding to the target instruction is then determined based on the user intent and the motion endpoint information. Accordingly, the step of querying the corresponding environmental entity information from the target vector knowledge base based on the query vector includes: From the first vectors stored in the target vector knowledge base, a target vector whose first similarity to the query vector exceeds a first preset similarity threshold is determined, and the environmental entity information is determined based on the text description corresponding to the target vector; the first similarity is the cosine similarity, Euclidean distance, or inner product between the query vector and the first vector.

[0011] Optionally, the step of using a preset path planner to plan the path of the inspection robot based on the environmental entity information to generate corresponding path instructions includes: Using the preset path planner, the inspection robot's path is planned according to the first path planning algorithm, the second path planning algorithm, and the environmental entity information, so as to generate the path instructions; The objective functions of both the first and second path planning algorithms incorporate the stability weight constraint term to control the inspection robot to bypass a first object with a stability weight higher than a preset threshold, pass through a second object with a stability weight not higher than the preset threshold, or wait for the second object to be removed before re-planning the path. The first path planning algorithm is the A* algorithm or the D Lite algorithm, and the second path planning algorithm is the TEB algorithm, the dynamic window method, or the model predictive control algorithm.

[0012] Optionally, the process of controlling the inspection robot to execute the task according to the path instructions further includes: The inspection robot collects real-time data from the target scene using its multimodal sensors. The real-time data includes environmental images collected in real-time by a camera, environmental point clouds collected in real-time by a lidar, and speed data of the inspection robot collected in real-time by an inertial measurement unit. The object detection model is used to identify dynamic objects in the target scene based on the real-time data, and a preset semantic segmentation network is used to identify static objects in the target scene based on the real-time data. Based on the dynamic objects and the static objects, determine the real-time objects in the target scene, and determine the stability weights of the real-time objects; Based on the ORB-SLAM3 framework, the real-time object, the stability weight of the real-time object, and the environmental point cloud in the real-time data are associated. A semantic octree map is constructed based on the association results, and a real-time perception result of the target scene is generated based on the semantic octree map. The real-time perception result includes the real-time object and the stability weight of the real-time object. The real-time perception result is encoded into a corresponding second vector, and the second similarity between the second vector and each of the first vectors stored in the target vector knowledge base is calculated. If none of the second similarities exceed the second preset similarity threshold, the target vector knowledge base is updated based on the second vector to update the semantic environment model of the target scene, so as to perform new path planning for the inspection robot based on the updated semantic environment model.

[0013] Secondly, this application provides a path planning device for an inspection robot, comprising: The target instruction acquisition module is used to acquire the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction; The query vector generation module is used to generate a corresponding query vector based on the task to be executed, and to query the corresponding environmental entity information from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, the stability weights being used to represent the immobility of the objects; The path planning module is used to plan the path of the inspection robot based on the environmental entity information using a preset path planner, so as to generate corresponding path instructions and control the inspection robot to execute the task to be performed according to the path instructions.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned inspection robot path planning method.

[0015] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned inspection robot path planning method.

[0016] In this application, the target instruction issued by the user is first obtained and parsed to determine the task to be executed corresponding to the target instruction. Then, a corresponding query vector is generated based on the task to be executed, and corresponding environmental entity information is retrieved from the target vector knowledge base according to the query vector. The target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction. The environmental entity information includes objects in the target scene and the stability weights of the objects, whereby the stability weights represent the immobility of the objects. Finally, a preset path planner is used to plan the path for the inspection robot based on the environmental entity information to generate corresponding path instructions and control the inspection robot to execute the task to be executed according to the path instructions. As can be seen from the above, this application first obtains and parses the user's target instruction to determine the task to be executed, then generates a query vector based on the task to be executed, retrieves environmental entity information containing objects in the target scene and stability weights representing their immobility from the target vector knowledge base storing the scene semantic environment model, and finally plans a path for the inspection robot and generates path instructions by combining the preset path planner with the environmental entity information, controlling the robot to execute the task. In this way, the semantic environment model stored in the target vector knowledge base of this application contains various objects in the target scene and stability weights representing their immobility. This enables the inspection robot not only to know the existence of objects in the environment, but also to understand the relative stability and mobility of objects in the environment. This allows the inspection robot to go beyond geometric obstacle avoidance and perform accurate path planning based on object stability. This effectively improves the adaptability and decision-making rationality in complex dynamic environments, reduces the need for human intervention, and ensures the safety and efficiency of task execution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1A flowchart of a path planning method for an inspection robot provided in this application; Figure 2 This application provides a specific flowchart for the path planning of an inspection robot. Figure 3 A schematic diagram of a path planning device for an inspection robot provided in this application; Figure 4 This application provides a structural diagram of an electronic device. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Inspection robots, as an important branch of special-purpose robots, have been widely used in high-risk or highly repetitive fields such as power, petrochemical, and rail transportation. Their core value lies in reducing human risks and improving operational efficiency and accuracy. Traditional manual inspections face problems such as threats to personal safety, low efficiency, and strong data subjectivity. Intelligent inspection robots, by integrating technologies such as environmental perception, autonomous path planning, intelligent recognition, and decision-making, achieve a closed-loop operation of the entire process of "perception-processing-decision-execution," becoming a key automated solution to replace manual labor. From a technological evolution perspective, inspection robots have developed from early preset tracks or remote control operations to highly autonomous third-generation intelligent systems. Their intelligence mainly relies on the collaborative work of core technologies such as autonomous path planning and SLAM technology, intelligent recognition and edge computing, multi-sensor fusion, and collaborative operation and maintenance. Laser SLAM and visual SLAM provide robots with localization and mapping capabilities, deep learning models assist in fault diagnosis and anomaly identification, and multi-sensor fusion improves the completeness and reliability of environmental perception. However, existing technologies still have significant shortcomings: First, they lack environmental adaptability, and SLAM technology is prone to positioning drift under sudden changes in lighting, electromagnetic interference, or dynamic obstacles; second, intelligent decision-making has limitations, relying heavily on preset rules or single-point recognition, lacking a deep understanding of environmental semantics, and making it difficult to perform complex tasks requiring common-sense reasoning; third, cost and deployment efficiency are prominent issues, with high-performance sensors driving up system costs, and requiring professional personnel to remap and debug when the environment changes, resulting in poor versatility and ease of maintenance. Therefore, this application provides a path planning scheme for inspection robots, enabling low-cost and highly adaptive path planning for inspection robots in complex environments.

[0021] See Figure 1As shown in the figure, an embodiment of the present invention discloses a path planning method for an inspection robot, which may include: Step S11: Obtain the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction.

[0022] In this embodiment, the user-issued target instruction is first parsed to determine the corresponding task to be executed. The specific process may include: parsing the target instruction using natural language processing (NLP) technology, determining the user's intent and endpoint information based on the parsing results, and determining the corresponding task to be executed based on the user's intent and endpoint information. Specifically, if the user's target instruction is "go to meeting room 101 to retrieve a file," the NLP module, such as a BERT (Bidirectional Encoder Representations from Transformers) model, parses the target instruction, extracts the user's intent ("retrieve file") and endpoint information ("meeting room 101"), and determines the corresponding task to be executed.

[0023] It should be noted that for the target scenario in which the inspection robot operates, a semantic environment model rich in semantics and prior knowledge, which can be queried and reasoned about by the computer, needs to be constructed first. The process of determining the target semantic segmentation model can include: firstly, collecting image data and laser point cloud data of the target scenario, and determining the objects in the target scenario based on the image data and laser point cloud data; then, labeling the categories and stability weights of the objects in the image data to obtain the corresponding labeled images; finally, using the labeled images to train and fine-tune the initial semantic segmentation model to obtain the corresponding target semantic segmentation model; the initial semantic segmentation model is a model based on the Transformer architecture.

[0024] Specifically, RGB-D image data and laser point cloud data of target scenes, such as offices and warehouses, are collected to identify several objects within the scene. Next, the image data is labeled, including not only pixel-level object categories such as "chair," "table," and "electrical distribution box," but also initial stability weight labels are associated with each category to obtain corresponding labeled images. A dataset is then constructed based on these labeled images. Subsequently, a lightweight semantic segmentation model based on the Transformer architecture, such as SegFormer, is used. This model is pre-trained on large public datasets, such as COCO and ADE20K, and fine-tuned using a precisely labeled RGB-D image dataset. The fine-tuning process employs the Adam optimizer with weight decay, monitoring the MIoU (Mean Intersection over Union) metric on the validation set to prevent overfitting. During training, the model can accept an input resolution of 640x480 and output semantic labels covering 8-10 categories of objects specific to the target scene. During fine-tuning, the learning rate is set to 1e-4, and the model is iterated approximately 10,000 times to achieve convergence. The final trained target semantic segmentation model has the ability to accurately identify environmental elements in the target scene.

[0025] It should be noted that this embodiment can employ a variety of different methods to determine the stability weights corresponding to the objects.

[0026] In one specific implementation, the stability weight of an object can be matched from a preset stability weight setting rule library based on the object's category. This preset stability weight setting rule library stores stability weights corresponding to the object's category. Specifically, based on semantic recognition, a stability weight coefficient can be automatically associated with each identified object according to the predefined stability weight setting rule library. This coefficient quantifies the degree to which the object's position is fixed in the environment, i.e., its immobility. For example, a load-bearing wall has a coefficient of 0.95, representing a long-term stable spatial constraint; a fixed cabinet has a coefficient of 0.7, an office chair has a coefficient of 0.3, and a pedestrian has a coefficient of 0.1, representing a short-term variable passage condition.

[0027] In another specific implementation, the historical trajectory of an object can be acquired, and the historical movement frequency and amplitude of the object can be determined based on the historical trajectory. Then, a preset time-series prediction model can be used to generate stability weights for the object based on its historical movement frequency and amplitude. Specifically, this can be dynamically generated using online learning. For example, a lightweight time-series prediction model, such as LSTM (Long Short-Term Memory) or Transformer, can be deployed to automatically learn and update the object's stability weights by analyzing the object's movement frequency and amplitude in its historical trajectory. The stability weights of objects with high movement frequency, such as pedestrians, will automatically decrease, while the stability weights of objects that remain stationary for a long time, such as shelves, will remain unchanged or increase.

[0028] In this embodiment, a target semantic segmentation model with scene fine-tuning can be used to perform deep semantic analysis on the environmental data of the target scene collected by the inspection robot. Stability weight coefficients are injected into all identified environmental entities, such as walls, doors, tables, chairs, and equipment, thereby forming a semantic environment model containing multi-dimensional information of geometry, semantics, and stability. After encoding, the model is stored in the target vector database as the core knowledge base. The specific process may include: first, using the inspection robot to perform SLAM mapping on the target scene to generate a point cloud map corresponding to the target scene, and acquiring keyframe images generated during the SLAM mapping process; then, using the target semantic segmentation model to perform semantic segmentation on the keyframe images, and associating the semantic segmentation results with the point cloud map to determine the text descriptions of objects in the point cloud map based on the association results; the text descriptions include the object's category, stability weight, and geometric position information; finally, based on the text descriptions of the objects, a semantic environment model of the target scene is constructed, and the text descriptions of the objects in the semantic environment model are encoded into a first vector through a preset sentence embedding model, and the first vector is stored in the target vector knowledge base.

[0029] Specifically, when the inspection robot first explores the target scene, it runs the SLAM mapping process and uses the finely tuned target semantic segmentation model to perform semantic segmentation on the keyframe images generated during the SLAM mapping process. The semantic segmentation results (object category, pixel location) are associated with the point cloud map generated by SLAM, assigning semantic labels, such as "workstation-chair" or "equipment-distribution box," and stability weights to each map point or object instance in the point cloud map, resulting in corresponding text descriptions, such as category: chair; initial stability weight: 0.3; approximate location: area A. Then, a semantic environment model of the target scene is constructed based on the text descriptions of each object. Finally, using a sentence embedding model, such as Sentence-BERT, the text descriptions of each object corresponding to the semantic environment model are transformed into high-dimensional vectors, such as 512-dimensional first vectors, and stored in a target vector database, such as ChromaDB or Milvus, completing the initialization of the knowledge base. Each vector in the target vector database corresponds one-to-one with an entity in the environment, and the vector knowledge base serves as a persistent storage and efficient retrieval center for the semantic environment model.

[0030] Step S12: Generate a corresponding query vector based on the task to be executed, and query the corresponding environmental entity information from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, and the stability weights are used to represent the immobility of the objects.

[0031] In this embodiment, the task to be executed obtained from parsing the target instruction is transformed into a corresponding query vector. Then, based on the query vector, the corresponding environmental entity information is queried from the target vector knowledge base. The specific process may include: determining the target vector whose first similarity to the query vector exceeds a first preset similarity threshold from the first vectors stored in the target vector knowledge base, and determining the environmental entity information based on the text description corresponding to the target vector; the first similarity is the cosine similarity, Euclidean distance, or inner product between the query vector and the first vector. In this way, by performing a similarity search in the target vector knowledge base, the environmental entity corresponding to "Meeting Room 101" and all its attributes, such as location, surrounding environment, and access rules, can be quickly located.

[0032] Step S13: Using a preset path planner, the inspection robot's path is planned based on the environmental entity information to generate corresponding path instructions, and the inspection robot is controlled to execute the task to be performed according to the path instructions.

[0033] In this embodiment, a preset path planner is used to plan the path of the inspection robot based on environmental entity information to generate corresponding path instructions. The specific process may include: using the preset path planner to plan the path of the inspection robot based on a first path planning algorithm, a second path planning algorithm, and environmental entity information to generate path instructions; wherein, the objective functions of the first path planning algorithm and the second path planning algorithm both introduce a constraint term of stability weight to control the inspection robot to bypass a first object with a stability weight higher than a preset threshold, pass through a second object with a stability weight not higher than the preset threshold, or wait for the second object to be removed before re-planning the path. The first path planning algorithm is the A* algorithm or the D Lite algorithm, and the second path planning algorithm is the TEB algorithm, the dynamic window method, or the model predictive control algorithm.

[0034] Specifically, the path planner employs a hierarchical planning strategy, using a global planning layer and local planning layers for path planning of the inspection robot. In one specific implementation, the global planning layer uses an improved A... The algorithm, whose cost function has been redesigned and deeply integrated with stability weight coefficients, takes the following form: ; Where f(n) is the estimated total cost from the starting point through node n to the target point (end point); g(n) is the actual cost already spent from the starting point to node n; h(n) is the heuristic cost from node n to the target point, such as Euclidean distance; StabilityWeight(n) is the average stability weight of the environment associated with the location or path segment of node n, which can be calculated from the stability weights of related objects in the target vector knowledge base, such as taking the maximum value or average value; This is an adjustable gain coefficient used to control the sensitivity of the path planning system to stability weights. For example, a larger value can be set in "safety first" mode. This value enables the inspection robot to avoid low-stability areas to an extreme extent; a smaller value is set in the "efficiency-first" mode. value.

[0035] Based on the above A The algorithm's cost function significantly reduces the cost when the inspection robot traverses a low-stability region, such as when StabilityWeight(n) < 0.3, encouraging the robot to wait or proceed cautiously. Conversely, when the inspection robot traverses a high-stability region, such as when StabilityWeight(n) > 0.7, the cost increases sharply, forcing the robot to plan an alternative path. It should be noted that the global planning layer can also employ the D Lite algorithm to better adapt to dynamic environmental changes.

[0036] In one specific implementation, the local planning layer employs the TEB (Time Elastic Band) algorithm. The TEB algorithm's objective function also incorporates a constraint term on the stability weights of the trajectory points' locations to ensure smooth local paths that conform to stability decision logic. It should be noted that the local planning layer can also use the dynamic window method or model predictive control algorithms, incorporating stability weights into the cost function or constraints.

[0037] It should be noted that the path planning rule can be formalized as: IF ("glass door") AND target = "ahead" THEN set speed ≤ 0.5m / s AND send audio-visual cues. The stability rule is directly reflected in the assignment of stability weight factors. In this way, system administrators can intuitively control the behavior style of the inspection robot by adjusting the influence of the stability weight factors in the cost function. For example, in the "safety first" mode, a more conservative strategy is adopted even for low-stability obstacles, thereby achieving predictability and controllability of path planning behavior.

[0038] In this embodiment, the path instructions generated by the path planner are sent to the motion controller at the bottom layer of the inspection robot via the ROS (Robot Operating System), controlling the inspection robot to move according to the planned path. Simultaneously, the actual pose and speed of the inspection robot, as well as real-time readings from surrounding environmental sensors, can be continuously monitored. All monitored data, especially environmental changes that do not conform to expectations, will be used to update the semantic environment model in real time. For example, when the inspection robot discovers that a temporary obstacle with a low stability weight on a path has been cleared, the obstacle change information will be updated in the semantic environment model so that a better path can be selected during the next path planning.

[0039] In other words, to ensure that the semantic environment model can reflect changes in the target scene in real time, the process of controlling the inspection robot to execute the task according to the path instructions may also include: firstly, collecting real-time data in the target scene through the inspection robot's multimodal sensors; the real-time data includes environmental images collected in real time by the camera, environmental point clouds collected in real time by the lidar, and speed data of the inspection robot collected in real time by the inertial measurement unit; then, using the target detection model to identify dynamic objects in the target scene based on the real-time data, and using a preset semantic segmentation network to identify static objects in the target scene based on the real-time data; subsequently, determining the real-time objects in the target scene based on the dynamic and static objects, and determining the stability weights of the real-time objects; Next, based on the ORB-SLAM3 framework, real-time objects, their stability weights, and environmental point clouds in real-time data are associated. A semantic octree map is constructed based on the association results, and real-time perception results of the target scene are generated based on the semantic octree map. The real-time perception results include real-time objects and their stability weights. Then, the real-time perception results are encoded into corresponding second vectors, and the second similarity between the second vectors and each first vector stored in the target vector knowledge base is calculated. If each second similarity does not exceed the second preset similarity threshold, the target vector knowledge base is updated based on the second vectors to update the semantic environment model of the target scene, so as to perform new path planning for the inspection robot based on the updated semantic environment model.

[0040] Specifically, the inspection robot's multimodal sensors include LiDAR, cameras, and IMU (Inertial Measurement Unit). Each sensor employs a hardware synchronization mechanism to control timestamp deviations to less than 1ms. In this embodiment, an extended Kalman filter algorithm can be used to fuse precise geometric information from the LiDAR point cloud with rich texture and semantic information provided by vision. During the inspection robot's operation, real-time data from the target scene is collected through the multimodal sensors. The front end uses a lightweight object detection model, such as YOLOv5s (YOLO, You Only Look Once), to quickly identify dynamic objects. Combined with a semantic segmentation network, such as DeepLabv3+, static objects in the target scene are identified, thus refining the understanding of the static environment. This allows for the determination of real-time objects in the target scene and their stability weights. Next, the back end, based on the ORB-SLAM3 framework, associates semantic labels, including the categories and stability weights of real-time objects, with the environmental point cloud in the real-time data. Based on the association results, a semantic octree map is constructed and updated in real-time, and a real-time perception result of the target scene is generated based on the semantic octree map. Then, the second vector corresponding to the real-time perception result is compared with the first vectors stored in the target vector knowledge base for similarity. If the second similarity between the second vector and each first vector does not exceed the second preset similarity threshold, it is determined that the object in the target scene has a significant difference, such as a highly stable object disappearing or moving. The target vector knowledge base is then updated according to the second vector to realize the real-time update of the semantic environment model, so as to plan the next path of the inspection robot based on the updated semantic environment model.

[0041] It should be noted that, because this embodiment deeply integrates multimodal data from LiDAR, visual cameras, and other sources, and is equipped with a dedicated dynamic spatiotemporal synchronization module for timestamp alignment, it possesses an inherent redundancy and complementary mechanism when facing single sensor failure or data conflict. Specifically, by introducing stability weights, during data fusion and SLAM localization, it prioritizes trusting and matching the features of highly stable objects, such as corners and pillars (stability weight > 0.8), effectively suppressing interference from dynamic objects, such as moving pedestrians, or unstable temporary obstacles, reducing localization drift, and significantly improving the robustness of the inspection robot's localization in dynamic environments. As a result, the stability and reliability of this embodiment are greatly enhanced in challenging scenarios such as dynamic environments, complex lighting conditions, or temporary failures of some sensors. For example, when the visual sensor temporarily fails due to strong light, basic path planning can be maintained by relying on laser point cloud data and ultrasonic data; in a busy hall, it can maintain more stable localization, avoiding localization loss or path oscillation caused by mismatches.

[0042] In summary, see Figure 2 As shown, the specific process of path planning for the inspection robot can be described as follows: (1) Environmental semantic modeling and knowledge base construction: Using a target semantic segmentation model that has been fine-tuned for the scene, deep semantic analysis is performed on the environmental data of the target scene collected by the inspection robot. The key innovation is that stability weight coefficients are injected into all identified environmental entities, such as walls, doors, tables and chairs, and equipment, thereby forming a semantic environment model containing multi-dimensional information of geometry, semantics, and stability. After encoding, the model is stored in the target vector database to obtain the corresponding vector knowledge base, which serves as the core knowledge base.

[0043] (2) Perception-driven dynamic model update: During operation, the inspection robot collects multimodal perception data in real time, including laser, vision, and ultrasound, as continuous input to drive the online evolution of the semantic environment model. By parsing the real-time data and comparing it with the semantic environment model in the vector knowledge base, the semantic environment model is dynamically updated, such as adjusting the position of moving objects and correcting their stability weights, to ensure that the semantic environment model is always synchronized with the real environment.

[0044] (3) Task parsing and reasoning: Upon receiving a task instruction, such as a natural language instruction, the planning agent directly interacts with the latest semantic environment model in the vector knowledge base. By querying the vector knowledge base and integrating prior knowledge such as stability weights, the planning algorithm generates a path with common sense judgment. For example, for low-stability obstacles, such as a handcart, it chooses to wait or approach cautiously, while for high-stability obstacles, such as a load-bearing wall, it resolutely detours.

[0045] (4) Path execution and closed-loop feedback: The inspection robot moves along the planned path and monitors the data during the operation of the inspection robot in real time. Based on the monitoring data, the semantic environment model in the vector knowledge base is updated in real time so as to select a better path when planning the path next time.

[0046] As can be seen from the above, this embodiment first obtains the target instruction issued by the user and parses the target instruction to determine the task to be executed corresponding to the target instruction; then, a corresponding query vector is generated based on the task to be executed, and the corresponding environmental entity information is queried from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, the stability weights being used to represent the immobility of the objects; finally, a preset path planner is used to plan the path of the inspection robot according to the environmental entity information to generate corresponding path instructions, and control the inspection robot to execute the task to be executed according to the path instructions. As can be seen from the above, this embodiment first obtains and parses the user's target instruction to determine the task to be executed, then generates a query vector based on the task to be executed, retrieves environmental entity information containing objects in the target scene and stability weights representing their immobility from the target vector knowledge base storing the scene semantic environment model, and finally plans a path for the inspection robot and generates path instructions by combining the preset path planner with the environmental entity information, controlling the robot to execute the task. In this way, this embodiment stores a semantic environment model containing various objects in the target scene and stability weights representing their immobility through a target vector knowledge base. This enables the inspection robot not only to know the existence of objects in the environment, but also to understand the relative stability and mobility of objects in the environment. This allows the inspection robot to move beyond obstacle avoidance at the geometric level and perform precise path planning based on object stability. This effectively improves the adaptability and decision-making rationality in complex dynamic environments, reduces the need for human intervention, and ensures the safety and efficiency of task execution.

[0047] Next, this embodiment will describe the specific process of path planning for the inspection robot in a specific scenario.

[0048] In the first specific implementation, the scenario is described as follows: An inspection robot is performing a delivery task in an open-plan office. The target location is partially blocked by a temporarily parked trolley (stability weight of 0.2). Traditional path planning algorithms would rigidly perform detours, while this embodiment achieves human-like intelligent decision-making by introducing the stability weight of the trolley. The technical implementation is as follows: 1. Multimodal perception and semantic understanding The inspection robot is equipped with a visual sensor (RGB-D camera) and a LiDAR simultaneously to collect environmental data. The visual sensor captures RGB images at a resolution of 640×480, while the LiDAR provides 360-degree point cloud data. A lightweight initial semantic segmentation model based on the Transformer architecture, such as SegFormer, is used to perform pixel-level semantic segmentation of the images, identifying object categories such as "handcart," "desk," and "wall." The initial semantic segmentation model is fine-tuned using labeled data from an office scene, with a learning rate of 1e-4, and converges after approximately 10,000 iterations, yielding the target semantic segmentation model.

[0049] 2. Stability weight coefficient injection

[0050] Based on a predefined rule base, a stability weight is automatically associated with each identified object. A handcart, as a temporary obstacle, is assigned a low stability weight of 0.2; fixed workstations and walls are assigned high stability weights of 0.7-0.9. These weight coefficients quantify the degree to which an object's position is fixed in the environment, serving as a key basis for path planning decisions.

[0051] 3. Improved Path Planning Algorithm

[0052] Adopting improved A The algorithm performs global path planning. The cost function has been redesigned, and stability weights have been deeply integrated. ; Where g(n) is the actual cost from the starting point to the current node n, h(n) is the heuristic cost from node n to the destination, and w(n) is the dynamic weight coefficient. When the path passes through a low-stability region, such as near a handcart, the cost decreases, allowing the inspection robot to approach or wait; when passing through a high-stability region, the cost increases sharply, forcing a detour.

[0053] 4. Dynamic obstacle avoidance and waiting strategies

[0054] As the inspection robot slowly approaches the trolley, its visual sensors continuously monitor environmental changes. When it detects a worker approaching the trolley, the robot initiates a waiting strategy. By fusing data from multiple sensors using a Kalman filter algorithm, it estimates the worker's position and movement in real time. During the waiting period, the robot maintains a safe distance, such as 0.5 meters, and sends out audio-visual cues. After the worker pushes the trolley away (further confirming its low stability), the robot continues its journey.

[0055] 5. Decision-making process analysis

[0056] The cost of the two paths is calculated: due to the low stability weight of the handcart, the path directly approaching the handcart has a higher cost, while the path that detours around the fixed workstation has a higher cost. Based on the common-sense reasoning that "low-stability objects may move," the path with the lower cost is selected. This decision reflects anthropomorphic intelligent judgment, rather than simple geometric obstacle avoidance. Compared to traditional algorithms that rigidly execute detours, this embodiment achieves anthropomorphic decision-making efficiency through stability weights, saving time and energy.

[0057] In the second specific implementation, the scenario is described as follows: The inspection robot is navigating through a crowded shopping mall lobby. The GPS (Global Positioning System) signal is weak, and there are numerous moving pedestrians (stability weight 0.1) and glass curtain walls (unfriendly to LiDAR). Traditional SLAM algorithms are prone to positioning drift in this highly dynamic environment. The technical implementation is as follows: 1. Multi-sensor fusion positioning Multi-sensor fusion localization is achieved using LiDAR, an RGB-D camera, and an IMU. A hardware synchronization mechanism ensures that the timestamp deviation between each sensor is less than 1ms. The LiDAR provides 360-degree point cloud data, the RGB-D camera provides color images and depth information, and the IMU provides acceleration and angular velocity data.

[0058] 2. Feature selection based on stability weights

[0059] During SLAM localization, the system prioritizes features matched with highly stable landmarks, such as load-bearing columns in shopping malls (weight = 0.9) and fixed service counters (weight = 0.7). This strategy is implemented using an extended Kalman filter algorithm; the observation noise covariance matrix is ​​dynamically adjusted based on the stability weights. ; in, The uncertainty weight matrix of the object in three-dimensional space is a diagonal matrix constructed by the diag function and is used to quantify the degree of influence of the dynamic changes in the object's position on path planning. The variance of the object's position in the three-dimensional coordinate system x, y, z axes reflects the measurement error or distribution range of the object's initial position. The larger the variance, the higher the uncertainty of the initial position. This represents the stability weight of the object.

[0060] Landmarks with higher stability weights have lower observation noise and higher confidence in location estimation. This mechanism effectively filters out interference from unstable moving pedestrians.

[0061] 3. Removal of dynamic objects

[0062] By employing an improved real-time semantic SLAM framework, combined with YOLOv8 object detection and a CRF (Conditional Random Field) fine-grained segmentation module, dynamic objects are accurately identified and eliminated. During feature tracking, only static feature points are used for pose estimation, avoiding localization drift caused by dynamic objects.

[0063] 4. Multimodal data fusion

[0064] Visual sensor and LiDAR data are fused at the feature level using a Transformer architecture. The Transformer's self-attention mechanism models long-range dependencies. In areas unfavorable to LiDAR, such as glass curtain walls, the system prioritizes visual features; in texture-rich areas, it fuses precise geometric information from the LiDAR. It's worth noting that besides using the Transformer's cross-attention mechanism, early fusion based on convolutional neural networks (merging data at the input layer), or late fusion (extracting features separately and then concatenating them), or using graph neural networks to model the environment as a graph structure for relational reasoning, can all achieve complementary fusion of multimodal information.

[0065] 5. Decision-making process analysis

[0066] Despite visual sensors being obstructed by pedestrians and lidar malfunctioning in front of glass curtain walls, the system still maintains centimeter-level high-precision positioning by "locking onto" several highly stable landmarks. This process demonstrates the robust advantages of multimodal fusion and stability weights in complex environments.

[0067] In the third specific implementation, the scenario is described as follows: A user instructs the inspection robot to "go check the innermost part of the third row of shelves." The inspection robot needs to understand the semantics of the instruction, plan its path, and adjust its behavior as it approaches the target. The technical implementation is as follows: 1. Natural Language Command Parsing User commands are first parsed by a BERT-based natural language processing module. This module employs a bidirectional encoder architecture, enabling it to consider contextual information simultaneously. The commands are broken down into two subtasks: intent recognition and slot filling. Intent recognition: The user's intent is identified as "navigation + inspection"; Slot filling: Extract key parameters "third row of shelves" and "innermost"; The system captures key information in instructions through an attention mechanism, such as "third row" indicating spatial order and "innermost" indicating the end of the path, and constructs tasks to be executed.

[0068] 2. Vector knowledge base query

[0069] The parsed task to be executed is transformed into a query vector, and a similarity search is performed in the vector knowledge base. The vector knowledge base uses the Sentence-BERT model to encode the semantic environment model into a 512-dimensional feature vector. Through cosine similarity calculation, the environmental entity corresponding to "shelf" and its attributes, such as location, stability weight = 0.6, and surrounding environment, are quickly located.

[0070] 3. Semantic rule reasoning

[0071] Reasoning based on semantic rules in the knowledge base: "Third row": It is necessary to identify the spatial arrangement order of the shelves; "Innermost": usually means the end of the path, where a direct route needs to be planned; "Inspect": This implies that you may need to observe a low point or a specific angle; These rules are formalized into logical expressions to guide path planning and behavior adjustment.

[0072] 4. Path planning and behavior adjustment

[0073] The path planner uses an improved A The algorithm is integrated with the TEB algorithm. Upon approaching the target, the system automatically reduces its speed to 0.3 m / s and adjusts the sensor angle to scan the bottom of the shelf. This behavioral adjustment is based on rule hints in the knowledge base: the "inspection" action may require observing the lower areas.

[0074] 5. Multimodal sensing feedback

[0075] During the process, the inspection robot continuously monitors the status of the shelves using visual sensors. If an anomaly is detected, such as goods falling off, the system triggers an alarm and replans its route. This closed-loop feedback mechanism ensures reliable task execution.

[0076] 6. Decision-making process analysis

[0077] The inspection robot plans a path directly to the target and automatically reduces its speed as it approaches, while adjusting the sensor angles to scan the bottom of the shelf. This series of actions demonstrates the system's deep understanding of natural language commands, rather than just point-to-point movement. This makes the robot's path planning behavior no longer just mechanical obstacle avoidance, but possesses common-sense reasoning and foresight, resulting in safer behavior, such as anticipating glass doors and slowing down, and better conforming to social norms, such as maintaining a polite distance from pedestrians. The overall intelligence level is significantly improved, making it more adaptable to dynamic and complex real-world scenarios.

[0078] In the fourth specific implementation, the scenario is described as follows: When the inspection robot is performing inspection tasks in an outdoor park, GPS signal obstruction, drastic environmental changes, or sensor malfunctions may cause map drift or complete loss of positioning information in the SLAM system. Traditional path planning methods require manual intervention or returning to the starting point for repositioning. However, in this embodiment, autonomous map recovery and map updating can be achieved through environmental exploration and semantic understanding. The technical implementation is as follows: 1. Image loss detection and status determination Determine location status by fusing multi-source information: Visual relocalization failed: No map feature points could be matched for 10 consecutive frames; IMU integral drift: The cumulative pose error exceeds a threshold, such as 2 meters; GPS signal loss: No valid GPS data for 5 consecutive seconds; Multi-sensor consistency verification: the pose estimation difference between LiDAR, vision, and IMU exceeds 0.5 meters; When the above conditions are met, the system triggers the "image loss recovery mode" and starts the exploration and reconstruction process.

[0079] 2. Environmental exploration and feature collection

[0080] The inspection robot performs a 360-degree rotating scan, collecting environmental information through multimodal sensors. Visual feature extraction: Using the ORB-SLAM3 feature extractor, FAST corner points and ORB (OrientedFAST and Rotated BRIEF) descriptors are extracted to construct a local feature map; Semantic recognition: Identify iconic objects in the environment, such as buildings, trees, streetlights, and signs, using the SegFormer target semantic segmentation model and assign them stability weights; Laser point cloud acquisition: Construct a local point cloud map and extract geometric features such as planes and corners.

[0081] 3. Semantic-based exploration strategies

[0082] The system employs an improved exploration algorithm, prioritizing regions with high stability weights: First priority: building exterior walls (weight = 0.95), fixed facilities (weight = 0.8); Second priority: Trees (weight = 0.6), signs (weight = 0.5); Third priority: Open areas, find matches through visual loop closure detection; The exploration path employs a spiral or grid-based search to ensure maximum coverage.

[0083] 4. Multimodal data fusion and relocation

[0084] The collected environmental data is matched using a multimodal fusion algorithm: Vision-laser joint relocalization: The visual feature points are registered with the laser point cloud using the PnP (Perspective-n-Point) algorithm to solve the pose of the inspection robot; Semantic-assisted matching: Using semantic information, such as "the building is in the northeast direction", to constrain the search space; Graph optimization backend: Construct a factor graph and integrate constraints such as IMU pre-integration, visual relocalization, and laser odometry for global optimization.

[0085] 5. Map update and loop closure detection

[0086] Once the inspection robot successfully repositions itself, the system initiates a map update mechanism: Incremental map update: newly collected environmental information is integrated into the global map, and voxel filtering is used to remove redundant points; Semantic map update: Update semantic labels and stability weights, such as discovering new iconic objects and adding them to the knowledge base; Loop closure detection: Visual loop closure detection is performed using the DBoW2 bag-of-words model to correct accumulated errors.

[0087] Next, we will explain image loss recovery and map updates using specific scenarios: Scenario 1: Lost images of park roads When the inspection robot was patrolling the park roads, its positioning drifted due to GPS signal loss and sparse visual features. The system triggered the lost image recovery mode, and the inspection robot rotated 360 degrees to scan the environment, identifying a distant office building (semantic label "office building", stability weight 0.95) and a nearby street lamp (semantic label "street lamp", weight 0.7). According to the park map information stored in the knowledge base, the office building is located on the north side of the park, and the street lamps are distributed along the road. The system planned an exploration path and moved towards the office building along the road, continuously collecting visual features and laser point clouds along the way. When approaching the office building, it matched feature points in the historical map through visual loop closure detection, successfully relocalized, and corrected the pose error by approximately 1.2 meters.

[0088] Scenario 2: Changes in the parking lot environment

[0089] When the inspection robot is performing tasks in the parking lot, the drastic changes in the environment caused by vehicle movement render the traditional SLAM system ineffective. The system activates exploration mode, identifying landmarks in the parking lot, such as parking space numbers and entrance / exit signs. Through semantic matching, the system determines that its current location is in parking lot area B. Based on the parking lot layout in the knowledge base, it plans an exploration path to the entrance / exit signs (highly stable objects), collecting multi-angle images and point cloud data. Through multimodal fusion relocalization, the system successfully recovers its localization and updates the vehicle location information on the map.

[0090] Scenario 3: Self-correction of matching errors

[0091] During the exploration process, the inspection robot may make matching errors, such as mistaking building A for building B. As the robot continues to move, it continuously perceives the environment and detects inconsistencies between its current pose and the map, such as mismatched road directions or building dimensions. The system triggers a pose hypothesis test, generating multiple candidate poses. Through multi-frame data consistency verification, the optimal pose is selected. Simultaneously, the system lowers the confidence level of the current pose and re-explores the area, ultimately finding the correct matching location.

[0092] Understandably, the technical advantages of map loss recovery and map updating in the above steps are as follows: (1) Multimodal fusion relocalization: Combining information from multiple sources such as vision, laser, semantics, and IMU improves the robustness and accuracy of relocalization; (2) Semantic-guided exploration: Using semantic information to guide the exploration direction, avoiding blind searching and improving efficiency; (3) Error tolerance and self-correction: Through multiple hypothesis testing and continuous verification, it can detect and correct matching errors; (4) Incremental map update: Supports online map updates, adapts to environmental changes, and enables continuous learning.

[0093] As can be seen from the above, the inspection robot path planning method proposed in this embodiment can handle various map loss scenarios such as GPS signal loss, sparse visual features, and drastic environmental changes. Compared with traditional methods, the environmental adaptability is improved by 60%, and map recovery and map updates can be completed without manual intervention, significantly reducing operation and maintenance costs. At the same time, the system continuously updates the map and knowledge base during the exploration process, and the positioning recovery speed and success rate continue to improve with the increase of usage time.

[0094] Accordingly, see Figure 3 As shown in the figure, this application embodiment also provides a path planning device for an inspection robot, which may include: The target instruction acquisition module 11 is used to acquire the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction; The query vector generation module 12 is used to generate a corresponding query vector based on the task to be executed, and to query the corresponding environmental entity information from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, the stability weights being used to represent the immobility of the objects; The path planning module 13 is used to plan the path of the inspection robot based on the environmental entity information using a preset path planner, so as to generate corresponding path instructions and control the inspection robot to execute the task to be performed according to the path instructions.

[0095] In some specific embodiments, the inspection robot path planning device may further include: An object determination module is used to acquire image data and laser point cloud data of the target scene, and determine the objects in the target scene based on the image data and the laser point cloud data; The labeled image determination module is used to label the category and stability weight of the object in the image data to obtain the corresponding labeled image; The target semantic segmentation model determination module is used to train and fine-tune the initial semantic segmentation model using the labeled image to obtain the corresponding target semantic segmentation model; the initial semantic segmentation model is a model based on the Transformer architecture.

[0096] In some specific embodiments, the labeled image determination module may include: The first stability weight determination unit is used to match the stability weight of the object from a preset stability weight setting rule library based on the category corresponding to the object; the preset stability weight setting rule library is used to store the stability weights corresponding to the categories of the object. The second stability weight determination unit is used to acquire the historical trajectory of the object, determine the historical movement frequency and historical movement amplitude of the object based on the historical trajectory, and generate the stability weight of the object based on the historical movement frequency and historical movement amplitude using a preset time series prediction model.

[0097] In some specific embodiments, the inspection robot path planning device may further include: The keyframe image acquisition module is used to use the inspection robot to perform SLAM mapping on the target scene to generate a point cloud map corresponding to the target scene, and to acquire keyframe images generated during the SLAM mapping process. The text description determination module is used to perform semantic segmentation on the keyframe image using the target semantic segmentation model, and associate the semantic segmentation result with the point cloud map to determine the text description of the object in the point cloud map based on the association result; the text description includes the object's category, stability weight, and geometric position information; The first vector storage module is used to construct the semantic environment model of the target scene based on the textual description of the object, encode the textual description of the object in the semantic environment model into a first vector through a preset sentence embedding model, and store the first vector in the target vector knowledge base.

[0098] In some specific embodiments, the target instruction acquisition module 11 may include: The target instruction parsing unit is used to parse the target instruction using natural language processing technology, determine the user intent and motion endpoint information based on the parsing result, and determine the task to be executed corresponding to the target instruction based on the user intent and the motion endpoint information. Accordingly, the query vector generation module 12 may include: An environmental entity information determination unit is used to determine, from each of the first vectors stored in the target vector knowledge base, a target vector whose first similarity to the query vector exceeds a first preset similarity threshold, and to determine the environmental entity information based on the text description corresponding to the target vector; the first similarity is the cosine similarity, Euclidean distance, or inner product between the query vector and the first vector.

[0099] In some specific embodiments, the path planning module 13 may include: The path planning unit is used to perform path planning for the inspection robot based on the preset path planner, the first path planning algorithm, the second path planning algorithm, and the environmental entity information, so as to generate the path instructions. The objective functions of both the first and second path planning algorithms incorporate the stability weight constraint term to control the inspection robot to bypass a first object with a stability weight higher than a preset threshold, pass through a second object with a stability weight not higher than the preset threshold, or wait for the second object to be removed before re-planning the path. The first path planning algorithm is an A* algorithm or a D Lite algorithm, and the second path planning algorithm is a TEB algorithm, a dynamic window method, or a model predictive control algorithm.

[0100] In some specific embodiments, the path planning module 13 may include: The real-time data acquisition unit is used to acquire real-time data in the target scene through the multimodal sensors of the inspection robot; the real-time data includes environmental images acquired in real time by the camera, environmental point clouds acquired in real time by the lidar, and speed data of the inspection robot acquired in real time by the inertial measurement unit. An object recognition unit is used to identify dynamic objects in the target scene based on the real-time data using a target detection model, and to identify static objects in the target scene based on the real-time data using a preset semantic segmentation network. A real-time object determination unit is used to determine real-time objects in the target scene based on the dynamic objects and the static objects, and to determine the stability weights of the real-time objects. The real-time perception result generation unit is used to associate the real-time object, the stability weight of the real-time object, and the environmental point cloud in the real-time data based on the ORB-SLAM3 framework, and to construct a semantic octree map based on the association result, and to generate the real-time perception result of the target scene based on the semantic octree map; the real-time perception result includes the real-time object and the stability weight of the real-time object. A similarity calculation unit is used to encode the real-time perception result into a corresponding second vector and calculate the second similarity between the second vector and each of the first vectors stored in the target vector knowledge base; The vector knowledge base update unit is used to update the target vector knowledge base based on the second vector if each of the second similarities does not exceed the second preset similarity threshold, so as to update the semantic environment model of the target scene, so as to perform new path planning for the inspection robot based on the updated semantic environment model.

[0101] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the inspection robot path planning method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the inspection robot path planning method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0105] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned path planning method for an inspection robot. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0107] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A path planning method for an inspection robot, characterized in that, include: Obtain the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction; Based on the task to be executed, a corresponding query vector is generated, and the corresponding environmental entity information is queried from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, and the stability weights are used to represent the immobility of the objects; Using a preset path planner, the inspection robot's path is planned based on the environmental entity information to generate corresponding path instructions, and the inspection robot is controlled to execute the task to be performed according to the path instructions.

2. The inspection robot path planning method according to claim 1, characterized in that, Also includes: The system acquires image data and laser point cloud data of the target scene, and determines the objects in the target scene based on the image data and the laser point cloud data. Label the category and stability weight of the object in the image data to obtain the corresponding labeled image; The initial semantic segmentation model is trained and fine-tuned using the labeled images to obtain the corresponding target semantic segmentation model; the initial semantic segmentation model is a model based on the Transformer architecture.

3. The inspection robot path planning method according to claim 2, characterized in that, The process of labeling the category and stability weight of the object in the image data includes: Based on the category corresponding to the object, the stability weight of the object is matched from a preset stability weight setting rule library; the preset stability weight setting rule library is used to store the stability weights corresponding to the categories of the object; Alternatively, the historical trajectory of the object can be obtained, and the historical movement frequency and historical movement amplitude of the object can be determined based on the historical trajectory. Then, the stability weight of the object can be generated based on the historical movement frequency and historical movement amplitude using a preset time series prediction model.

4. The inspection robot path planning method according to claim 2, characterized in that, Also includes: The inspection robot is used to perform SLAM mapping on the target scene to generate a point cloud map corresponding to the target scene, and key frame images generated during the SLAM mapping process are obtained. The target semantic segmentation model is used to perform semantic segmentation on the keyframe image, and the semantic segmentation result is associated with the point cloud map to determine the text description of the object in the point cloud map based on the association result; the text description includes the object's category, stability weight, and geometric position information; The semantic environment model of the target scene is constructed based on the textual description of the object, and the textual description of the object in the semantic environment model is encoded into a first vector through a preset sentence embedding model, and the first vector is stored in the target vector knowledge base.

5. The inspection robot path planning method according to claim 4, characterized in that, The step of parsing the target instruction to determine the task to be executed corresponding to the target instruction includes: The target instruction is parsed using natural language processing technology, and the user intent and motion endpoint information are determined based on the parsing results. The task to be executed corresponding to the target instruction is then determined based on the user intent and the motion endpoint information. Accordingly, the step of querying the corresponding environmental entity information from the target vector knowledge base based on the query vector includes: From the first vectors stored in the target vector knowledge base, a target vector whose first similarity to the query vector exceeds a first preset similarity threshold is determined, and the environmental entity information is determined based on the text description corresponding to the target vector; the first similarity is the cosine similarity, Euclidean distance, or inner product between the query vector and the first vector.

6. The inspection robot path planning method according to any one of claims 1 to 5, characterized in that, The process of using a preset path planner to plan the path of the inspection robot based on the environmental entity information to generate corresponding path instructions includes: Using the preset path planner, the inspection robot's path is planned according to the first path planning algorithm, the second path planning algorithm, and the environmental entity information, so as to generate the path instructions; The objective functions of both the first and second path planning algorithms incorporate the stability weight constraint term to control the inspection robot to bypass a first object with a stability weight higher than a preset threshold, pass through a second object with a stability weight not higher than the preset threshold, or wait for the second object to be removed before re-planning the path. The first path planning algorithm is the A* algorithm or the D Lite algorithm, and the second path planning algorithm is the TEB algorithm, the dynamic window method, or the model predictive control algorithm.

7. The inspection robot path planning method according to claim 4, characterized in that, The process of controlling the inspection robot to execute the task to be performed according to the path instructions also includes: The inspection robot collects real-time data from the target scene using its multimodal sensors. The real-time data includes environmental images collected in real-time by a camera, environmental point clouds collected in real-time by a lidar, and speed data of the inspection robot collected in real-time by an inertial measurement unit. The object detection model is used to identify dynamic objects in the target scene based on the real-time data, and a preset semantic segmentation network is used to identify static objects in the target scene based on the real-time data. Based on the dynamic objects and the static objects, determine the real-time objects in the target scene, and determine the stability weights of the real-time objects; Based on the ORB-SLAM3 framework, the real-time object, the stability weight of the real-time object, and the environmental point cloud in the real-time data are associated. A semantic octree map is constructed based on the association results, and a real-time perception result of the target scene is generated based on the semantic octree map. The real-time perception result includes the real-time object and the stability weight of the real-time object. The real-time perception result is encoded into a corresponding second vector, and the second similarity between the second vector and each of the first vectors stored in the target vector knowledge base is calculated. If none of the second similarities exceed the second preset similarity threshold, the target vector knowledge base is updated based on the second vector to update the semantic environment model of the target scene, so as to perform new path planning for the inspection robot based on the updated semantic environment model.

8. A path planning device for an inspection robot, characterized in that, include: The target instruction acquisition module is used to acquire the target instruction issued by the user and parse the target instruction to determine the task to be executed corresponding to the target instruction; The query vector generation module is used to generate a corresponding query vector based on the task to be executed, and to query the corresponding environmental entity information from the target vector knowledge base according to the query vector; the target vector knowledge base stores the semantic environment model of the target scene corresponding to the target instruction, and the environmental entity information includes objects in the target scene and the stability weights of the objects, the stability weights being used to represent the immobility of the objects; The path planning module is used to plan the path of the inspection robot based on the environmental entity information using a preset path planner, so as to generate corresponding path instructions and control the inspection robot to execute the task to be performed according to the path instructions.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the inspection robot path planning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the inspection robot path planning method as described in any one of claims 1 to 7.

Images