Intelligent robot man-machine collaborative remote operation and maintenance method and system

Abstract

The application relates to the technical field of intelligent robots, in particular to an intelligent robot man-machine cooperation remote operation and maintenance method and system; the method comprises the following steps: acquiring environment data in an intelligent robot operation process, wherein the environment data comprises two-dimensional image frames output by a visual sensor and three-dimensional point clouds output by a laser radar sensor; extracting an abnormal feature quantity from the environment data and comparing and identifying an abnormal scene with a pre-established judgment condition; in response to the identification of the abnormal scene, an artificial takeover request is sent; otherwise, the intelligent robot is autonomously operated. The application can improve the operation stability of the intelligent robot and reduce the artificial management pressure.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to an intelligent robot human-machine collaborative remote operation and maintenance method and system. Background Technology

[0002] Intelligent robots are intelligent devices that integrate environmental perception, motion control, task execution, and human-computer interaction capabilities. They utilize visual sensors, LiDAR, inertial measurement units, and artificial intelligence algorithms to perceive and understand the external environment, and combine with mechanical actuators to perform tasks such as welcoming and guiding customers, logistics delivery, security patrols, and environmental maintenance. In recent years, with the development of artificial intelligence, robot control technology, and wireless communication technology, intelligent robots have been increasingly applied in complex public scenarios such as commercial complexes, smart parks, airports, and exhibition halls to improve operational efficiency and service levels.

[0003] In related technologies, intelligent robots typically operate autonomously, using pre-trained algorithm models and preset task flows to complete path planning, obstacle avoidance, human-robot interaction, and task execution. However, in open environments such as commercial complexes, due to frequent personnel movement, significant dynamic changes in the environment, and a variety of unexpected events, robots often face complex situations during actual operation. For example, when encountering congested passageways, temporary obstacles, special user interaction needs, or equipment malfunctions in densely populated areas, robots cannot effectively handle these situations solely through autonomous decision-making mechanisms.

[0004] To ensure the continuous and stable operation of robots, staff are sometimes assigned to be on-site to provide manual intervention when robots malfunction. However, this approach requires a significant number of maintenance personnel, which not only increases operating costs but also raises the workload of manual management as the number of robots deployed increases, making it difficult to meet the maintenance needs of large-scale robot clusters. Summary of the Invention

[0005] To alleviate the burden of manual management in large-scale robot cluster scenarios, this application provides a method and system for remote operation and maintenance of intelligent robots in human-machine collaboration.

[0006] Firstly, this application provides a method for remote operation and maintenance of intelligent robots through human-machine collaboration, employing the following technical solution: A method for remote operation and maintenance of a cloud server using intelligent robots in a human-machine collaborative manner is characterized by comprising: The environmental data acquired during the operation of the intelligent robot includes two-dimensional image frames output by a visual sensor and three-dimensional point clouds output by a lidar sensor. The system extracts abnormal features from environmental data and compares them with pre-established judgment conditions to identify abnormal scenarios. In response to the identification of an abnormal scenario, it sends abnormal scenario information to the backend monitoring terminal, which then issues a manual takeover request.

[0007] In open public settings with frequent personnel flow and significant environmental changes, such as commercial complexes, smart parks, airports, and exhibition halls, this application aggregates two-dimensional image frames output from visual sensors and three-dimensional point clouds output from LiDAR to a cloud server. These are compared with pre-built criteria, and a manual takeover request is issued when an abnormal scenario is confirmed. This transforms the previously scattered multi-robot monitoring into centralized, on-demand remote operation and maintenance via a backend monitoring terminal, allowing a single monitoring personnel to simultaneously manage a large number of robots. Compared to related technologies that rely on autonomous robot operation and on-site maintenance by personnel, the method in this application reduces maintenance labor costs while ensuring continuous and stable service for large-scale robot clusters in complex environments.

[0008] Optionally, the abnormal scenarios corresponding to the abnormal feature quantities include at least one of long-term stagnation, path deviation, obstacle avoidance failure, and interaction response timeout; wherein the long-term stagnation refers to the robot's cumulative displacement being less than a displacement threshold within a continuous time window, and the obstacle avoidance failure refers to the failure to pass the same obstacle after more than the number of consecutive obstacle avoidance attempts exceeding the number threshold.

[0009] Abnormal scenarios are subdivided into quantifiable types such as prolonged stagnation, path deviation, obstacle avoidance failure, and interaction response timeout, and clear physical criteria are given for each type. In actual operation and maintenance, operation and maintenance personnel can quickly locate the root cause of the anomaly based on this, and each threshold can be flexibly adjusted according to the on-site security level, which improves the accuracy and adaptability of anomaly identification in different scenarios.

[0010] Optionally, the abnormal scenario also includes high-level anomalies, which are determined as follows: when the robot enters a preset high-risk area, if the value of the working area in front of the robot in the congestion heat map representing the density of people exceeds the density threshold, and the robot fails to break through the obstacle after multiple attempts to avoid it, it is determined to be a high-level anomaly.

[0011] For high-risk nodes such as elevator entrances, corridor corners, and densely populated areas, a high-level anomaly detection system is introduced. The system combines the value of the congestion heat map, which represents the density of people, exceeding the density threshold with the inability to break through after multiple attempts to avoid obstacles. This allows the system to upgrade the warning when the robot is indeed trapped in a high-risk situation of crowd congestion or corridor blockage, so that subsequent personnel can allocate human resources according to the risk level.

[0012] Optionally, the population density is the ratio of the number of people in the area to the area of ​​the area.

[0013] Optionally, the steps for obtaining the number of people in the area include: Human detection is performed on the two-dimensional image frame to obtain the human detection box and the image side confidence of the target; Clustering of 3D point clouds yields candidate target clusters, and noise points are removed. Geometric features of height, lateral width, and number of points for each candidate target cluster are extracted, and radar side confidence is calculated based on their consistency with the prior human body interval. Each candidate target cluster is projected onto the image plane according to pre-calibrated extrinsic parameters. When the overlap between its projection frame and the human body detection frame exceeds a set threshold, it is determined to be the same person. For the same person, the confidence level is obtained by fusing the image-side confidence level and the radar-side confidence level. When the confidence level is greater than or equal to the preset human body confirmation threshold, the person is confirmed as a valid human body. The number of valid human bodies in the region is obtained by accumulating the number of people in the region.

[0014] To address the shortcomings of single-vision detection, which is prone to missed detections and false detections under varying lighting conditions, and the difficulty of single-lidar detection in distinguishing non-human targets such as pillars, this paper proposes a method that performs human detection on 2D image frames to obtain image-side confidence. Simultaneously, the paper performs DBSCAN clustering on 3D point clouds to remove noise and calculates radar-side confidence based on the degree of agreement between height, horizontal width, point count, and prior human body intervals. Then, cross-pairing is performed by extrinsic parameter projection and detection box overlap, and the confidence scores from both sides are fused. False targets are then screened out using a human body confirmation threshold before accumulating valid human bodies. This dual verification significantly reduces the probability of miscounting non-human objects such as pillars and luggage in the number of people, improving the accuracy and robustness of people counting in dense crowds and complex backgrounds.

[0015] Optionally, the population density of each area is backfilled into each of its constituent grids to obtain a congestion heat map, and the congestion heat map is cumulatively updated grid by grid using an exponentially weighted moving average with time decay.

[0016] After the pedestrian density of each area is backfilled into the contained 0.5m × 0.5m grid, it is updated grid by grid using an exponentially weighted moving average with time decay, so that the congestion heat map can cover the entire area and evolve smoothly over time. In actual operation, this mechanism suppresses density fluctuations caused by people moving back and forth, making the congestion situation on which the robot makes decisions more stable and continuous.

[0017] Optionally, it also includes: dynamically and globally optimizing the intelligent robot's route based on the congestion heat map.

[0018] Optionally, the A* heuristic search algorithm is used to dynamically and globally optimize the intelligent robot's route; its cost function includes: path length cost, congestion cost, and safety distance violation cost.

[0019] The A* heuristic search algorithm is adopted and the cost function is constructed as a weighted combination of path length cost, congestion cost and safety distance violation cost, so that the planning result can take into account path length, congestion avoidance and safety.

[0020] Secondly, this application provides a remote operation and maintenance method for intelligent robot human-machine collaboration, used in an intelligent robot terminal. The method is characterized in that, in the autonomous operation mode of the intelligent robot, the edge computing module in the intelligent robot identifies the human actions in the collected two-dimensional image frames, analyzes the user's interaction intent through preset action template matching and feature comparison, and provides feedback.

[0021] A lightweight visual recognition model is deployed in the edge computing module of the intelligent robot terminal to extract user key points, gesture contours and action trajectories in real time from two-dimensional image frames. By matching preset action templates and comparing features, the user's interaction intent is analyzed locally and feedback is given directly. Compared with the centralized decision-making solution of cloud server, the interaction latency is significantly reduced and the immediate response that meets the needs of users on site is guaranteed.

[0022] Thirdly, this application provides a cloud server, which adopts the following technical solution: A cloud server includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a remote operation and maintenance method for intelligent robot human-machine collaboration is implemented according to any one of claims 1-8.

[0023] The aforementioned intelligent robot human-machine collaborative remote operation and maintenance method generates a computer program, which is stored in a memory so that it can be loaded and executed by a processor. Thus, a system can be built based on the memory and processor for convenient use.

[0024] This application has the following technical effects: Based on environmental data, the system determines whether the intelligent robot is in an abnormal environment. If so, it requests manual intervention and issues control commands. By centrally monitoring multiple intelligent robots in the backend and taking over control in abnormal situations, the system ensures stable operation of the intelligent robots during actual work. Simultaneously, centralized monitoring of the intelligent robot cluster reduces labor costs. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a method for remote operation and maintenance of an intelligent robot in human-machine collaboration, as disclosed in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of intelligent robot interaction in an intelligent robot human-machine collaborative remote operation and maintenance method disclosed in an embodiment of this application. Detailed Implementation

[0027] In a first aspect, embodiments of this application disclose a remote operation and maintenance method for intelligent robot human-machine collaboration, used on a cloud server.

[0028] Reference Figure 1 A method for remote operation and maintenance of intelligent robots in human-machine collaboration includes steps S1-S2.

[0029] S1: Acquire environmental data during the operation of the intelligent robot.

[0030] To more clearly illustrate this technical solution, the application scenario of this application is briefly described: This embodiment takes an intelligent business environment as the application scenario, including a front-end intelligent robot, a sensor data acquisition unit, a cloud server, a back-end monitoring terminal, etc., and the units interact with each other in real time through a high-speed communication network.

[0031] The sensor data acquisition unit includes visual sensors and LiDAR sensors deployed on the intelligent robot itself. It's understood that intelligent robots are used to serve customers at the front end, and multiple units are typically deployed simultaneously in one area. The visual and LiDAR sensors on the intelligent robot are primarily used to collect information from the surrounding environment, such as images and sounds. The back-end monitoring terminal is mainly used to receive information collected by the intelligent robot and analyzed by a cloud server for staff reference, and can also issue commands to the front-end intelligent robot.

[0032] In actual operation, each intelligent robot performs its own task. For example, shopping guide robots greet customers and guide them at the mall entrance and on each floor, delivery robots complete deliveries between merchants and consumers, and patrol robots perform security patrols in public areas.

[0033] To ensure the timeliness and continuity of environmental data acquisition, the sensor data acquisition unit samples at a sampling frequency of 10Hz. The information collected in this embodiment mainly includes: real-time output of two-dimensional image frames containing user gestures, postures and motion trajectories from the vision sensor; real-time output of three-dimensional point clouds reflecting obstacle positions and scene congestion from the lidar sensor; real-time output of the robot's own three-axis posture angle scalar from the posture sensor; and real-time output of interactive contact pressure scalar from the tactile sensor. The above four types of data are synchronously transmitted to the cloud server via the 5G-A network.

[0034] S2: Extract abnormal features from environmental data and compare them with pre-established judgment conditions to identify abnormal scenarios. In response to the identification of an abnormal scenario, send abnormal scenario information to the background monitoring terminal, and the background monitoring terminal issues a manual takeover request.

[0035] In high-risk areas such as elevator entrances, corridor corners, and densely populated areas, robots may still encounter unforeseen situations that they cannot handle on their own. If these situations are not identified and warned of in a timely manner, they will degenerate into prolonged shutdowns or even service interruptions, as is the case with current technology. Therefore, this system identifies abnormal scenarios based on environmental data during robot operation and requests manual intervention when an abnormal scenario is detected.

[0036] On the one hand, monitoring personnel can obtain real-time operational data of the intelligent robot through the back-end monitoring terminal for visualization. The operational data can include the intelligent robot's precise positioning coordinates, remaining battery power, real-time posture, action execution status, raw sensor data and data parsing results, forming a full-state view of the robot's operation.

[0037] On the other hand, real-time monitoring of environmental data during robot operation is implemented as follows: Abnormal features are extracted from the environmental data during robot operation, and each feature is compared with pre-established judgment conditions.

[0038] This embodiment provides the following quantifiable judgment criteria for various operational anomalies: Long-term stagnation refers to the robot's walking motor execution module being enabled but the actual displacement being less than the threshold within a continuous time window. In this embodiment, the cumulative displacement is less than 0.2 meters within 8 consecutive seconds.

[0039] Path deviation refers to the lateral deviation between the robot's actual position and the planned path exceeding a threshold, which in this embodiment is taken as exceeding 0.5 meters continuously.

[0040] Obstacle avoidance failure refers to failing to pass through an obstacle after more than a threshold number of obstacle avoidance attempts. In this embodiment, it is defined as three consecutive failed obstacle avoidance attempts. When the robot fails to pass through at the current position, it re-executes path search to generate a path around the current obstacle and travels along the updated route. When the cumulative number of re-executions of path search reaches a set threshold, if the robot's cumulative forward displacement at the current position is still less than the passage determination threshold, it is confirmed as obstacle avoidance failure.

[0041] Interaction response timeout refers to the failure to receive execution confirmation or user response within a set time limit after the interaction action command is issued; in this embodiment, it is set to exceed 2 seconds. The above-mentioned abnormal feature model can be set directly by a threshold, or it can be improved by training a classifier to distinguish between abnormal and normal states after accumulating sufficient labeled samples, thereby improving the accuracy of identification in complex scenarios.

[0042] Based on the above instantaneous criteria, the robot's running state sequence is statistically analyzed within a sliding time window (in this embodiment, the most recent 10 seconds are taken, and 100 frames are sampled at 10Hz). The abnormality is only confirmed when a certain abnormal feature appears more than the set criteria in a time window, so as to suppress misjudgment caused by instantaneous jitter.

[0043] In some embodiments, dynamic environmental changes in high-risk areas (elevator entrances, corridor corners, densely populated areas) can be preset in real time. When the robot enters such an area, if it detects dense crowds ahead (i.e., the value of the congestion heatmap of the grid in front of the robot's working area) for 3 consecutive seconds (this value can be adjusted within the range of 2 to 6 seconds; the shorter the threshold, the earlier the intervention but the more false alarms; the longer the threshold, the fewer false alarms but the relatively delayed intervention), it will be detected that the robot is in a congested area. Exceeding the density threshold A high-level anomaly is identified when a robot fails to overcome obstacles in scenarios such as obstacle blockage (where a LiDAR echo obstacle persists within a set distance (e.g., 1.5 meters) directly in front of the robot, and the walking motor is enabled but the actual forward displacement is less than the preset distance), and the robot is unable to do so after multiple autonomous obstacle avoidance attempts. Autonomous obstacle avoidance refers to the robot re-executing a path search to generate a path around the current obstacle when it fails to pass the current obstacle, and then proceeding along the updated route. When the cumulative number of re-executing path searches reaches a set threshold, and the robot's cumulative forward displacement at the current obstacle is still less than the passage threshold, the robot is deemed to have failed to pass the current obstacle, and this is confirmed as a high-level anomaly.

[0044] The steps to obtain a congestion heatmap are as follows: (1) Calculate the population density in each area.

[0045] The method for calculating pedestrian density is as follows: a single-stage object detection network (such as the YOLO series of lightweight detection models) is run on the visual image frame to perform human detection. The network directly outputs the classification confidence score of each candidate box belonging to the "human" category. Redundant boxes are removed and retained after nonmaximum suppression. For those exceeding the detection threshold, several human detection frames are obtained.

[0046] Simultaneously, the lidar point cloud is filtered according to floor height range and horizontal working range, and then clustered using the density-based DBSCAN clustering algorithm (neighborhood radius). Take 0.3 meters and the minimum number of points. Using 10 (or Euclidean distance clustering), densely connected points are grouped into candidate target clusters and noise points are removed; height is extracted for each cluster. Horizontal width With points The lidar side confidence score is calculated by multiplying the geometric features by their degree of similarity to the prior human body interval (in this embodiment, the height is 1.4–1.9 meters, the horizontal width is 0.2–0.8 meters, and the number of points is not less than 10). This ensures that clusters with geometric features as close to the human body as possible are generated. The higher the value, the more it deviates from the expected value, and the lower the value.

[0047] The formula for calculating radar-side confidence can be expressed as: In the formula, Indicates the radar-side confidence level of a certain cluster; Indicates a high degree of similarity. Indicates width fit; Indicates the degree of quantitative agreement.

[0048] All three fit parameters are calculated using interval membership functions: for features with upper and lower bounds, such as height and horizontal width, trapezoidal membership functions are used, meaning the feature values ​​fall within the prior human body interval. (The height in this embodiment is 1.4~1.9 meters) or (In this embodiment, the width is 0.2~0.8 meters.) When the fit is within a certain range, a tolerance of 1 is taken, with tolerances left on both sides of the interval. (In this embodiment, the height tolerance is 0.3 meters and the width tolerance is 0.2 meters.) Linear attenuation is applied, and values ​​outside the tolerance range are set to 0. For point count features with only a lower limit requirement, a one-sided ramp function is used, meaning the number of points below the lower limit is considered zero. (In this embodiment, 10 is used) When 0 is used, the number of saturation points is reached. (In this embodiment, 40) and above are taken as 1, with linear increments in between. Therefore... , , All The confidence level of the lidar is obtained by multiplying the reproducible values ​​within the interval. This causes any geometric feature to deviate significantly from the prior knowledge of the human body (e.g., a column exceeding its height limit). )hour That is, 0, thus eliminating non-human targets.

[0049] Then, each candidate cluster is projected onto the image plane according to the pre-calibrated extrinsic parameters. When the overlap (IoU) between its projection frame and a certain visual detection frame exceeds a set threshold (such as 0.5), it is determined to be the same person, thus completing the cross-pairing verification between LiDAR and vision. Then apply the following to each target: ; Indicates the fusion confidence of a certain target; Indicates the radar-side confidence level of the target; The image-side confidence score represents the target, i.e., the classification confidence score output by the target detection network. and These are the first weighting coefficient and the second weighting coefficient, respectively, preset based on the experience of those skilled in the art; they satisfy... .

[0050] when Greater than or equal to the human body confirmation threshold A count of 0.6 is considered a valid human body and included in the count; otherwise, it is discarded. The final count is based on the number of valid human bodies identified in that area. Divide by its actual area To obtain the population density .

[0051] It is understandable that "area" here refers to the area pre-defined in step S1 based on the mall's floor plan functions. Each area is composed of several adjacent passable grids and has independent passage functions. Examples include nodal areas such as elevator lobbies, escalator entrances, passageway intersections, and shop entrances, as well as passageway areas such as corridors divided into fixed length segments. Each area is assigned a unique number and its actual area is pre-stored offline. The density of this area is then backfilled into each of its 0.5m x 0.5m grid cells.

[0052] (2) After backfilling the pedestrian density of each area into each of the 0.5m × 0.5m grids, update the grid by grid in a time decay manner to form a congestion heat map that covers the entire field and updates smoothly over time.

[0053] Specifically, no. A grid in Congestion heatmap values ​​at any given time are updated using an exponentially weighted moving average: ,in The real-time density of the raster at that moment is the density backfilled to the raster by the density of the region it belongs to. This represents the congestion heatmap value of the grid at the previous moment, i.e., the area density of the grid at the previous moment. The time decay coefficient (0.7 in this embodiment) is used to suppress density fluctuations caused by momentary movement of people. The larger the value, the smoother the heatmap and the slower the response to abrupt changes.

[0054] It is understood that the pedestrian density in each area in this embodiment is obtained by spatial collaborative sampling and splicing by intelligent robots distributed in different spatial locations.

[0055] In other embodiments, the pedestrian density in each area can also be obtained from a top-down view by fixed visual sensors (such as high-position passenger flow cameras, fisheye cameras) and / or lidar sensors that are pre-set at high positions such as ceilings and columns in each area of ​​the mall, and uploaded to a cloud server synchronously via a high-speed communication network. This data is then combined with local data collected by mobile robots to form a dynamic congestion heat map covering the entire mall.

[0056] Combination Figure 2Upon detecting a high-level anomaly report or an abnormal scenario, the cloud server sends the identified anomaly information to the backend monitoring terminal, which then issues a manual takeover request. At this point, monitoring personnel can initiate remote takeover through the backend monitoring terminal, switching the robot from autonomous control to manual takeover mode and sending manual instructions to the execution control unit in real time. For example, for a delivery robot stopped at an elevator entrance, monitoring personnel can view the scene in real time, adjust the robot's position using the drive motor execution module via the backend monitoring terminal, and wait outside the elevator. Once the elevator is detected as empty, the robot is controlled to enter, and if necessary, the interaction motor execution module and the grasping motor execution module are activated to perform corresponding coordinated actions, ultimately completing the delivery task. It is understood that the high-level anomaly events and abnormal scenarios here are primarily for demonstration purposes to monitoring personnel, allowing them to choose to take over autonomously.

[0057] After this step, the robot receives precise remote human intervention in unexpected situations that it cannot handle on its own. This resolves issues that would otherwise lead to prolonged downtime and service interruptions in real time, ensuring the continuity of on-site services.

[0058] After the unexpected problem was resolved, the robot was switched from manual takeover mode back to autonomous operation mode, and the robot continued to perform its tasks according to the pre-set algorithm. At the same time, all operational data from this round of manual takeover, as well as the entire process data such as congestion data and elevator adaptation data collected by sensors during the anomaly, were synchronously stored in the backend database via the 5G-A network.

[0059] Meanwhile, during the operation of the intelligent robot, the robot's route is dynamically and globally optimized based on the population density in each area.

[0060] Combining the real-time location of the intelligent robot, the task destination, and the trajectory of dynamic obstacles, path search is performed on a weighted grid map with the following cost function as the objective: ; In the formula, This represents the total cost of the path. The total cost of this path. The cumulative congestion cost for this path is obtained by mapping values ​​from the congestion heatmap. The cost of violating the safety distance for this path is accumulated (this item increases as the distance between the planned path and obstacles or crowds decreases when the distance is less than the safety threshold). , , The weights are set according to the experience of those skilled in the art. In this embodiment, they are set to 0.3, 0.5, and 0.2 respectively, so as to achieve a compromise between path length, congestion avoidance, and safety.

[0061] The path The sum of congestion costs for each grid. The congestion cost of a grid is obtained by normalizing and mapping the heatmap values ​​of that grid. For example, ( To normalize the baseline density, this embodiment uses 2 people / m².

[0062] The cost of violating the safe distance can be calculated using the following methods: The formula for calculating the safety distance violation cost for any grid cell traversed by the path can be expressed as: ; In the formula, For the first in a certain path The cost of violating the safety distance of each grid; For the first The distance (in meters) between the robot and the nearest obstacle / crowd at each grid point, measured by LiDAR point cloud. The safe distance threshold (in this embodiment, it is taken as...) (meters). Subsequently, the safety distance violation costs corresponding to each grid are accumulated to obtain the cumulative path safety distance violation cost.

[0063] The constraints include: the grids traversed must be passable grids, a safe distance must be maintained from the predicted positions of dynamic obstacle trajectories, and the net width requirement for the occupancy state of the passage must be met.

[0064] The solution employs the A* (A-star) heuristic search algorithm based on a weighted cost map, with an evaluation function... Sort the nodes to be expanded, where From the starting point to the node The actual cost (i.e., the total path cost) obtained by summing the costs of length, congestion, and safety along the searched path as described above. ), For nodes The admissibility heuristic is formed by the sum of the remaining Euclidean distance to the destination and the estimated cumulative congestion cost along the route; the algorithm maintains an open list, and selects from the list each time. The minimum node expansion is incremented progressively by cost, until the node reaches the destination node or the minimum node in the open list. When the price stops decreasing, convergence is determined and the output is backtracked. The algorithm for finding the minimum optimal path follows a standard technique in this field and will not be described in detail here. However, when the environmental situation dynamically updates (e.g., a sudden increase in congestion costs in a certain area), the cost of the current path may increase. If the replanning trigger condition is exceeded, the above search is re-executed to complete the dynamic replanning.

[0065] In this embodiment, the threshold for triggering route replanning is set at 1.2 people per square meter and the threshold for congestion duration is set at 5 seconds. These thresholds are empirical values ​​under the constraints of delivery timeliness and detour costs. In scenarios with narrow passages or higher safety requirements, the threshold for pedestrian density can be lowered to 0.8 people per square meter and the threshold for duration can be shortened to 3 seconds, allowing the robot to avoid obstacles earlier. In open areas with ample traffic capacity, the thresholds can be appropriately raised.

[0066] It should be noted that the heuristic search algorithm based on weighted grid maps used in this step can be replaced by Dijkstra's algorithm based on topological graphs. In this replacement implementation, the shopping mall passage is abstracted into a topological graph consisting of nodes (intersections, elevator entrances, merchant entrances and exits) and edges (connected passages). The weight of each edge is obtained by weighting the length of the passage segment with the real-time congestion cost, and the shortest weighted path search is used to find a passable route that avoids congestion. This algorithm is a conventional technique in this field and will not be described in detail here.

[0067] Secondly, this application discloses a method for remote operation and maintenance of intelligent robots through human-machine collaboration, which is used in intelligent robot terminals.

[0068] Understandably, when the robot is working normally, it operates in an autonomous mode, executing different commands based on the actions of the person.

[0069] Specifically, a lightweight visual recognition model (such as a YOLO series model) is deployed in the edge computing module of the intelligent robot to extract user key points, gesture contours, and motion trajectories in real time from two-dimensional image frames. The user's interaction intent is then quickly analyzed through preset action template matching and feature comparison. For example, when the shopping guide robot's visual sensor captures a user raising their hand and waving towards the robot, and this action sequence lasts for more than 2 seconds, the edge computing module immediately recognizes it as an active greeting intent. It then generates a combined control command consisting of three sub-actions: stopping movement, turning towards the user, and playing a greeting voice message. This command is then sent in real time to the execution control unit via the command issuing unit, driving the walking motor execution module to brake, driving the interaction motor execution module to turn, and completing the voice broadcast, thus completing a standardized welcoming interaction. In this embodiment, the duration of the action used to identify the intent is set to 2 seconds. This value is used to balance response sensitivity and false trigger suppression. In scenarios with a fast interaction pace, it can be shortened to 1 second, and in noisy or frequently moving scenarios, it can be extended to 3 seconds. This adjustment does not change the processing path from action recognition to command generation in this step.

[0070] In this step, the key point and gesture extraction based on the lightweight visual recognition model is only one specific method to achieve human action intention recognition. Those skilled in the art will understand that any technical means capable of obtaining user interaction intentions from human action data output by visual and tactile sensors and generating corresponding control commands can be applied to this step without departing from the technical concept of this application. After this step, the originally unstructured human action data is transformed into structured basic control commands that the robot can directly execute, enabling the response to interactive actions to be completed at the edge, significantly reducing interaction latency and ensuring the robot's immediate response to the needs of users on-site.

[0071] Thirdly, embodiments of this application also disclose a cloud server, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a remote operation and maintenance method for intelligent robot human-machine collaboration according to this application is implemented.

[0072] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0073] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for remote operation and maintenance of intelligent robots in human-machine collaboration, used on a cloud server, characterized in that, include: The environmental data acquired during the operation of the intelligent robot includes two-dimensional image frames output by a visual sensor and three-dimensional point clouds output by a lidar sensor. The system extracts abnormal features from environmental data and compares them with pre-established judgment conditions to identify abnormal scenarios. In response to the identification of an abnormal scenario, it sends abnormal scenario information to the backend monitoring terminal, which then issues a manual takeover request.

2. The intelligent robot human-machine collaborative remote operation and maintenance method according to claim 1, characterized in that, The abnormal scenarios corresponding to the abnormal feature quantities include at least one of long-term stagnation, path deviation, obstacle avoidance failure, and interaction response timeout; wherein the long-term stagnation refers to the robot's cumulative displacement being less than the displacement threshold within a continuous time window, and the obstacle avoidance failure refers to the failure to pass the threshold number of consecutive obstacle avoidance attempts against the same obstacle.

3. The intelligent robot human-machine collaborative remote operation and maintenance method according to claim 2, characterized in that, Abnormal scenarios also include high-level anomalies, which are determined by: obtaining a congestion heat map representing the density of people; when the robot enters a preset high-risk area, if the value of the working area in front of the robot in the congestion heat map exceeds the density threshold, and the robot fails to break through after multiple attempts to avoid obstacles, it is determined to be a high-level anomaly.

4. The intelligent robot human-machine collaborative remote operation and maintenance method according to claim 3, characterized in that, Pedestrian density is the ratio of the number of people in a region to the area of ​​that region.

5. A method for remote operation and maintenance of intelligent robots in human-machine collaboration according to claim 4, characterized in that, The steps to obtain the number of people in the area include: Human detection is performed on the two-dimensional image frame to obtain the human detection box and the image side confidence of the target; Clustering of 3D point clouds yields candidate target clusters, and noise points are removed. Geometric features of height, lateral width, and number of points for each candidate target cluster are extracted, and radar side confidence is calculated based on their consistency with the prior human body interval. Each candidate target cluster is projected onto the image plane according to pre-calibrated extrinsic parameters. When the overlap between its projection frame and the human body detection frame exceeds a set threshold, it is determined to be the same person. For the same person, the confidence level is obtained by fusing the image-side confidence level and the radar-side confidence level. When the confidence level is greater than or equal to the preset human body confirmation threshold, the person is confirmed as a valid human body. The number of valid human bodies in the region is obtained by accumulating the number of people in the region.

6. A method for remote operation and maintenance of intelligent robots in human-machine collaboration according to claim 3, characterized in that, The congestion heatmap is obtained by backfilling the pedestrian density of each area into each grid cell, and then updating the congestion heatmap cell by cell using an exponentially weighted moving average with time decay.

7. A method for remote operation and maintenance of intelligent robots in human-machine collaboration according to claim 3, characterized in that, Also includes: The intelligent robot's route is dynamically and globally optimized based on the congestion heat map.

8. A method for remote operation and maintenance of intelligent robots in human-machine collaboration according to claim 7, characterized in that, The A* heuristic search algorithm is used to dynamically and globally optimize the intelligent robot's path. Its cost function includes: path length cost, congestion cost, and safe distance violation cost.

9. A method for remote operation and maintenance of intelligent robots through human-machine collaboration, used in intelligent robot terminals, characterized in that, In the autonomous operation mode of the intelligent robot, the edge computing module in the intelligent robot identifies the human actions in the collected two-dimensional image frames, analyzes the user's interaction intent through preset action template matching and feature comparison, and provides feedback.

10. A cloud server, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions, which, when executed by the processor, implement a remote operation and maintenance method for intelligent robots in human-machine collaboration according to any one of claims 1-8.

Images