Sensor-based human-machine collaborative monitoring internet of things system, method and medium

Abstract

The application provides a kind of sensor-based man-machine collaborative monitoring internet of things system, method and medium, it is related to man-machine collaborative safety technical field.The method comprises: obtaining the monitoring data of the working area of collaborative robot;Determine the collision risk value based on the monitoring data, the first motion trajectory of operator and the second motion trajectory of collaborative robot;Determine the avoidance parameter based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency and the moving space range of collaborative robot;Generate motion control instruction based on avoidance parameter, and send to the motion control system of collaborative robot;Motion control system is based on motion control instruction, adjusts physical driving signal to drive mechanical link to produce compound motion, so that end effector completes avoidance.The internet of things system includes collaborative supervision management platform, the computer instruction stored by computer readable storage medium is read and run after running.The application can improve the safety and efficiency of man-machine collaboration.

Description

Technical Field

[0001] This invention relates to the field of human-machine collaborative safety technology, and in particular to a sensor-based human-machine collaborative monitoring Internet of Things system, method and medium. Background Technology

[0002] Currently, collaborative robots have been widely adopted in various industrial sectors. Since collaborative robots share the same physical space with operators, monitoring human-robot collaboration is particularly important to ensure safety and operational efficiency.

[0003] Currently, human-machine collaborative monitoring systems generally suffer from problems such as limited sensing methods and poor robustness. They are prone to false alarms or missed alarms in complex industrial environments such as changes in lighting and metal reflections. Furthermore, they cannot perceive and adapt to different operational scenarios, resulting in delayed safety responses and stiff human-machine interaction, making it difficult to achieve a balance between ensuring absolute safety and achieving efficient and smooth collaborative operations.

[0004] Therefore, there is an urgent need to provide a sensor-based human-machine collaborative monitoring IoT system, method, and medium to solve the problems of inaccurate perception, lack of predictive ability, and inability to adapt to dynamic tasks in human-machine collaborative monitoring systems, so as to achieve full automation from perception and decision-making to execution, thereby significantly improving the safety and operational efficiency of human-machine collaboration. Summary of the Invention

[0005] To address the problems of inaccurate perception, lack of predictive ability, and inability to adapt to dynamic tasks in human-machine collaborative monitoring systems, this invention provides a sensor-based human-machine collaborative monitoring Internet of Things system, method, and medium.

[0006] The invention includes a sensor-based human-machine collaborative monitoring IoT system, comprising a collaborative monitoring and management platform. The collaborative monitoring and management platform is configured to: acquire monitoring data of the collaborative robot's work area; determine a collision risk value based on the monitoring data, a first motion trajectory of the operator, and a second motion trajectory of the collaborative robot; determine avoidance parameters of the collaborative robot based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the collaborative robot's movement space range; and generate motion control commands based on the avoidance parameters and send them to the collaborative robot's motion control system. The motion control system is configured to, based on the motion control commands, adjust the physical drive signals applied to the joint motors to drive the mechanical links of the collaborative robot to generate compound motion, enabling the end effector to complete avoidance.

[0007] The invention includes a sensor-based human-machine collaborative monitoring method, executed by a collaborative supervision and management platform within a sensor-based human-machine collaborative monitoring IoT system. The method includes: acquiring monitoring data of the collaborative robot's work area; determining a collision risk value based on the monitoring data, a first motion trajectory of the operator, and a second motion trajectory of the collaborative robot; determining avoidance parameters of the collaborative robot based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the collaborative robot's movement space range; and generating motion control commands based on the avoidance parameters and sending them to the collaborative robot's motion control system. The motion control system is configured to adjust the physical drive signals applied to the joint motors based on the motion control commands to drive the mechanical links of the collaborative robot to generate compound motion, enabling the end effector to complete avoidance.

[0008] The invention includes a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes the aforementioned sensor-based human-machine collaborative monitoring method.

[0009] The beneficial effects of the above invention include, but are not limited to: (1) a complete and closed-loop human-machine collaborative safety monitoring process is constructed, and a forward-looking risk assessment is achieved through multi-source perception and trajectory prediction, which effectively prevents collisions and minimizes the interruption of the work process, significantly improving the safety and overall work efficiency of human-machine collaboration; (2) through the sequence prediction model, the operator's first state data at the current moment and the preset state data for the future period are used to predict the movement trajectory of the operator in the future period, so that the system can perceive the movement of personnel in advance, provide sufficient time for predicting collisions, reduce unnecessary emergency stops of collaborative robots, and balance human-machine collaboration safety and work efficiency; (3) by combining task type and task stage to make collision avoidance decisions, the system can distinguish between real dangers and necessary collaborative interactions, avoid excessive avoidance or insufficient reaction, and achieve the optimal balance between safety and efficiency; (4) by complementing the first collision risk value and the second collision risk value, and combining empirical risk and forward-looking analysis, the comprehensiveness and reliability of collision risk assessment are improved, and the limitations of a single method are avoided. Attached Figure Description

[0010] The present invention will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same reference numerals denote the same structures, wherein:

[0011] Figure 1 This is a platform architecture diagram of a sensor-based human-machine collaborative monitoring Internet of Things system, as shown in some embodiments of this specification.

[0012] Figure 2 This is an exemplary flowchart of a sensor-based human-machine collaborative monitoring method according to some embodiments of this specification;

[0013] Figure 3 This is an exemplary schematic diagram illustrating the determination of collision risk values ​​according to some embodiments of this specification;

[0014] Figure 4 This is an exemplary schematic diagram of a sequence prediction model according to some embodiments of this specification. Detailed Implementation

[0015] The accompanying drawings used in the description of the embodiments will be briefly introduced below. The drawings do not represent all embodiments.

[0016] The terms “system,” “device,” “unit,” and / or “module” as used herein are one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0017] Unless the context clearly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0018] Figure 1 This is a platform architecture diagram of a sensor-based human-machine collaborative monitoring Internet of Things system, as shown in some embodiments of this specification.

[0019] In some embodiments, such as Figure 1 As shown, the sensor-based human-machine collaborative monitoring Internet of Things system 100 includes a collaborative monitoring user platform 110, a collaborative monitoring service platform 120, a collaborative monitoring management platform 130, a collaborative monitoring sensor network platform 140, and a collaborative monitoring perception and control platform 150.

[0020] The Collaborative Supervision User Platform 110 refers to a platform that initiates collaborative supervision requests and receives collaborative supervision feedback information. It can be configured as a user terminal, such as a computer or other device with input and / or output functions.

[0021] The Collaborative Supervision Service Platform 120 refers to an interactive service platform that receives and transmits collaborative supervision data, including servers, gateways, and routers.

[0022] In some embodiments, the collaborative supervision service platform interacts upward with the collaborative supervision user platform and downward with the collaborative supervision management platform.

[0023] The collaborative supervision management platform 130 refers to a comprehensive platform for processing and managing collaborative supervision data. In some embodiments, the collaborative supervision management platform is configured to execute a sensor-based human-machine collaborative monitoring method. For more information on this method, see [link to relevant documentation]. Figures 2-4 Related descriptions.

[0024] In some embodiments, the collaborative monitoring and management platform may include processors and / or servers, data centers, etc. The data center is equipped with storage devices.

[0025] The collaborative supervision sensor network platform 140 refers to a platform for transmitting sensor data or information related to collaborative supervision, including communication transmission networks and routing devices.

[0026] In some embodiments, the collaborative regulatory sensor network platform interacts upward with the collaborative regulatory management platform and downward with the collaborative regulatory perception and control platform.

[0027] The Collaborative Supervision Perception and Control Platform 150 refers to a platform for collaborative supervision data collection and execution of instructions.

[0028] In some embodiments, the collaborative regulatory perception and control platform 150 may include a variety of sensors. These sensors include, but are not limited to, distributed millimeter-wave radar arrays, 3D vision sensors, and torque sensors, among other different types of sensors.

[0029] In some embodiments, the collaborative regulatory perception and control platform may also include a collaborative robot.

[0030] Collaborative robots are robots that work in conjunction with an operator. In some embodiments, a collaborative robot may include multiple components such as a motion control system, joint motors, mechanical links, an end effector, and a servo driver.

[0031] A motion control system is a computing unit that receives commands and generates control signals. Examples include controllers.

[0032] A servo driver is a power electronic device that drives a joint motor. Under the drive of the servo driver, the joint motor outputs the required torque and speed, providing power to the mechanical linkage.

[0033] Mechanical linkages are rigid components that connect various joints and transmit force and motion.

[0034] An end effector is a tool or device installed at the end of the mechanical structure of a collaborative robot to directly perform a task.

[0035] In some embodiments, the motion control system is configured to adjust the physical drive signals applied to the joint motors based on motion control commands to drive the mechanical links of the collaborative robot to generate compound movements, enabling the end effector to perform an avoidance maneuver. That is, the motion control system drives the joint motors by sending physical drive signals to the servo driver, which in turn moves the mechanical links, ultimately causing the end effector to perform an avoidance maneuver. In scenarios requiring an emergency stop of the collaborative robot, the motion control system directly sends hardware commands to the servo driver to perform emergency braking. For further explanation of this section, please refer to [link to relevant documentation]. Figure 2 Related descriptions.

[0036] For more information about the above platforms, please refer to [link / reference]. Figures 2-4 And related explanations.

[0037] In some embodiments of this specification, the sensor-based human-machine collaborative monitoring Internet of Things system can form an information operation closed loop between various functional platforms, operate in a coordinated and regular manner, and achieve safe and efficient human-machine collaborative operation through real-time perception, dynamic prediction and intelligent decision-making.

[0038] Figure 2 This is an exemplary flowchart illustrating a sensor-based human-machine collaborative monitoring method according to some embodiments of this specification. Figure 2 As shown, process 200 includes the following steps. In some embodiments, process 200 may be executed by the collaborative regulatory management platform 130.

[0039] Step S210: Obtain monitoring data of the collaborative robot's work area.

[0040] For more information on collaborative robots, please see [link to relevant information]. Figure 1 Related explanations.

[0041] The work area refers to the working space of a collaborative robot.

[0042] Monitoring data refers to data that characterizes the real-time status of the operator and collaborative robot at the current moment.

[0043] An operator is a human worker who works collaboratively with a collaborative robot within the robot's work area.

[0044] In some embodiments, the monitoring data may include first state data of the operator and second state data of the collaborative robot. The first state data refers to the operator's state data at the current moment, which may include the operator's position, velocity, and acceleration at the current moment. The second state data refers to the collaborative robot's state data at the current moment, which may include the joint angles, end effector position, velocity, and acceleration at the current moment. Both the operator's position and the end effector's position can be represented by position coordinates in the world coordinate system.

[0045] For more information on end effectors, please refer to [link / reference]. Figure 1 Related explanations.

[0046] In some embodiments, the monitoring data can be multi-source data, and the collaborative supervision and management platform can acquire monitoring data of the work area through various sensors deployed in the work area.

[0047] Step S220: Based on the monitoring data, the operator's first motion trajectory, and the collaborative robot's second motion trajectory, determine the collision risk value.

[0048] The first motion trajectory refers to the operator's motion trajectory within a future time period.

[0049] A future time period refers to a period of time after the current moment. For example, 2 seconds or 5 seconds in the future. In some embodiments, the future time period can be preset manually.

[0050] In some embodiments, the first motion trajectory may be pre-planned by the user, and the collaborative monitoring and management platform can obtain the first motion trajectory through user input. The user may be an operator or other technical personnel.

[0051] Figure 4 This is an exemplary schematic diagram of a sequence prediction model according to some embodiments of this specification.

[0052] In some embodiments, such as Figure 4 As shown, the collaborative supervision and management platform can also predict the first motion trajectory 311 based on the monitoring data 341 and the operator's preset status data 420 through the sequence prediction model 430. The sequence prediction model is a machine learning model.

[0053] Sequence prediction models are models used to predict the first motion trajectory.

[0054] In some embodiments, the sequence prediction model can be a machine learning model, such as a Long Short-Term Memory (LSTM) network model.

[0055] Preset status data refers to the operator's status data for a future time period. For example, the operator's position, speed, acceleration, etc., at multiple preset future time points within a future time period. In some embodiments, preset status data can be pre-planned by the user, and the collaborative monitoring and management platform can obtain the operator's preset status data through user input.

[0056] In some embodiments, such as Figure 4 As shown, the input to the sequence prediction model 430 includes the first state data 341-1 from the monitoring data 341 and the operator's preset state data 420, and the output includes the first motion trajectory 311. That is, by inputting the operator's first state data at the current moment and the preset state data for the future time period into the sequence prediction model, the sequence prediction model can output the operator's first motion trajectory for the future time period.

[0057] In some embodiments, the collaborative regulatory management platform can train a sequence prediction model based on multiple training samples with training labels.

[0058] Training samples and training labels can be directly obtained based on historical data. A single training session may use a set of training samples that includes the historical operator's actual state data at the first historical moment and the first historical time period. The training label corresponding to this set of training samples is the historical operator's actual movement trajectory during the first historical time period. The first historical moment occurs before the first historical time period.

[0059] In some embodiments, the collaborative monitoring and management platform can input training samples into an initial sequence prediction model, construct a loss function based on the training labels and the output of the initial sequence prediction model, iteratively update the parameters of the initial sequence prediction model based on the loss function, and terminate the iteration when the iteration termination condition is met, thus obtaining the trained sequence prediction model. The iterative update method includes, but is not limited to, gradient descent, and the iteration termination condition can be the convergence of the loss function or the reaching of a threshold number of iterations.

[0060] In some embodiments of this specification, a sequence prediction model is used to predict the operator's movement trajectory in the future time period by using the operator's first state data at the current moment and preset state data for the future time period. This enables the system to sense the movement of personnel in advance, providing sufficient time for collision prediction, reducing unnecessary emergency stops for collaborative robots, and balancing human-robot collaboration safety and work efficiency.

[0061] The second motion trajectory refers to the motion trajectory of the collaborative robot in the future. In some embodiments, the collaborative monitoring and management platform can directly read the pre-calculated second motion trajectory from the motion control system of the collaborative robot.

[0062] Collision risk value is a quantitative assessment indicator that quantifies the probability of a physical collision between the collaborative robot and the operator within a future time period. In some embodiments, the collision risk value can be represented by a value between 0 and 1, with a higher value indicating a greater probability of a physical collision between the collaborative robot and the operator.

[0063] In some embodiments, the collaborative monitoring and management platform can determine the collision risk value in multiple ways based on monitoring data, the operator's first motion trajectory, and the collaborative robot's second motion trajectory.

[0064] For example, the collaborative supervision and management platform can construct a first target vector based on monitoring data, the operator's first motion trajectory, and the collaborative robot's second motion trajectory; based on the first target vector, it can retrieve the historical first vector with the highest vector similarity to the first target vector from the first vector database, and use its corresponding historical collision risk value as the collision risk value.

[0065] The first vector database contains multiple historical first vectors, and each historical first vector has a corresponding historical collision risk value. The historical first vectors are constructed based on historical monitoring data from the second historical moment, the historical actual movement trajectories of the operators during the second historical period, and the historical actual movement trajectories of the collaborative robots during the second historical period. The second historical moment occurs before the second historical period. The second historical moment and the aforementioned first historical moment, as well as the second historical period and the aforementioned first historical period, may be the same or different and are unrelated.

[0066] In some embodiments, for a historical first vector, the collaborative monitoring and management platform can count the number of times the collaborative robot actually performed avoidance M and the number of times the operator and the collaborative robot actually collided N in the historical second time period corresponding to the historical first vector, and then set N... M is defined as the historical collision risk value corresponding to the first historical vector. The number of times the collaborative robot actually performs avoidance, M, refers to the number of times the collaborative robot's motion control system actually receives motion control commands to enable the end effector to complete the avoidance. It can be determined based on the count of the command log of the collaborative supervision and management platform or the feedback count of the collaborative robot completing the avoidance.

[0067] In some embodiments, the collaborative monitoring and management platform may be further configured to: determine a first collision risk value based on monitoring data; obtain a first trajectory set and a second trajectory set based on a first motion trajectory and a second motion trajectory; construct a first envelope set and a second envelope set based on the first trajectory set, the second trajectory set, and the envelope radius; determine an intrusion depth set based on the first envelope set and the second envelope set; determine a second collision risk value based on the intrusion depth set; and determine a collision risk value by weighted summation based on the first collision risk value and the second collision risk value.

[0068] For more information on this section, please refer to [link / reference]. Figure 3 Related explanations.

[0069] Step S230: Based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot, determine the avoidance parameters of the collaborative robot.

[0070] Collision urgency refers to the degree of imminent danger of a collision between an operator and a collaborative robot. In some embodiments, collision urgency may include the minimum predicted distance and the shortest contact time within a future timeframe.

[0071] The minimum predicted distance in the future refers to the minimum spatial distance between the operator and the collaborative robot in the future.

[0072] In some embodiments, the collaborative monitoring and management platform can determine the minimum predicted distance in the future period based on the first motion trajectory and the second motion trajectory. For example, the collaborative monitoring and management platform can determine the minimum predicted distance in the future period by calculating the distance between the two closest points on the first motion trajectory and the second motion trajectory. A minimum predicted distance of 0 in the future period means that the operator and the collaborative robot will collide in the future period.

[0073] The shortest contact time is the shortest time required for the operator and the collaborative robot to collide in the current state.

[0074] In some embodiments, the collaborative monitoring and management platform can calculate the shortest contact time using kinematic formulas based on the relative position vectors and relative velocity vectors of the operator and the collaborative robot at the current moment.

[0075] For example, the collaborative monitoring and management platform can determine the relative position vector based on the current position of the collaborative robot and the operator in the monitoring data; determine the relative velocity vector based on the current velocity of the collaborative robot and the operator in the monitoring data; project the relative position vector onto the direction of the relative velocity vector, and record the resulting projection length as the relative separation distance; and calculate the shortest contact time based on the relative separation distance and the magnitude of the relative velocity vector using the following kinematic formula (1):

[0076] (1)

[0077] Where TTC is the shortest contact time, D is the relative separation distance, and V is the magnitude of the relative velocity vector.

[0078] The mobile space range refers to the geometric space within which a collaborative robot can move safely without collisions.

[0079] In some embodiments, the collaborative monitoring and management platform can calculate the range of movement space using the kinematic model of the collaborative robot and a 3D map of the work area. In some embodiments, the collaborative monitoring and management platform can obtain a 3D map of the work area using computer-aided design (CAD) models or simultaneous localization and mapping (SLAM) mapping; and obtain the kinematic model of the collaborative robot using mathematical calculation software or a robot simulation platform.

[0080] Avoidance parameters refer to the motion parameters related to the collaborative robot's avoidance actions. In some embodiments, avoidance parameters may include the collaborative robot's movement path, movement speed, and timestamps.

[0081] In some embodiments, the collaborative monitoring and management platform can determine avoidance parameters in various ways based on monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot.

[0082] For example, the collaborative monitoring and management platform can determine avoidance parameters by querying a first preset table based on monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot.

[0083] The first preset table includes monitoring data, a first motion trajectory, a collision risk value, a collision urgency, and the collaborative robot's movement space range and corresponding avoidance parameters. In some embodiments, the first preset table can be preset manually based on experience.

[0084] In some embodiments, the collaborative monitoring and management platform can be further configured to: determine the avoidance target based on the task type; and determine the avoidance parameters based on monitoring data, the first motion trajectory, the collision risk value, the collision urgency, the range of movement space, and the avoidance target, using a path planning algorithm.

[0085] The avoidance objective refers to the core performance indicator expected to be achieved when performing an avoidance action. In some embodiments, the avoidance objective may include minimizing the movement path of the collaborative robot during avoidance, avoiding obstacles while maintaining the original orientation and attitude of the end effector, etc.

[0086] In some embodiments, the collaborative supervision and management platform can determine the avoidance target by querying a third preset table based on the task type.

[0087] Task type refers to the category of the task currently being performed. Examples include welding, material handling, and painting. In some embodiments, the collaborative monitoring and management platform can determine the task type based on user input.

[0088] The third preset table includes the correspondence between task types and avoidance targets. This third preset table can be pre-set manually based on experience. For example, when the task type is a material handling task, the collaborative robot's end effector is not sensitive to attitude changes, so the avoidance target is to minimize the robot's movement path for avoidance. When the task type is a painting task, the collaborative robot's end effector is highly sensitive to attitude changes, so the avoidance target is to avoid obstacles while maintaining the end effector's original direction and attitude.

[0089] Path planning algorithms may include, but are not limited to, Rapidly-exploring RandomTree (RRT), Probabilistic Roadmap (PRM), and Dynamic Window Approach (DWA).

[0090] In some embodiments, the collaborative monitoring and management platform can solve the path planning algorithm by taking the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, the range of movement space, and the avoidance target. The path planning algorithm automatically outputs avoidance parameters such as the movement path, movement speed, and timestamp.

[0091] In some embodiments of this specification, by introducing an evasion target, evasion can be achieved while taking into account task efficiency and quality, and reducing interruptions to the task process.

[0092] Step S240: Based on the avoidance parameters, generate motion control commands and send them to the motion control system of the collaborative robot.

[0093] Motion control commands are instructions that control a collaborative robot to perform avoidance maneuvers. In some embodiments, the motion control system is configured to adjust the physical drive signals applied to the joint motors based on the motion control commands, thereby driving the mechanical links of the collaborative robot to produce compound movements, enabling the end effector to complete the avoidance maneuver.

[0094] For more information on motion control systems, articulated motors, mechanical linkages, and end effectors, please refer to [link / reference]. Figure 1 Related explanations.

[0095] In some embodiments, the collaborative monitoring and management platform can compile the movement path and speed in the avoidance parameters according to the controller protocol of the collaborative robot, and automatically generate motion control commands. The controller protocol refers to the communication specifications defined by the collaborative robot manufacturer.

[0096] In some embodiments, the motion control system parses motion control commands, adjusts the physical drive signals of the motors of each joint in real time, and drives the mechanical links of the collaborative robot to generate compound motion, so that the end effector completes the avoidance according to the movement path, movement speed and timestamp of the avoidance parameters.

[0097] In some embodiments, the collaborative monitoring and management platform may be further configured to determine the avoidance type of the collaborative robot based on the collision risk value, task type, and task stage.

[0098] The task phase refers to the current execution stage of a task. Examples include the human-computer interaction phase, the collaborative robot's independent operation phase, and the operator's independent work phase.

[0099] The human-computer interaction stage refers to the stage in which the operator and the collaborative robot work together to complete the task.

[0100] In some embodiments, the collaborative supervision and management platform can determine the task type and task stage through user input.

[0101] For an explanation of task types, please refer to the above text. Figure 2 The relevant description in the document.

[0102] Avoidance type refers to the type of avoidance action performed by the collaborative robot. Examples include normal execution, emergency stop, retreat, deceleration, and detour.

[0103] In some embodiments, the collaborative regulatory management platform can determine the avoidance type by querying a second preset table based on the collision risk value, task type, and task stage.

[0104] The second preset table includes the correspondence between collision risk values, task types, task stages, and avoidance types. This second preset table can be manually set based on experience. For example, when the task type is welding, if the task stage is the human-machine interaction stage and the collision risk value is in the medium risk range (e.g., 0.3-0.6), the corresponding avoidance type is deceleration; if the task stage is the collaborative robot's high-speed operation stage and the collision risk value is in the high risk range (e.g., greater than 0.6), the corresponding avoidance type is retreat.

[0105] In some embodiments of this specification, collision avoidance decisions are made in conjunction with task type and task stage, enabling the system to distinguish between real dangers and necessary collaborative interactions, avoiding over-avoidance or under-response, and achieving an optimal balance between safety and efficiency.

[0106] In some embodiments, the collaborative monitoring and management platform may be further configured to: in response to an emergency stop as the avoidance type, send hardware instructions to the servo driver of the collaborative robot to control the collaborative robot to stop urgently.

[0107] For more information on servo drives, please refer to [link / reference]. Figure 1 Related explanations.

[0108] Hardware commands refer to rapid braking commands that act directly on the servo drive. For example, commands that control the shut-off of safety torque. In some embodiments, hardware commands can be preset manually.

[0109] In some embodiments, when the avoidance type is determined to be an emergency stop, the collaborative monitoring and management platform immediately sends a hardware command to the servo driver, which can immediately shut off the safety torque, enabling the collaborative robot to achieve an emergency stop.

[0110] In some embodiments of this specification, hardware-level emergency stop commands can achieve the fastest and most reliable braking in extremely dangerous situations, ensuring personnel safety, reducing reliance on upper-level software, and improving response determinism.

[0111] In some embodiments of this specification, a complete and closed-loop human-machine collaborative safety monitoring process is constructed. Through multi-source perception and trajectory prediction, a forward-looking risk assessment is achieved, which not only effectively prevents collisions but also minimizes the interruption of the work process, significantly improving the safety of human-machine collaboration and the overall work efficiency.

[0112] Figure 3 This is an exemplary schematic diagram illustrating the determination of collision risk values ​​according to some embodiments of this specification.

[0113] In some embodiments, such as Figure 3 As shown, the collaborative supervision and management platform can be further configured to: determine a first collision risk value 351 based on monitoring data 341; obtain a first trajectory set 321 and a second trajectory set 322 based on a first motion trajectory 311 and a second motion trajectory 312; construct a first envelope set 331 and a second envelope set 332 based on the first trajectory set 321, the second trajectory set 322, and the envelope radius 323; determine an intrusion depth set 342 based on the first envelope set 331 and the second envelope set 332; determine a second collision risk value 352 based on the intrusion depth set 342; and determine a collision risk value 361 by weighted summation based on the first collision risk value 351 and the second collision risk value 352.

[0114] More information regarding monitoring data, the first trajectory, the second trajectory, and collision risk values ​​can be found in [link to relevant documentation]. Figure 2 And related explanations.

[0115] The first collision risk value refers to the component that characterizes the collision risk value of a physical collision between a collaborative robot and an operator, as determined based on historical experience.

[0116] In some embodiments, the collaborative regulatory management platform can determine the first collision risk value by querying a second vector database based on monitoring data.

[0117] For example, the collaborative supervision and management platform can construct a second target vector based on the operator's first state data at the current moment and the collaborative robot's second state data in the monitoring data. Based on the second target vector, it can retrieve the historical second vector with the highest vector similarity to the second target vector from the second vector database and use its corresponding historical first collision risk value as the first collision risk value.

[0118] The second vector database contains multiple historical second vectors, and each historical second vector has a corresponding historical first collision risk value. The historical second vectors are constructed based on the historical actual state data of the operator and the historical actual state data of the collaborative robot from the historical monitoring data at the historical third moment.

[0119] The historical first collision risk value corresponding to the historical second vector can be determined by the actual collision situation in the historical third time period. For example, for a historical second vector corresponding to a historical third moment, the collaborative supervision and management platform can determine the historical first collision risk value corresponding to the historical second vector by statistically analyzing the number of times the collaborative robot actually performed avoidance (M2) and the number of times the operator actually collided with the collaborative robot (N2) in the historical third time period, and then using N2 / M2 as the statistical value. The historical third time period is the subsequent time period of the historical third moment. The historical third moment can be the same as or different from the aforementioned historical first and historical second moments, and there is no correlation between the historical third time period and the aforementioned historical first and historical second moments.

[0120] The first trajectory set refers to the set of multiple discrete points on the operator's first motion trajectory.

[0121] In some embodiments, the collaborative monitoring and management platform can extract N time points within the future time period corresponding to the first motion trajectory; and determine the set of N first trajectory points (denoted as N first trajectory points) corresponding to the N time points in the first motion trajectory as the first trajectory set. Here, N can be set manually based on historical experience, and the intervals between adjacent time points can be equal or approximately equal.

[0122] The second trajectory set refers to the set of discrete points on the second motion trajectory of the collaborative robot. The method for obtaining the second trajectory set is the same as that for obtaining the first trajectory set. Specifically, the N positions corresponding to N time points in the second motion trajectory are denoted as N second trajectory points; that is, the second trajectory set is a set of N second trajectory points. The N time points taken from the first trajectory set and the N time points taken from the second trajectory set are the same N time points.

[0123] The envelope radius refers to the radius of the envelope.

[0124] An envelope is a three-dimensional geometric shape constructed to simplify collision detection and characterizes the safe operating range of an operator or collaborative robot. The shape of an envelope can be a sphere or a cylinder, etc.

[0125] For ease of explanation, this manual uses a spherical envelope as an example. The envelope can be constructed with the operator's or collaborative robot's location as the center and the envelope radius as the radius.

[0126] In some embodiments, the envelope radius includes a first envelope radius and a second envelope radius. The first envelope radius refers to the radius of the envelope centered on the operator. The second envelope radius refers to the radius of the envelope centered on the collaborative robot.

[0127] In some embodiments, the envelope radius can be determined based on the size of the operator or the collaborative robot. For example, the larger the size of the operator, the larger the first envelope radius; the larger the size of the collaborative robot, the larger the second envelope radius.

[0128] In some embodiments, the envelope radius is related to the current velocity and / or current acceleration of the enclosed object.

[0129] The enclosed object refers to the object surrounded by the envelope, including the operator or the body of the collaborative robot.

[0130] In some embodiments, the greater the current velocity and / or current acceleration of the enclosed object, the larger the envelope radius.

[0131] In some embodiments of this specification, the safety boundary is dynamically adjusted based on the object's velocity and acceleration to optimize the collaborative space while ensuring safety.

[0132] In some embodiments, the collaborative supervision and management platform may be further configured to: adjust the envelope radius based on a reduction factor in response to the human-computer interaction phase during the task phase.

[0133] For more information on the human-computer interaction stage, please refer to [link / reference]. Figure 2 Related explanations.

[0134] The reduction factor is a scaling factor used to reduce the radius of the envelope. In some embodiments, the reduction factor is less than 1, and can be set manually based on historical experience.

[0135] In some embodiments, when the task phase is the human-computer interaction phase, the envelope radius needs to be multiplied by a reduction factor for reduction adjustment.

[0136] In some embodiments of this specification, a reduction factor is introduced to narrow the safety boundary while ensuring safety, allowing humans and machines to perform collaborative operations at closer distances and avoiding excessive avoidance.

[0137] The first envelope set refers to the set of first envelopes.

[0138] The first envelope is an envelope constructed with the first trajectory point in the operator's first trajectory set as the center and the first envelope radius as the radius. One first trajectory point corresponds to one first envelope. The set of N first envelopes constructed with N first trajectory points in the first trajectory set is called the first envelope set.

[0139] The second envelope set refers to the set of second envelopes.

[0140] The second envelope is an envelope constructed with the second trajectory point in the second trajectory set of the collaborative robot as the center and the second envelope radius as the radius. One second trajectory point corresponds to one second envelope. The set of N second envelopes constructed with N second trajectory points in the second trajectory set is called the second envelope set.

[0141] The intrusion depth set refers to the set of intrusion depths.

[0142] Intrusion depth refers to the radii of the first and second envelopes at the same time point. The radii of overlap can be defined as the length of the line connecting the centers of the first and second envelopes within the common area where the two envelopes overlap. Each time point corresponds to one first and one second envelope; that is, one time point corresponds to one intrusion depth. The set of N intrusion depths corresponding to N time points is called the intrusion depth set.

[0143] In some embodiments, the collaborative regulatory management platform can calculate and determine the intrusion depth set using the following formula (2).

[0144] (2)

[0145] in, for Intrusion depth at a given point in time; for The first envelope radius of the operator's first envelope at a given time point; In order to be in The second envelope radius of the collaborative robot at a given time point; Let be the Euclidean distance between the centers of the first and second envelopes.

[0146] The second collision risk value refers to the component of the collision risk value that represents the risk value determined based on spatiotemporal overlap analysis.

[0147] In some embodiments, the collaborative regulatory management platform can iterate through time points. arrive The intrusion depth set consisting of the corresponding N intrusion depths is mapped to [0, 1] by a nonlinear function (e.g., the Sigmoid function). The mapping result is determined as the second collision risk value. The nonlinear function can be preset manually based on historical experience.

[0148] In some embodiments, the collaborative regulatory management platform can perform a weighted summation of the first collision risk value and the second collision risk value, and determine the result value as the collision risk value.

[0149] In some embodiments, the weights corresponding to the first collision risk value and the second collision risk value can be preset manually.

[0150] In some embodiments, the weights corresponding to the first collision risk value and the second collision risk value may be related to the confidence level of the first motion trajectory.

[0151] The confidence level of the first motion trajectory refers to the reliability assessment value of the predicted future motion trajectory of the operator.

[0152] In some embodiments, the output of the sequence prediction model may further include the confidence level of the first motion trajectory. For a description of the sequence prediction model, please refer to [link to relevant documentation]. Figure 4 The relevant description. Correspondingly, the training labels of the sequence prediction model can further include the confidence level of the first movement trajectory of the historical operator in the first historical time period (i.e., the historical estimated movement trajectory of the historical operator in the first historical time period). This can be manually labeled based on the degree of overlap between the first movement trajectory of the historical operator in the first historical time period and the historical actual movement trajectory of the historical operator in the first historical time period. The higher the degree of overlap, the greater the confidence level of the first movement trajectory of the historical operator in the first historical time period.

[0153] In some embodiments, when the confidence level of the first motion trajectory is greater than a confidence threshold, the weight corresponding to the second collision risk value is greater than the weight corresponding to the first collision risk value; when the confidence level of the first motion trajectory is equal to or less than the confidence threshold, the weight corresponding to the second collision risk value is less than or equal to the weight corresponding to the first collision risk value. The confidence threshold can be preset manually based on historical experience.

[0154] In some embodiments of this specification, the weights of empirical values ​​and predicted values ​​are dynamically adjusted based on the confidence level of the first motion trajectory. When the confidence level is high, the prediction results are emphasized, and when the confidence level is low, real-time data is emphasized, thereby improving the accuracy of risk assessment.

[0155] In some embodiments of this specification, the first collision risk value and the second collision risk value are complementary, and empirical risk and prospective analysis are combined to improve the comprehensiveness and reliability of collision risk assessment and avoid the limitations of a single method.

[0156] The embodiments described in this invention are merely illustrative and not intended to limit the scope of this specification. Furthermore, specific terms are used to describe the embodiments of this invention. For example, "some embodiments" refers to a particular feature, structure, or characteristic related to at least one embodiment of this invention.

[0157] Furthermore, certain features, structures, or characteristics in one or more embodiments of the present invention can be appropriately combined. Various modifications and alterations that can be made under the guidance of the present invention will be apparent to those skilled in the art and remain within the scope of this specification.

Claims

1. A sensor-based human-machine collaborative monitoring Internet of Things system, characterized in that, Including a collaborative regulatory management platform; The collaborative supervision and management platform is configured as follows: Acquire monitoring data of the collaborative robot's work area; Based on the monitoring data, a first collision risk value is determined; Based on the operator's first motion trajectory and the collaborative robot's second motion trajectory, a first trajectory set and a second trajectory set are obtained; Based on the first trajectory set, the second trajectory set, and the envelope radius, a first envelope volume set and a second envelope volume set are constructed; Based on the first envelope set and the second envelope set, the intrusion depth set is determined; Based on the intrusion depth set, a second collision risk value is determined; Based on the first collision risk value and the second collision risk value, the collision risk value is determined by weighted summation. Based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot, the avoidance parameters of the collaborative robot are determined; and, Based on the avoidance parameters, motion control commands are generated and sent to the motion control system of the collaborative robot. The motion control system is configured to adjust the physical drive signal applied to the joint motor based on the motion control command, so as to drive the mechanical links of the collaborative robot to generate compound motion, so that the end effector can complete the avoidance.

2. The system as described in claim 1, characterized in that, The collaborative supervision and management platform is further configured as follows: Based on the monitoring data and the operator's preset state data, the first motion trajectory is predicted using a sequence prediction model, which is a machine learning model.

3. The system as described in claim 1, characterized in that, The collaborative supervision and management platform is further configured as follows: Based on the task type, identify the avoidance target; Based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, the movement space range, and the avoidance target, the avoidance parameters are determined through a path planning algorithm.

4. A sensor-based human-machine collaborative monitoring method, characterized in that, The method is executed by a collaborative monitoring and management platform in a sensor-based human-machine collaborative monitoring IoT system, and the method includes: Acquire monitoring data of the collaborative robot's work area; Based on the monitoring data, a first collision risk value is determined; Based on the operator's first motion trajectory and the collaborative robot's second motion trajectory, a first trajectory set and a second trajectory set are obtained; Based on the first trajectory set, the second trajectory set, and the envelope radius, a first envelope volume set and a second envelope volume set are constructed; Based on the first envelope set and the second envelope set, the intrusion depth set is determined; Based on the intrusion depth set, a second collision risk value is determined; Based on the first collision risk value and the second collision risk value, the collision risk value is determined by weighted summation. Based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot, the avoidance parameters of the collaborative robot are determined; and, Based on the avoidance parameters, motion control commands are generated and sent to the motion control system of the collaborative robot. The motion control system is configured to adjust the physical drive signal applied to the joint motor based on the motion control command, so as to drive the mechanical links of the collaborative robot to generate compound motion, so that the end effector can complete the avoidance.

5. The method as described in claim 4, characterized in that, The method further includes: Based on the monitoring data and the operator's preset state data, the first motion trajectory is predicted using a sequence prediction model, which is a machine learning model.

6. The method as described in claim 4, characterized in that, The determination of the avoidance parameters of the collaborative robot based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, and the mobile space range of the collaborative robot includes: Based on the task type, identify the avoidance target; Based on the monitoring data, the first motion trajectory, the collision risk value, the collision urgency, the movement space range, and the avoidance target, the avoidance parameters are determined through a path planning algorithm.

7. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the sensor-based human-machine collaborative monitoring method as described in claim 4.

Images