Method for protecting safety of equipment maintenance by fusing mobile network bridge signal and AI image behavior analysis

Abstract

The present disclosure provides a device maintenance safety protection method for fusing mobile network bridge signals and AI image behavior analysis, which comprises collecting mobile network bridge signals, images and controlled device state data and unifying their time sequences and spatial coordinates, binding operator labels, handheld operation terminal numbers and current debugging operation tasks; performing access verification of personnel uniqueness, handheld operation terminal carrying relationship, observation station and enable state; determining safe observation area, controlled device restricted motion area and motion parameters under braking constraints; performing real-time verification of actual motion state of the controlled device and operator station, and outputting corresponding point operation risk control state; executing a hierarchical control strategy, allowing to re-perform access verification after the controlled device stops and the on-site conditions reset, and outputting safety protection results, which effectively protects against risks such as key sticking, inertial overtravel and non-operation shaft linkage, and takes into account the implementability of debugging operation and the safety of restricted motion.

Description

Technical Field

[0001] This disclosure relates to the field of industrial controlled equipment control and equipment maintenance safety protection technology, and in particular to a method for equipment maintenance safety protection that integrates mobile bridge signals and AI image behavior analysis. Background Technology

[0002] In industrial controlled equipment commissioning and maintenance scenarios, maintenance personnel must carry handheld operating terminals into the working area of ​​the controlled equipment. They control the equipment to move at low speeds and short strokes by continuously pressing the enable device, and closely observe the positional relationship between the end effector and the tooling to complete necessary jogging and trajectory debugging tasks. However, existing safety protection solutions generally employ a logic of prohibiting operation upon personnel entry and area locking. This logic triggers a shutdown protection when maintenance personnel enter the area affected by residual movement of the controlled equipment, preventing the controlled equipment from performing any jogging actions. Simply releasing the lock in the jogging state fails to effectively prevent safety risks such as continuous movement due to button sticking on the handheld operating terminal, inertial overtravel exceeding the expected position, unexpected linkage of non-operating axes resulting in abnormal postures, and unauthorized personnel entering the movement path of the controlled equipment. Therefore, existing technical solutions cannot simultaneously meet the basic requirements of close-range jogging operation by maintenance personnel and safety protection for restricted movement of controlled equipment, creating an irreconcilable contradiction between the feasibility of commissioning operations and the safety of the operation process.

[0003] Therefore, there is an urgent need for a device maintenance safety protection method that integrates mobile bridge signals and AI image behavior analysis. This method can effectively protect against risks such as button sticking, inertial overtravel, non-operating axis linkage, and other personnel entry while allowing inching operations under limited personnel, limited positions, limited axes, limited speeds, and limited travel conditions. It can also balance the feasibility of debugging operations with the safety of restricted movement. Summary of the Invention

[0004] In view of this, in order to solve the problems caused by the existing technology, this application provides a device maintenance safety protection method that integrates mobile bridge signal and AI image behavior analysis.

[0005] In a first aspect, this disclosure provides a method for safe maintenance of equipment that integrates mobile bridge signals and AI image behavior analysis, the method comprising: S1: Collect mobile bridge signals, images and controlled device status data and unify their temporal and spatial coordinates, bind the operator's tag, handheld operating terminal number and the current debugging operation task to form a point-motion sensing benchmark; S2: Based on the aforementioned point-sensing benchmark, perform access verification of personnel uniqueness, handheld operation terminal carrying relationship, observation position and enabling status. When the verification is passed, a point-sensing access status with personnel is formed. S3: In the jog entry state with people, determine the safe observation area, the restricted motion area of ​​the controlled equipment, and the motion parameters under braking constraints, and solidify the specified operating axis, allowable speed, and allowable stroke as the jog safety envelope; S4: Under the constraint of the inching safety envelope, the actual motion state of the controlled equipment and the operator's position are verified in real time, and the corresponding inching risk control status is output. S5: Execute the hierarchical control strategy according to the jog risk control status, and allow re-access verification after the controlled equipment stops and the on-site conditions are reset, and output the safety protection result.

[0006] Optionally, S1 includes: S1.1: Record the field status information synchronously from the mobile bridge system, image acquisition device and controlled equipment controller according to a unified acquisition cycle to obtain a multi-source point dynamic observation sequence; S1.2: Using the origin of the controlled equipment base coordinate system as a unified reference origin, the image pixel coordinates are transformed to the controlled equipment base coordinate system using calibration points. The candidate area location of the personnel carrying the tag is determined according to the coordinates of the mobile bridge node and the tag signal strength. For each acquisition time in the multi-source point motion observation sequence, the spatial consistency between the operator's position determined by the image and the candidate area location determined by the bridge signal, as well as the temporal consistency between the acquisition times of various information, are verified. The observation data corresponding to the time when both spatial consistency and temporal consistency reach the preset threshold are included in the unified point motion observation chain. S1.3: Based on the unified point-motion observation chain, according to the spatial proximity and motion synchronization between each tag-carrying person and each handheld operating terminal within each effective acquisition time, determine whether the binding credibility between the person and the handheld operating terminal reaches a preset threshold. When the preset threshold is reached, a candidate binding relationship containing the identity identifier and task configuration information of the bound object and the specified operating axis is formed. S1.4: Perform a uniqueness check on the candidate binding relationships, retain the combination with the highest binding confidence and solidify it to obtain the point motion perception benchmark.

[0007] Optionally, S2 includes: S2.1: Read the point-sensing reference and verify whether the personnel and tags are consistent through cross-comparison of artificial intelligence image recognition and mobile bridge positioning; when no other tags or unbound personnel appear in the entire observation window, and based on the cross-comparison results, when the correspondence between the bound personnel and the tags is stable and consistent in the entire observation window, the personnel uniqueness verification is determined to be passed. S2.2: During the observation period when the personnel uniqueness verification is qualified, the position information of the handheld operating terminal and the holding hand is extracted by artificial intelligence image recognition. Based on the spatial correlation and motion consistency between the personnel and the handheld operating terminal, it is determined whether the personnel and the handheld operating terminal have passed the continuous carrying verification. S2.3: Verify the operator's position space compliance through artificial intelligence image recognition, and whether the enabling device is in the middle effective pressing position, to form a joint verification result of position and enabling; S2.4: When the personnel uniqueness verification, the continuous carrying verification of the handheld operating terminal, and the joint verification of the station position and enable are all passed, the personnel-carrying point access state is formed.

[0008] Optionally, S3 includes: S3.1: Use artificial intelligence image recognition to extract key points of the operator's human posture, determine the dynamic extension radius based on the human posture and motion parameters, and extend along the direction pointing to the controlled equipment based on the operator's body outline to form the operator's dynamic occupancy boundary. S3.2: Overlay and analyze the dynamic occupation boundary of the operator with the specified observation station area, delete the positions with intrusion risk to obtain the safe observation area, and generate candidate restricted motion areas of the controlled equipment by advancing step by step according to the specified operating axis and allowed motion direction. Calculate the minimum distance from the dynamic occupation boundary of the operator to the leading edge of the end effector's motion direction as the available human-machine separation distance, and the available path length from the current position of the end effector along the specified motion direction to the tooling boundary. S3.3: Obtain the braking parameters corresponding to the current debugging operation task based on the pre-implemented braking test, and determine the maximum stroke of a single jog based on the available separation distance between the human and machine, the available path length on the tooling side, and the braking parameters; S3.4: Based on the maximum stroke of a single jog, select the actual allowable stroke that does not exceed the maximum stroke, and reverse the calculation of the upper limit of the allowable speed; solidify the specified operating axis, the allowable motion direction, the spatial constraint information and the motion constraint information together to form the jog safety envelope.

[0009] Optionally, the spatial constraint information includes the coordinates of the safe observation area, the dynamic occupancy boundary of personnel, and the restricted movement area of ​​the controlled equipment; the braking parameters include braking response time and braking deceleration parameters; and the motion constraint information includes the single allowable travel distance, the upper limit of allowable speed, braking response time, and braking deceleration parameters.

[0010] Optionally, S4 includes: S4.1: Read the jog safety envelope, continuously collect the real-time status of the controlled device when the enabling device enters the effective pressing state and triggers the jog button, calculate the cumulative jog stroke segment by segment based on the actual speed of the end effector along the specified axis direction, and verify whether the operator remains within the safe observation area. S4.2: When the cumulative travel reaches the single-operation allowable travel, the new travel is set to zero; at the same time, the timer starts from the moment the jog button is pressed, and when the continuous pressing time reaches the single operation limit, the increase of travel stops and the pressing timeout is determined; when the cumulative travel enters the critical area with the single-operation allowable travel as the upper limit, the travel is determined to be critical; when the actual movement direction is opposite to the allowable direction, the operator leaves the observation area, the non-designated operating axis is displaced, or other personnel enter, the boundary is determined to be exceeded. S4.3: If the pressing signal continues after the operator releases the jog button, it is determined that the button is stuck; if the pressing signal has been released but the controlled equipment still has residual speed, the remaining braking margin is calculated from the moment the button is released. When the remaining braking margin always meets the braking distance required for complete stop and the speed drops to within the stationary limit, it is determined to be controlled inertial braking. When the final stopping position exceeds the jog safety envelope boundary, or the displacement of the non-specified operating axis exceeds the natural fluctuation reference, it is determined to be inertial overtravel. S4.4: Summarize the above verification and judgment results, and output the corresponding jog risk control status.

[0011] Optionally, the summary of the above verification and judgment results, and the jogging risk control status include: When there is no press timeout, out-of-bounds, button sticking, controlled inertial braking, or inertial overtravel, the output is in normal jogging mode. When a critical travel threshold is reached, a critical stop state is output. When controlled inertial braking occurs, output the controlled inertial braking state; When any of the following occurs: key sticking, inertia overtravel, or exceeding the limit, an out-of-limit danger status will be output.

[0012] Optionally, S5 includes: S5.1: Generate corresponding graded control instructions based on the jogging risk control status. For normal jogging status, maintain the current speed limit and continue to perform jogging operation. For critical stop status, gradually reduce the speed limit according to the remaining allowable travel and the current remaining braking margin to stop the controlled equipment before the boundary. For controlled inertial braking status, maintain braking monitoring and wait for natural stopping. For dangerous out-of-bounds status, directly revoke the jogging permission and trigger emergency braking. S5.2: Execute the corresponding physical control operation according to the hierarchical control command, wherein a lockout flag is written in the boundary crossing danger state; S5.3: After braking or locking is completed, verify the stopping stability of all moving shafts; after the stopping stability is qualified, confirm that the site conditions meet the re-access requirements and form a re-access permit; S5.4: Return the re-access permission to step S2 and output the security protection result of the corresponding branch.

[0013] In a second aspect, this disclosure provides an electronic device including a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to implement the method of the first aspect described above.

[0014] Thirdly, this disclosure provides a computer storage medium storing a computer program that, when executed, implements the method described in the first aspect.

[0015] The beneficial effects of the present invention are as follows: Compared with the prior art, the present invention has the following advantages: By uniformly collecting and synchronously transmitting mobile bridge signals, images, and controlled equipment status data in time and space, personnel tags, handheld operating terminals, and tasks are bound to form a perception benchmark. Based on this, joint access verification is performed on personnel uniqueness, handheld operating terminal carrying, observation position, and enabled status. Inching is only allowed when all conditions are met. This determines the safe observation area and motion parameters under braking constraints and solidifies them into an inching safety envelope. During inching, the motion status of the controlled equipment and the personnel's position are verified in real time. According to the risk level, graded control is implemented, including normal hold, critical deceleration, or emergency braking. Re-access is only allowed after the controlled equipment stops and is reset on-site. While ensuring the feasibility of close-range inching operation by maintenance personnel, effective protection against button sticking, inertial overtravel, non-designated axis linkage, and unauthorized personnel entry is achieved throughout the entire process, balancing commissioning efficiency and operational safety. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0017] Figure 1 A flowchart of a device maintenance safety protection method integrating mobile bridge signal and AI image behavior analysis provided in an embodiment of this disclosure is shown. Figure 2 A flowchart illustrating the multi-source data fusion and point-motion sensing reference formation provided in an embodiment of this disclosure is shown. Figure 3 A flowchart illustrating the access verification process for bringing people along, as provided in an embodiment of this disclosure, is shown.

[0018] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0019] The present disclosure will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solutions of the present disclosure more clearly, and should not be used to limit the scope of protection of the present disclosure.

[0020] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] Figure 1 A flowchart illustrating the equipment maintenance safety protection method integrating mobile bridge signal and AI image behavior analysis provided in this disclosure embodiment is shown below. Figure 1 As shown, the process may include the following steps: S1: Collect mobile bridge signals, images, and controlled device status data, unify their timing and spatial coordinates, bind operator tags, handheld terminal numbers, and current debugging operation tasks to form a point-motion sensing reference.

[0022] S1.1: Record on-site status information synchronously from the mobile bridge system, image acquisition device and controlled equipment controller according to a unified acquisition cycle to obtain a multi-source point dynamic observation sequence.

[0023] Figure 2 A flowchart illustrating the multi-source data fusion and point-motion sensing reference formation process provided in this disclosure embodiment is shown, such as... Figure 2 As shown, a synchronous acquisition command is triggered every 50 milliseconds according to a pre-set unified acquisition cycle, and the on-site status information is recorded from the mobile bridge system, the image acquisition device, and the controller of the controlled equipment.

[0024] Specifically, the mobile bridge system includes multiple bridge nodes permanently installed within the work area, and operators carry active tags with unique numbers. At each data collection moment, the system records the tag signal strength received by each bridge node, the node number currently connected to the tag, the time when the tag switches between adjacent nodes, and the duration of continuous tag residence within the current area. This information is used to subsequently determine the spatial area where the tag is located.

[0025] Simultaneously, multiple visual acquisition devices deployed within the work area capture images including the operator's full-body silhouette, the appearance of the handheld operating terminal, the motion axes of the controlled equipment, end effectors, tooling, and work area boundary markings. The number and installation location of the visual acquisition devices should ensure complete coverage of the work area without blind spots.

[0026] Simultaneously, the press status of the enable device of the handheld operating terminal, the press status of the jog button, the currently specified operating axis number, the actual speed of each motion axis of the controlled device, and the axial displacement of each axis are read from the controller of the controlled device.

[0027] For each piece of information collected above, its source number, acquisition time, and valid status marker are appended to form a multi-source point motion observation sequence aligned according to the acquisition time. Each data item in this sequence can be indexed by acquisition time and spatial source for spatiotemporal unified processing in subsequent sub-steps.

[0028] S1.2: Verify the spatial consistency between the operator's position determined by the image and the candidate area position determined by the bridge signal, as well as the temporal consistency between the acquisition times of various information items. Incorporate the observation data corresponding to the time when both spatial consistency and temporal consistency reach the preset threshold into the unified point motion observation chain.

[0029] After obtaining the multi-source point motion observation sequence, for each acquisition time in the sequence, since the mobile bridge positioning, image coordinates and controlled equipment coordinates use different reference benchmarks, it is necessary to unify these heterogeneous information into a common spatiotemporal framework.

[0030] In the spatial dimension, the origin of the controlled equipment base coordinate system is used as a unified reference origin. Multiple pre-set calibration points with known coordinates are utilized within the working area. These calibration points should simultaneously appear in the visual image, and their base coordinates should be accurately determined. A coordinate transformation matrix is ​​used to transform the operator's position, handheld terminal position, end effector position, and tooling boundary contour identified in the image from image pixel coordinates to the controlled equipment base coordinate system. Simultaneously, based on the fixed installation coordinates of each mobile bridge node and the sorting of tag signal strength received by each node at the current acquisition time, combined with the connection relationship between tags and nodes, candidate locations for the tagged personnel within the working area are determined.

[0031] In terms of time, considering the potential slight discrepancies between the mobile bridge recording time, image acquisition time, and controlled device status recording time, for the k-th acquisition time, the spatial consistency between the operator's position determined by the image and the candidate area position determined by the bridge signal at that time, as well as the temporal consistency between the mobile bridge recording time, image acquisition time, and controlled device status recording time, are verified. As an optional implementation, the spatiotemporal consistency at that time is calculated using the following formula: ; in, For spatiotemporal consistency; Record time for mobile bridge, The time of image acquisition. The time of recording the status of the controlled device is in milliseconds; The distance between the center of the label area and the operator's position in the image; The duration of stay in a continuous region within the current observation window. The duration of the observation window; To allow for a time skew scale, it is set to 40 to 50 milliseconds; The corresponding spatial scale is set to 0.18 to 0.22 meters; , and All are dimensionless attenuation coefficients, set to 0.9 to 1.1 respectively. Understandably, the attenuation coefficients are only used to suppress significantly misaligned observations, while accommodating inherent differences within a normal acquisition cycle.

[0032] The spatiotemporal consistency degree calculated using the above formula This reflects the degree of temporal and spatial matching of information from various sources at the time of acquisition. When When a preset spatiotemporal consistency threshold is reached, the observation data at that moment is considered valid. This threshold is set between 0.82 and 0.88. For collection moments that do not reach the threshold, the corresponding observation data will not be included in subsequent personnel binding processing to avoid incorrect associations due to spatiotemporal misalignment.

[0033] Through the above processing, we obtain a result containing each valid acquisition time and its corresponding... A unified point-motion observation chain.

[0034] S1.3: Based on the spatial proximity and motion synchronization between each tag-carrying person and each handheld operating terminal within each valid collection time, determine whether the binding credibility between the person and the handheld operating terminal reaches a preset threshold. When the preset threshold is reached, a candidate binding relationship is formed, which includes the identity identifier and task configuration information of the bound object and the specified operating axis.

[0035] After obtaining a unified point-motion observation chain that has passed spatiotemporal consistency screening, the binding reliability between each tag-carrying personnel and each handheld operating terminal within the effective acquisition time is calculated.

[0036] Specifically, within the same effective acquisition time, the positions of the operator and the handheld operating terminal identified in the image are extracted, and the spatial distance between them is calculated to characterize the spatial proximity between the operator and the handheld operating terminal. Simultaneously, the movement speeds of the operator and the handheld operating terminal are calculated using the tracking results of consecutive frames, and the speed difference between the two is calculated to characterize the motion synchronization between the operator and the handheld operating terminal. For the m-th tag-carrying personnel and the n-th handheld operating terminal, the binding confidence is calculated using the following formula: ; in, The binding reliability between the m-th tag-carrying person and the n-th handheld operating terminal is expressed as a unit of one; K is the total number of valid data collection moments that reach the spatiotemporal consistency threshold. The distance between the operator and the handheld terminal at the kth valid moment is expressed in meters. and These represent the moving speeds of the operator and the handheld terminal at the k-th valid moment, respectively, in meters per second. To allow for a safe carrying distance, the distance is set to 0.45 to 0.55 meters, which is used to constrain the normal holding distance between the user and the handheld operating terminal. To allow for speed difference, a range of 0.18 to 0.22 meters per second is set to constrain the degree of synchronous movement between the personnel and the handheld operating terminal. and The values ​​are dimensionless attenuation coefficients, all set to 0.9 to 1.1.

[0037] The significance of calculating binding reliability lies in the fact that only when the operator and the handheld terminal maintain a continuous close spatial distance and synchronized speed in the direction of movement, and these conditions are continuously met across multiple valid data acquisition moments, can the binding reliability be calculated. Only then will a higher value be achieved. When When a preset binding trust threshold is reached, a valid binding relationship is determined between the person and the handheld operating terminal. This threshold is set to be between 0.80 and 0.86. After reaching the threshold, the number of the current debugging operation task and the operation axis specified by the task are queried based on the handheld operating terminal number. This generates a candidate binding relationship containing the identity identifier and task configuration information of the bound object, as well as the specified operation axis. The identity identifier and task configuration information include at least the person's identity, tag number, handheld operating terminal number, and task number.

[0038] S1.4: Perform a uniqueness check on the candidate binding relationships, retain the combination with the highest binding confidence and solidify it to obtain the point motion perception benchmark.

[0039] After obtaining the candidate binding relationships, perform uniqueness checks and solidification operations.

[0040] Specifically, check whether the same debugging operation task corresponds to only one operator, one personnel tag, and one handheld operating terminal. If a debugging operation task has multiple candidate binding relationships, then follow the... Filter by high to low, and retain The highest combination. Only when... Reaching the threshold The binding relationship will only be solidified if the threshold is reached consecutively and the handheld operating terminal number matches the current debugging operation task number.

[0041] The solidified jog sensing baseline includes the following information: personnel tag number, handheld operating terminal number, debugging operation task number, designated operating axis, personnel's spatial position within the work area, current spatial position of the end effector, tooling boundary coordinates, press status of the handheld operating terminal's enable device, press status of the jog button, actual speed of each motion axis of the controlled equipment, and axial displacement of each axis. This baseline provides a unified sensing basis for subsequent personnel-assisted jog access judgment, ensuring that only a single personnel, a single handheld operating terminal, and an operation corresponding to a clearly defined task are allowed to obtain jog access in subsequent steps.

[0042] S2: Based on the jog sensing benchmark, perform access verification of personnel uniqueness, handheld operation terminal carrying relationship, observation position and enabling status. When the verification is passed, a jog access status is formed.

[0043] S2.1: Verify whether the personnel and tags are consistent by cross-referencing artificial intelligence image recognition and mobile bridge positioning; verify the uniqueness of personnel in the work area.

[0044] Figure 3 The flowchart of the manual access verification process provided in this embodiment of the present disclosure is shown, such as... Figure 3 As shown, the point-sensing reference output in step S1 is read, and the number of the bound tag, the spatial position of the bound person in the work area, the boundary coordinates of the work area, and the person recognition result obtained by artificial intelligence image recognition are obtained from it.

[0045] Specifically, the AI ​​image recognition process performs the following operations sequentially: extracting the person's silhouette from the image acquired by the visual acquisition device; determining the person's foot and hand positions using a human pose estimation model; and recording the moment the person enters the work area. Subsequently, each person identified in the image is cross-referenced with the tag area determined by the mobile bridge system to ensure that the person in the image matches the person carrying the tag. The person recognition confidence threshold is set to 0.86 to 0.90. This range effectively eliminates interference from human-like silhouettes formed by equipment components on the recognition results and strikes a balance between high confidence requirements and resistance to recognition fluctuations caused by local occlusion, thereby reducing the false alarm rate while maintaining effective detection capability for actual personnel.

[0046] Based on the cross-comparisons described above, the stability of personnel uniqueness is determined by considering multiple factors. Specifically, the proportion of the continuous presence time of the bound personnel within the current observation window to the total observation window duration is calculated. A higher proportion indicates that the bound personnel remain within the work area without leaving midway. The consistency between the bound label and the current debugging operation task number is verified to ensure that the currently present personnel are indeed the operators designated for that debugging operation task. The total number of personnel within the work area and the number of other non-bound personnel are counted. The higher the proportion of continuous presence time of bound personnel, the closer the total number of personnel within the work area is to uniqueness, and the fewer the number of non-bound personnel, the higher the stability of personnel uniqueness.

[0047] However, regardless of the stability assessment, if any other tag or unlinked personnel is detected within the work area at any given time, a non-uniqueness result is automatically generated, and the stability score is no longer considered. Personnel uniqueness stability is only calculated when no other tags or unlinked personnel appear within the entire observation window. Personnel uniqueness verification is considered successful when this stability reaches a preset threshold of 0.92 to 0.96. A higher threshold ensures that linked personnel remain present throughout the observation window, avoiding security uncertainties caused by frequent personnel movement.

[0048] S2.2: Extract the position information of the handheld operating terminal and the hand holding part through artificial intelligence image recognition, and verify the continuous carrying relationship of the handheld operating terminal based on the spatial correlation and motion consistency between the person and the handheld operating terminal.

[0049] During the observation period when the personnel uniqueness verification is qualified, further verification is conducted to determine whether the handheld operating terminal is continuously carried by the bound personnel.

[0050] Specifically, the handheld terminal number corresponding to the current task is read from the point-to-point sensing reference. Then, the bounding rectangle of the handheld terminal, the position of the operator's hand in the image, and the real-time movement speed of both are extracted using artificial intelligence image recognition results. Based on this, a comprehensive determination is made as to whether the person being bound to the device continues to hold the handheld terminal.

[0051] The following three factors are considered during the judgment process: the proportion of time the person holding the corresponding handheld operating terminal within the current observation window relative to the total observation window time; a higher proportion indicates that the handheld operating terminal is always in the person's hand and has not been removed; the spatial distance between the grip and the center of the outer rectangle of the handheld operating terminal, which should be within the normal grip range; and the degree of synchronous movement between the grip and the handheld operating terminal, i.e., whether their velocity vectors are consistent, to exclude the situation where the handheld operating terminal is placed in a fixed position and the person moves it on their own. In the above judgment, the grip distance scale is set to 0.18 to 0.22 meters, which covers the distance of normal one-handed and two-handed gripping of the handheld operating terminal; the synchronous movement speed difference scale is set to 0.18 to 0.22 meters per second, which is used to accommodate small speed differences during wrist adjustments, but does not allow the handheld operating terminal to be placed on the tool surface or handed over to other personnel. The above three factors together characterize the spatial correlation and motion consistency between the person and the handheld operating terminal.

[0052] When the combined result of the above three factors reaches the consistency threshold, the continuous carrying verification of the handheld operating terminal is deemed successful. This threshold is set between 0.86 and 0.90. If the consistency does not reach the threshold, it indicates that the handheld operating terminal may have been out of the control of the bound personnel, and in this case, jog permissions are not granted.

[0053] S2.3: Verify the operator's position space compliance through artificial intelligence image recognition, and whether the enabling device is in the middle of the effective pressing position, to form a joint verification result of position and enabling.

[0054] After verifying the uniqueness of the personnel and the continuous carrying of the handheld operating terminal, a further joint verification is conducted on the operator's observation position and the operational status of the enabling device. This joint verification is used to determine the compliance of the operator's position space and the effectiveness of the enabling mechanism.

[0055] Specifically, the coordinates of the pre-specified observation station area and the prescribed observation direction for this debugging operation task are read from the jog sensing reference. The coordinates of the operator's foot centers, the envelope of their body contour, and the angle of their facing direction are extracted using artificial intelligence image recognition results. Then, the following three conditions are verified: whether the center of the feet is located within the prescribed observation station area; whether the personnel occupancy envelope constructed based on the current body posture and the estimated dynamic extension range enters the boundary of the movement area reserved by the controlled equipment; and whether the personnel are facing the end effector to observe its movement state.

[0056] At the same time, the press position of the enabling device is read from the status information of the handheld operating terminal. The enabling device usually has three press positions: fully released, intermediate active press, and fully depressed. Only when the enabling device is in the intermediate active press position is it recorded as an enabled state; when the enabling device is in the fully released or fully depressed position, it is recorded as an disabled state.

[0057] The determination of joint validity comprehensively considers the distance of the bipedal center from the center of the designated observation station, the angular deviation of the person's facing direction from the designated observation direction, and whether the enabling state remains effective. The smaller the deviation distance, the smaller the directional deviation, and the more effective the enabling state, the higher the joint validity. Specifically, the allowable offset scale for the observation station is set to 0.45 to 0.55 meters, and the allowable directional deviation is set to 0.35 to 0.43 radians. The joint validity threshold is set to 0.80 to 0.84. In addition to meeting the joint validity threshold, it is strictly required that the enabling state remains effective throughout all acquisition moments within the entire observation window, and the person's body outline must not enter the reserved movement area of ​​the controlled equipment. If any of the above conditions are not met, the joint verification fails.

[0058] S2.4: When the personnel uniqueness verification, the continuous carrying verification of the handheld operating terminal, and the joint verification of the station position and enable are all passed, the personnel-carrying point access state is formed.

[0059] After the personnel uniqueness verification, continuous carrying verification of handheld operating terminal, and joint verification of position and enable are all passed, a comprehensive entry status with personnel is formed.

[0060] Specifically, when all three verifications above are passed, and there are no other abnormal situations within the entire observation window, such as unbound personnel, handheld operating terminals being released from the hands, personnel crossing boundaries, or enabling failures, an access permission status is output. This access permission status includes the following information: bound personnel tag number, handheld operating terminal number, debugging operation task number, specified operating axis, current observation station coordinates, and the time when access was granted.

[0061] Conversely, if any of the above three verifications is not met, or if any abnormality occurs within the entire observation window, an access-prohibited state will be output, and the corresponding reason for the prohibition will be recorded, such as the specific reason for the personnel's non-uniqueness, the moment the handheld operating terminal was released, or the moment the enable failed. In the access-prohibited state, the jog permission of the controlled device remains closed, and it will not enter the motion area calculation of subsequent steps. Only when all conditions are met simultaneously will step S3 establish a jog safety envelope for the access-allowed state.

[0062] S3: In the jog entry state with people, determine the safe observation area, the restricted movement area of ​​the controlled equipment, and the motion parameters under braking constraints, and solidify the specified operating axis, allowable speed, and allowable stroke into the jog safety envelope.

[0063] S3.1: Use artificial intelligence image recognition to extract key points of the operator's human posture, determine the dynamic extension radius based on the human posture and motion parameters, and extend along the direction pointing to the controlled equipment based on the operator's body contour to form the operator's dynamic occupancy boundary.

[0064] Determine the type of the jog access status output in step S2. If the input is a prohibited access status, a closed flag will be directly generated. This flag will be passed down level by level in subsequent sub-steps, and no calculation of any motion area will be enabled. If the input is an allowed access status, read the bound operator's number, the specified observation station coordinates, the person's body contour data, the real-time position of the hand, the current spatial position of the end effector, and the specified operating axis information.

[0065] Specifically, the system utilizes AI image recognition results to extract the positions of key points on the operator's shoulders, elbows, wrists, and hand extremities, while also identifying the position of the tool's end when the operator is holding it. For limb key points with a confidence score below 0.86 to 0.90, the recognition results are not discarded directly. Instead, they are evaluated by combining the results of multiple consecutive frames of images, and their reliable positions are determined through interpolation or voting methods. The estimation of the area occupied by the person is not narrowed down based on a single recognition result.

[0066] Based on this, considering that the operator may naturally extend their arm or adjust their body posture during observation, the current body outline cannot be used as the occupancy boundary alone; a dynamic extension radius that can cover short-term movement changes needs to be calculated. For the k-th acquisition time, the dynamic extension radius is determined according to the following formula: ; in, For dynamic extension radius; The actual unfolded length from the shoulder to the end of the hand or the end of the hand-held tool is calculated by combining the pixel distance between the shoulder key points and the end-of-hand key points in the image with camera calibration parameters. The outward movement speed of the hand along the direction pointing towards the controlled device; Let the outward acceleration of the hand be the acceleration in the direction pointing towards the controlled device. When the hand moves away from the controlled device, the outward velocity is... and outward acceleration All values ​​are set to zero to avoid incorrectly including the movement of personnel moving away from the controlled equipment in the extension trend toward the controlled equipment; Allow time for personnel actions, set to 0.45 to 0.55 seconds. This time covers the time required for the operator to continue reaching out and the processing delay required for the system to recognize the action. The allowance for body posture and clothing expansion is set to 0.08 to 0.10 meters to cover the edges of loose work clothes and the outline of hand tools.

[0067] The dynamic extension radius is defined as the sum of three parts: the current upper limb extension length, the distance of continued extension in a short period of time, and the margin for fixed posture. The distance of continued extension in a short period of time includes two components: uniform extension due to the current velocity and accelerated extension due to the current acceleration. By introducing velocity and acceleration terms, we can avoid determining the boundary of a person's occupation solely based on their current body contours, thus ignoring potential changes in movement that may occur within a very short timeframe.

[0068] Finally, based on the operator's body contours, the dynamic extension radius is used along the direction pointing towards the controlled equipment. The boundary is expanded to form an envelope, which is the dynamic boundary occupied by the operator.

[0069] S3.2: Overlay the dynamic occupation boundary of the operator with the specified observation station area for analysis, delete the location with intrusion risk to obtain the safe observation area, and calculate the available separation distance between the human and machine and the available path length.

[0070] After obtaining the dynamic occupancy boundary of the operator, the dynamic occupancy boundary is overlaid with the observation station area specified in step S2 for analysis.

[0071] Specifically, the coordinates of the observation station area specified in step S2 are read, and it is checked whether there is a location in the observation station area where, if an operator stands in that location, their dynamic occupancy boundary would encroach on the current outline of the controlled equipment, the tooling boundary, or the space traversed by the movement path of a designated operating axis. Locations with intrusion risk are removed from the observation station area, leaving only the area where personnel can stand, clearly observe the movement status of the end effector, and safely release the enabling device at any time. This area is the final safe observation area. It should be noted that the safe observation area must not cross the base side of the controlled equipment, the back side of the tooling, or locations where the full outline of the personnel cannot be continuously confirmed by the image, to ensure the continuous effectiveness of visual monitoring.

[0072] Simultaneously, candidate restricted motion areas for the controlled device are generated starting from the current position of the end effector, according to the specified operating axis and allowed motion direction. Specifically, for linear motion axes, the contour of the end effector and its auxiliary tools is progressively advanced along the specified motion direction at intervals of 4 to 5 millimeters; for rotary motion axes, the arc sweep position of the outermost edge of the tool in space is calculated at angular intervals of 0.8 to 1.0 degrees. These interval dimensions can effectively identify narrow gaps that may exist in jogging operation scenarios, while avoiding the generation of too many intermediate positions with no real difference due to excessively small intervals.

[0073] Each advance verifies whether the controlled equipment body, end effector, and tool outline are in spatial contact with the tooling boundary or the operator's dynamic occupancy boundary. The continuous advance area before the first contact is designated as the candidate restricted motion area for the controlled equipment. Within this candidate area, the minimum spatial distance between the operator's dynamic occupancy boundary and the leading edge of the end effector's movement direction is calculated; this distance is recorded as the available separation distance between the operator and the machine. Simultaneously, the available path length from the current position of the end effector along the specified movement direction to the tooling boundary is calculated and recorded as the tooling-side available path length.

[0074] If step S3.1 transmits the closing flag, or the safe observation area is an empty set, or the candidate restricted motion area is discontinuous in space, then the human-machine separation result in the closed state is directly output, and no further calculations are performed.

[0075] S3.3: Obtain the braking parameters corresponding to the current commissioning operation task based on the pre-implemented braking test, and determine the maximum stroke for a single jog.

[0076] After obtaining the human-machine separation results, considering that even if the controlled equipment moves in jog mode, it still needs a certain braking distance to come to a complete stop after receiving a stop command, it is necessary to determine the maximum allowable stroke for a single jog based on the actual braking parameters.

[0077] Specifically, a braking test is pre-conducted at the rated jogging speed of the controlled equipment. Multiple tests are performed under both no-load and current tooling load conditions. The response time from the release of the jogging command to the actual start of brake action, as well as the effective deceleration during braking, are recorded. The test results with the larger braking distance under both load conditions are taken as the response time and effective deceleration parameters for the current commissioning operation. These parameters are obtained through field testing and are not artificially set parameters, thus accurately reflecting the braking performance of the controlled equipment under current operating conditions.

[0078] The usable human-machine separation distance calculated based on sub-step S3.2 The available path length on the tooling side, braking response time, and braking deceleration parameters are used to determine the maximum stroke in a single inching motion using the following formula: ; in, This represents the maximum stroke in a single inching motion. The geometrically available path length from the current position of the controlled device along the specified direction to the boundary of the candidate restricted motion area; The usable human-machine separation distance calculated for sub-step S3.2 is the minimum distance between the dynamic occupation boundary of the operator and the leading edge of the end effector. The fixed safety margin after the inching stop is set to 0.18 to 0.22 meters to cover slight body sway of the operator and elastic recovery of the structure after the controlled equipment has come to a complete stop; The maximum speed limit for jog mode provided to the controller of the controlled equipment; The actual braking response time from the momentary release to the start of braking is measured by a braking test; This is the braking deceleration under the current load, measured by the braking test.

[0079] If the calculation result is not greater than zero, it means that the current human-machine distance is insufficient to accommodate the braking distance of the controlled equipment. In this case, the maximum stroke of a single jog is set to zero, and the system remains in the off state. If the calculation result is positive, the subsequent actual jog stroke must not exceed this value. The actual selected jog stroke is less than... The remaining distance is then written into the jog stroke constraint result as the braking margin.

[0080] S3.4: Based on the maximum stroke of a single jog, select the actual allowable stroke that does not exceed the maximum stroke, and reverse the calculation of the upper limit of the allowable speed; solidify the specified operating axis, the allowable motion direction, the spatial constraint information and the motion constraint information together to form a jog safety envelope.

[0081] After obtaining the maximum stroke of a single inching motion, the actual allowable stroke and the upper limit of allowable speed are further determined, and all constraints are solidified into the inching safety envelope.

[0082] Specifically, select a distance that does not exceed the maximum stroke of a single jog. The actual permitted itinerary is recorded as .Should This is the maximum travel distance allowed for a single press during this inching operation. Based on this, and according to the available separation distance between the human and machine... and selected The maximum permissible speed is calculated using the following formula: ; in, The maximum permissible speed; The upper limit of the jog mode speed specified by the controller of the controlled equipment; The permissible stroke for a single inching operation must be fixed and must not exceed the maximum stroke for a single inching operation. Other variables , , , The meaning is the same as in sub-step S3.3.

[0083] This formula is derived by working backward from the condition that the braking distance does not exceed the usable separation distance between the human and machine.

[0084] Subsequently, the following information will be bound: personnel tag number, handheld operating terminal number, debugging operation task number, specified operating axis, permitted movement direction, and spatial constraint information including the coordinates of the safety observation area, the dynamic occupancy boundary of personnel, and the restricted movement area of ​​controlled equipment, as well as information including the single permitted travel distance. Maximum permissible speed Braking parameters and braking deceleration Including motion constraint information, human-machine usable separation distance, and initial braking margin. (Its value is equal to the maximum travel of a single jog) With single permitted trip The difference between the two (and the effective time of the safety envelope) are combined to form the inching safety envelope. Among these, if the travel is allowed... Or the maximum speed allowed If any one of these values ​​is zero, a closed-type jog safety envelope is output; if both are positive, an open-type jog safety envelope is output. Step S4 will perform actual motion verification on the open-type jog safety envelope and continuously maintain jog prohibition on the closed-type jog safety envelope.

[0085] S4: Under the constraint of the inching safety envelope, the actual motion state of the controlled equipment and the operator's position are verified in real time, and the corresponding inching risk control status is output.

[0086] S4.1: When the enabling device enters the effective pressing state and triggers the jog button, continuously collect the real-time status of the controlled device, calculate the cumulative jog stroke, and verify whether the operator remains within the safe observation area.

[0087] Read the jog safety envelope output in step S3 and determine whether it is open or closed. If it is a closed jog safety envelope, a motion prohibition record is directly generated, and no jog increments are accepted. The subsequent verification operation in step S4 is not performed. If it is an open jog safety envelope, extract the following constraint information from the jog safety envelope, including the specified operating axis number, allowed motion direction, and allowed speed limit. Single permitted trip Coordinates of the safety observation area and the boundary of the restricted movement area of ​​the controlled equipment.

[0088] When the operator enables the device to enter a valid pressing state and triggers the jog button, the real-time status of the controlled device begins to be continuously collected. Specifically, at each collection moment, the position data of each motion axis of the controlled device, the actual speed of the end effector along each direction, the input direction of the jog command, and the pressing status of the handheld operating terminal button are read. Then, based on the motion speed of the end effector along the specified operating axis direction, the cumulative jog travel from the start of this jog to the current moment is calculated. For the nth collection moment, the cumulative jog travel is calculated according to the following formula: ; in, This represents the cumulative jog distance from the start of this jog to the nth data acquisition moment; and These represent the actual speeds of the end effector along the specified axis at the (k-1)th and kth acquisition times, respectively. and Let k be the sequence number of two adjacent acquisition times, and n be the sequence number of the current acquisition time. This formula uses the product of the average velocity of two adjacent times and the time interval as the displacement estimate within that time interval, and then accumulates these values ​​segment by segment. This calculation method can continuously estimate the actual travel distance during different motion stages of the controlled equipment, such as acceleration, constant speed, and deceleration, and after accumulation, obtain the total travel distance from the start of the jogging motion to the current time.

[0089] Simultaneously, AI image analysis continuously extracts the operator's foot position, body contour envelope, hand position, and the total number of personnel in the work area to verify whether the operator remains within the safe observation area determined in step S3. The personnel identification confidence level remains at 0.86 to 0.90 as set in step S2. For non-designated operating axes, the maximum axial fluctuation value of each non-designated axis in the locked state is obtained through a controlled equipment static locking test, and this fluctuation value is used as the natural fluctuation benchmark. In actual operation, if the actual displacement of a non-designated operating axis exceeds this natural fluctuation benchmark, it is recorded as a non-designated axis displacement. No artificially set small values ​​are used to mask this, thus preventing the safety hazard of ignoring abnormal axis movement due to improper threshold settings.

[0090] Through the above processing, a real-time jog verification sequence is obtained, which includes the jog safety envelope constraint, the actual movement of the controlled equipment, and the occupancy status of the personnel area. This sequence is then used by subsequent sub-steps to determine the travel and duration limits during the pressing process.

[0091] S4.2: Determine if the pressing timeout, stroke limit, or out-of-bounds.

[0092] After obtaining the real-time jog verification sequence, dual limits are imposed on the cumulative stroke and continuous compression duration during the compression period.

[0093] Specifically, the current cumulative jog stroke calculated in sub-step S4.1 is... Single permitted travel in the inching safety envelope Comparison. Based on single-trip allowable journeys. The cumulative travel distance is defined as 92% to 96% as the starting point of the critical deceleration zone. Entering this range signifies entering the critical region. This range allows for a small lead time for control command refresh and mechanical response, while preventing premature deceleration from significantly shortening the effective range of normal jogging travel. Once the cumulative travel enters the critical deceleration zone, the control logic no longer allows the controlled equipment to further increase its speed and gradually reduces the increment corresponding to subsequent jogging commands, allowing the controlled equipment to decelerate naturally as it approaches the boundary. When the cumulative travel reaches the single-stroke allowable travel distance... Even if the enable device and the jog button remain pressed, the additional jog travel will be set to zero, and the command to continue pressing will no longer be responded to.

[0094] Simultaneously, the timing begins from the moment the button changes from an unpressed state to a pressed state, calculating the duration of continuous pressing. The single continuous pressing time limit is set to 2.7 to 3.3 seconds. This time range can accommodate a single intentional observational pressing operation while preventing the button from remaining pressed continuously and forming uncontrolled continuous movement. When the continuous pressing duration reaches this time limit, the increasing pressing distance stops, and a pressing timeout state is entered.

[0095] During the verification process, if the actual speed exceeds the permitted speed limit... If the actual direction of motion is opposite to the permissible direction specified in the jog safety envelope, the operator leaves the observation area, a non-designated operating axis is displaced, or other personnel appear in the work area, the normal travel limit branch will no longer be entered, but will be passed to the subsequent sub-step as the basis for boundary judgment.

[0096] S4.3: Determine button sticking, controlled inertial braking, and inertial overtravel.

[0097] After the operator releases the jog button, it is determined whether the controlled equipment can stop normally and whether there is any abnormal movement state.

[0098] Specifically, when the AI ​​image recognition result indicates that the operator's finger has left the jog button, the pressing electrical signal status output by the handheld operating terminal is simultaneously verified. The finger-off state needs to be maintained continuously for 80 to 100 milliseconds before confirmation. This time range covers short-term recognition interruptions caused by brief finger obstruction or being blocked by the handheld operating terminal's casing in continuous images, while avoiding excessive delays that could hinder the identification of button sticking risks. If the finger has been confirmed to have left the jog button, but the handheld operating terminal continues to output a pressing status signal, a button sticking judgment result is generated, indicating that the jog button may have mechanical jamming or an electrical fault.

[0099] If the finger is removed and the handheld operating terminal's pressing status signal is simultaneously released, but the controlled device still has residual motion speed, then the remaining braking margin is calculated starting from the moment the pressing is released. For the nth acquisition moment, the remaining braking margin is calculated according to the following formula, based on the same kinematic model as in step S3.3: ; in, This represents the remaining braking margin at the nth acquisition time. The remaining braking margin in the jog safety envelope at the moment of release is the actual allowable stroke in step S3.3. and The difference between them is the part that has not been consumed when the press is released; r is the collection time number corresponding to the press release time; the summation term is the actual movement distance that has occurred from the (r+1)th time after the press is released to the current nth time, which is calculated using the same trapezoidal integral method as sub-step S4.1; The remaining speed at the current acquisition moment; The braking response time is measured in step S3.3; For effective braking deceleration, the two terms in parentheses at the end of the formula represent the distance the vehicle would continue to move during the braking response period and the braking distance required for the deceleration phase if braking were triggered immediately from the current moment, given the current remaining speed. The sum of these two terms is the total braking distance required to bring the vehicle to a complete stop from the current speed.

[0100] If this value is always greater than or equal to zero, it indicates that the controlled equipment can come to a complete stop within the remaining safe space under the existing braking capacity, and is judged as controlled inertial braking. After the speed of the controlled equipment has dropped to within the stationary speed limit and all moving axes have stopped, the final stopping position is verified. If the final stopping position exceeds the boundary of the inching safety envelope, or the actual displacement of any non-specified operating axis exceeds its natural fluctuation reference, it is judged as inertial overtravel, indicating that the existing braking distance or braking effect is insufficient to stop the controlled equipment within the safety envelope. If the speed does not continue to decrease as expected during deceleration, it is also considered abnormal and falls under the trigger condition of inertial overtravel. The button sticking judgment result and the inertial overtravel judgment result are recorded independently so that subsequent steps can take differentiated control strategies for different risk types.

[0101] S4.4: Summarize the verification and judgment results and output the corresponding jog risk control status.

[0102] After determining the travel and duration limits during compression and the residual motion characteristics after compression is released, all verification results are summarized to form a unified jogging risk control status.

[0103] Specifically, the following verification results are summarized: whether the specified operating axis conforms to the axis number specified in the jog safety envelope, whether the actual direction of motion conforms to the specified permissible direction, and whether the actual speed exceeds the permissible speed limit. Cumulative Trip Does it exceed the single-trip allowance? Check whether the duration of continuous pressing exceeds the single operation limit, whether the operator remains within the safe observation area, whether there are other personnel in the work area, whether the buttons are stuck together, whether the controlled equipment has inertial overtravel, and whether there is non-specified axis displacement.

[0104] Based on the summarized results of the above items, the corresponding jogging risk control status will be output according to the following rules: When there is no press timeout, out-of-bounds, button sticking, controlled inertial braking, or inertial overtravel, the output is in normal jogging mode. When a critical travel threshold is reached, a critical stop state is output. When controlled inertial braking occurs, output the controlled inertial braking state; When any of the following situations occur, such as button sticking, inertia overtravel, exceeding the limit, or other personnel entering, an "exceeding the limit danger" status will be output.

[0105] Write the following detailed records into the output jog risk control status: risk category label, time of risk occurrence, axis number of actual displacement, actual direction of motion, actual speed value, and cumulative stroke. Remaining braking margin And the status of personnel occupancy. Step S5 will implement a tiered control strategy based on the different risk control statuses mentioned above.

[0106] S5: Execute the hierarchical control strategy according to the jog risk control status, and allow re-access verification after the controlled equipment stops and the on-site conditions are reset, and output the safety protection result.

[0107] S5.1: Generate corresponding hierarchical control instructions based on the jogging risk control status.

[0108] Read the jog risk control status output in step S4, and execute the corresponding control strategy according to the risk category label it carries.

[0109] For risk control states marked as normal jogging, maintain jogging permission for the specified operating axis and permitted direction, and the controlled equipment operates at the current speed limit. Continue performing the inching operation without further intervention.

[0110] For risk control states marked as critical stop, it indicates that the controlled equipment's current cumulative travel or speed is approaching the boundary of the inching safety envelope, requiring speed limiting. In this case, the remaining permissible travel in the inching safety envelope is read. and current remaining braking margin The current maximum speed limit to be issued should be calculated using the following formula: ; in, This represents the current maximum speed for issuing data. The maximum permissible speed limit specified for the inching safety envelope; The remaining permitted trips, i.e., single-trip permitted trips. Subtract the cumulative trips already taken ; This is the allowable travel distance for a single inching motion; The current remaining braking margin is calculated in real time by sub-step S4.3; This represents the initial braking margin at the start of this inching motion; This is the percentage of travel deceleration trigger. The braking margin deceleration trigger ratio is set to 0.18 to 0.22. This ratio means that continuous deceleration is triggered when the remaining travel or remaining braking margin drops to about one-fifth of the initial value. and The dimensionless speed variation index is set to 0.9 to 1.1. An index close to 1 helps prevent sudden speed command changes that could cause a shock. The above formula shows that the actual speed limit is determined by the more stringent of the remaining travel condition and the remaining braking margin condition. When the remaining travel or remaining braking margin is sufficient, the ratio within the parentheses is greater than or equal to 1. Keep as As the remaining amount gradually decreases, the ratio decreases accordingly. Continuous descent gradually decelerates the controlled equipment before reaching the safety envelope boundary. When any remaining quantity is not greater than zero, the ratio within the parentheses is zero, and the formula is not executed again; a stop command is generated directly. This continuous deceleration mechanism avoids the possibility of workpiece detachment or mechanical impact caused by sudden braking only upon reaching the boundary.

[0111] For risk control states marked as controlled inertial braking, it indicates that the operator has released the jog button and the controlled equipment is in the process of natural deceleration. At this time, the current braking monitoring is maintained, no new speed command is added, and the existing braking process is not canceled. Wait for the controlled equipment to stop naturally.

[0112] For risk control states marked as "out of bounds danger", it indicates that serious abnormalities have occurred, such as button sticking, inertial overtravel, non-designated operating axis linkage, operator leaving the designated position or other personnel entering. At this time, a control command to revoke the jog permission and trigger emergency braking is directly generated.

[0113] S5.2: Execute the corresponding physical control operation according to the hierarchical control command.

[0114] After receiving the graded security control instructions, the corresponding physical control operations are executed according to the instruction type.

[0115] Specifically, for control commands corresponding to normal jogging states, only the specified operating axis is allowed to move in the prescribed permissible direction, with the speed limit set at the currently issued speed. All other axes remain locked.

[0116] For the control commands corresponding to the critical stop state, according to The actual speed is gradually reduced, and the cumulative travel distance is... Reaching a single permitted trip Before the boundary, the momentum of the newly added point is gradually reduced to zero, eventually causing the controlled device to come to a complete stop at the boundary.

[0117] For control commands corresponding to controlled inertial braking states, immediately revoke subsequent jogging permissions while maintaining the existing braking process and continuously monitoring the remaining braking margin. Until all motion axes come to a stop.

[0118] For control commands corresponding to boundary violation hazards, immediately set the jog speed command to zero, revoke the current jog permission of the handheld operating terminal, trigger the safety braking of the controlled equipment, and write a lockout flag. Once the lockout flag is formed, even if the operator presses the enable device or jog button again, the controlled equipment must not resume any movement and must wait for subsequent sub-steps to complete stop verification and reset confirmation before regaining access permission.

[0119] During the execution of the above control operations, the following execution status information is recorded: the type of reason for triggering the control command, the axis number where the dangerous displacement occurred, the time when braking started, the speed change curve of each motion axis during the braking process, the coordinates of the final stopping position, and whether the final stopping position exceeds the jog safety envelope boundary.

[0120] S5.3: After braking or locking is completed, verify the stopping stability of all moving shafts; after the stopping stability is qualified, confirm that the site conditions meet the re-access requirements and form a re-access permit.

[0121] After completing the braking or locking operation, the stopped status of the controlled equipment and the site conditions are verified to confirm whether the requirements for re-access are met.

[0122] Specifically, for controlled equipment that has undergone braking or locking operations, the stopping stability of all moving axes is first verified. The verification process is as follows: During a continuous verification period, it is checked whether the maximum actual speed of each moving axis is within the static speed limit, and simultaneously, it is checked whether the maximum position change of each axis is within the static position fluctuation limit. The verification period is set to 0.45 to 0.55 seconds, which is long enough to eliminate the interference of accidental fluctuations on the stopping determination. The static speed limit and static position fluctuation limit are determined jointly by the controlled equipment's axis locking static test and the controller resolution. These are not manually set parameters and can truly reflect the normal fluctuation range of the controlled equipment in a static state. When all axes continuously meet the above two limit requirements during the verification period, the stopping stability is considered to have reached the qualified standard.

[0123] After the stability threshold is reached, it is necessary to confirm that the site conditions meet the requirements for re-access, specifically including the following four items: the operator has re-entered the safe observation area determined in step S3; all personnel other than the bound personnel have exited the work area; the jog button on the handheld operating terminal has been released and is no longer in a pressed state; the enable device on the handheld operating terminal is in the fully released position, and the dedicated reset signal provided by the handheld operating terminal controller has been successfully recognized by the system. This reset signal is triggered by the operator performing a manual reset operation on the handheld operating terminal, which is achieved by simultaneously pressing the enable device and a specific function button or operating the dedicated reset switch. Its electrical signal characteristics are distinguishable from the normal enable press state. If any of the above four conditions are not met, the locked state remains unchanged, and no recovery operation is performed. Only when all four conditions are met is re-access permission granted.

[0124] It should be noted that re-access permission only indicates that on-site conditions allow for restarting a new jog operation; it does not directly restore the movement of the controlled equipment. The controlled equipment must undergo the complete access verification process in step S2 to resume movement.

[0125] S5.4: Return the re-access permission to step S2 and output the security protection result of the corresponding branch.

[0126] After completing the hierarchical control execution and stop status verification, the corresponding security protection results are output according to different branches.

[0127] Specifically, for the normal jog branch, output the currently valid specified operation axis number, allowed motion direction, and current allowed speed limit. Remaining allowed travel time The operator's position and stance indicate that the jog operation is proceeding normally.

[0128] For the critical stop branch, the output records the deceleration process and the result of the controlled device coming to a complete stop before the boundary, indicating that the system achieves a safe stop by continuously decelerating when approaching the boundary without triggering emergency braking.

[0129] For the controlled inertial braking branch, the output shows the result of the controlled device decelerating and stopping normally after being pressed and released, as well as the remaining braking margin during the entire deceleration process. The change records indicate that the inertial braking process is under control.

[0130] For dangerous branches that cross the boundary, the system outputs detailed information on the cause of the danger, the result of the emergency braking, the stopping stability of each axis, and the locking status, indicating that the system has taken the most stringent safety measures when the abnormality occurs.

[0131] Once the re-access permission is granted in step S5.3, this permission, along with the stop verification result, is returned to step S2 as the trigger condition for re-executing access verification in step S2. At this time, step S2 will re-verify the unique access personnel, the handheld operating terminal carrying relationship, the observation station position, and the enabled status based on the latest sensing data, without inheriting any jogging permissions from before the locking mechanism was established. Through this closed-loop design, each jogging operation begins with an independent access verification, ensuring that the security of each jogging operation is guaranteed based on the latest on-site conditions.

[0132] The final output of the industrial controlled equipment jogging and restricted motion safety protection results includes normal jogging process records, critical deceleration and stop process records, controlled inertial braking process records, dangerous lockout and emergency braking process records, and re-access closed-loop records, thus forming a complete jogging safety protection system that allows restricted jogging, critical deceleration, dangerous braking, lockout maintenance and re-access to be interconnected.

[0133] According to embodiments of this disclosure, an electronic device is also provided, which may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor can invoke logical instructions stored in the memory to execute the methods provided in the above embodiments.

[0134] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, this disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

Claims

1. A method for safe maintenance of equipment that integrates mobile bridge signal and AI image behavior analysis, characterized in that, The method includes: S1: Collect mobile bridge signals, images and controlled device status data and unify their temporal and spatial coordinates, bind the operator's tag, handheld operating terminal number and the current debugging operation task to form a point-motion sensing benchmark; S2: Based on the aforementioned point-and-click sensing benchmark, perform access verification of personnel uniqueness, handheld operating terminal carrying relationship, observation position, and enabling status. Upon successful verification, a point-and-click access status is established; S2 includes: S2.1: Read the point-sensing reference and verify whether the personnel and tags are consistent through cross-comparison of artificial intelligence image recognition and mobile bridge positioning; when no other tags or unbound personnel appear in the entire observation window, and based on the cross-comparison results, when the correspondence between the bound personnel and the tags is stable and consistent in the entire observation window, the personnel uniqueness verification is determined to be passed. S2.2: During the observation period when the personnel uniqueness verification is qualified, the position information of the handheld operating terminal and the holding hand is extracted by artificial intelligence image recognition. Based on the spatial correlation and motion consistency between the personnel and the handheld operating terminal, it is determined whether the personnel and the handheld operating terminal have passed the continuous carrying verification. S2.3: Verify the operator's position space compliance through artificial intelligence image recognition, and whether the enabling device is in the middle effective pressing position, to form a joint verification result of position and enabling; S2.4: When the personnel uniqueness verification, the continuous carrying verification of the handheld operating terminal, and the joint verification of the station position and enable are all passed, the personnel-carrying point access state is formed; S3: In the jog entry state with people, determine the safe observation area, the restricted motion area of ​​the controlled equipment, and the motion parameters under braking constraints, and solidify the specified operating axis, allowable speed, and allowable stroke as the jog safety envelope; S4: Under the constraint of the inching safety envelope, the actual motion state of the controlled equipment and the operator's position are verified in real time, and the corresponding inching risk control status is output. S5: Execute the hierarchical control strategy according to the jog risk control status, and allow re-access verification after the controlled equipment stops and the on-site conditions are reset, and output the safety protection result.

2. The method according to claim 1, characterized in that, S1 includes: S1.1: Record the field status information synchronously from the mobile bridge system, image acquisition device and controlled equipment controller according to a unified acquisition cycle to obtain a multi-source point dynamic observation sequence; S1.2: Using the origin of the controlled equipment base coordinate system as a unified reference origin, the image pixel coordinates are transformed to the controlled equipment base coordinate system using calibration points. The candidate area location of the personnel carrying the tag is determined according to the coordinates of the mobile bridge node and the tag signal strength. For each acquisition time in the multi-source point motion observation sequence, the spatial consistency between the operator's position determined by the image and the candidate area location determined by the bridge signal, as well as the temporal consistency between the acquisition times of various information, are verified. The observation data corresponding to the time when both spatial consistency and temporal consistency reach the preset threshold are included in the unified point motion observation chain. S1.3: Based on the unified point-motion observation chain, according to the spatial proximity and motion synchronization between each tag-carrying person and each handheld operating terminal within each effective acquisition time, determine whether the binding credibility between the person and the handheld operating terminal reaches a preset threshold. When the preset threshold is reached, a candidate binding relationship containing the identity identifier and task configuration information of the bound object and the specified operating axis is formed. S1.4: Perform a uniqueness check on the candidate binding relationships, retain the combination with the highest binding confidence and solidify it to obtain the point motion perception benchmark.

3. The method according to claim 1, characterized in that, S3 includes: S3.1: Use artificial intelligence image recognition to extract key points of the operator's human posture, determine the dynamic extension radius based on the human posture and motion parameters, and extend along the direction pointing to the controlled equipment based on the operator's body outline to form the operator's dynamic occupancy boundary. S3.2: Overlay and analyze the dynamic occupation boundary of the operator with the specified observation station area, delete the positions with intrusion risk to obtain the safe observation area, and generate candidate restricted motion areas of the controlled equipment by advancing step by step according to the specified operating axis and allowed motion direction. Calculate the minimum distance from the dynamic occupation boundary of the operator to the leading edge of the end effector's motion direction as the available human-machine separation distance, and the available path length from the current position of the end effector along the specified motion direction to the tooling boundary. S3.3: Obtain the braking parameters corresponding to the current debugging operation task based on the pre-implemented braking test, and determine the maximum stroke of a single jog based on the available human-machine separation distance, the available path length of the tooling boundary and the braking parameters; S3.4: Based on the maximum stroke of a single jog, select the actual allowable stroke that does not exceed the maximum stroke, and reverse the calculation of the upper limit of the allowable speed; solidify the specified operating axis, the allowable motion direction, the spatial constraint information and the motion constraint information together to form the jog safety envelope.

4. The method according to claim 3, characterized in that, The spatial constraint information includes the coordinates of the safe observation area, the dynamic occupancy boundary of personnel, and the restricted movement area of ​​the controlled equipment; the braking parameters include braking response time and braking deceleration parameters; the motion constraint information includes the single allowable travel distance, the upper limit of allowable speed, braking response time, and braking deceleration parameters.

5. The method according to claim 1, characterized in that, S4 includes: S4.1: Read the jog safety envelope, continuously collect the real-time status of the controlled device when the enabling device enters the effective pressing state and triggers the jog button, calculate the cumulative jog stroke segment by segment based on the actual speed of the end effector along the specified axis direction, and verify whether the operator remains within the safe observation area. S4.2: When the cumulative travel reaches the single-operation allowable travel, the new travel is set to zero; at the same time, the timer starts from the moment the jog button is pressed, and when the continuous pressing time reaches the single operation limit, the increase of travel stops and the pressing timeout is determined; when the cumulative travel enters the critical area with the single-operation allowable travel as the upper limit, the travel is determined to be critical; when the actual movement direction is opposite to the allowable direction, the operator leaves the observation area, the non-designated operating axis is displaced, or other personnel enter, the boundary is determined to be exceeded. S4.3: If the pressing signal continues after the operator releases the jog button, it is determined that the button is stuck; if the pressing signal has been released but the controlled equipment still has residual speed, the remaining braking margin is calculated from the moment the button is released. When the remaining braking margin always meets the braking distance required for complete stop and the speed drops to within the stationary limit, it is determined to be controlled inertial braking. When the final stopping position exceeds the jog safety envelope boundary, or the displacement of the non-specified operating axis exceeds the natural fluctuation reference, it is determined to be inertial overtravel. S4.4: Summarize the above verification and judgment results, and output the corresponding jog risk control status.

6. The method according to claim 5, characterized in that, The summary of the above verification and judgment results, and the corresponding jog risk control status output, include: When there is no press timeout, out-of-bounds, button sticking, controlled inertial braking, or inertial overtravel, the output is in normal jogging mode. When a critical travel threshold is reached, a critical stop state is output. When controlled inertial braking occurs, output the controlled inertial braking state; When any of the following occurs: key sticking, inertia overtravel, or exceeding the limit, an out-of-limit danger status will be output.

7. The method according to claim 1, characterized in that, S5 includes: S5.1: Generate corresponding graded control instructions based on the jogging risk control status. For normal jogging status, maintain the current speed limit and continue to perform jogging operation. For critical stop status, gradually reduce the speed limit according to the remaining allowable travel and the current remaining braking margin to stop the controlled equipment before the boundary. For controlled inertial braking status, maintain braking monitoring and wait for natural stopping. For dangerous out-of-bounds status, directly revoke the jogging permission and trigger emergency braking. S5.2: Execute the corresponding physical control operation according to the hierarchical control command, wherein a lockout flag is written in the boundary crossing danger state; S5.3: After braking or locking is completed, verify the stopping stability of all moving shafts; after the stopping stability is qualified, confirm that the site conditions meet the re-access requirements and form a re-access permit; S5.4: Use the re-access permission as the trigger condition for returning to step S2, and output the security protection result of the corresponding branch.

8. An electronic device, characterized in that, The electronic device includes a memory and at least one processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1-7.

9. A computer storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1-7.

Images