Equipment operation behavior identification method and system based on attitude and spatial position relation

Abstract

The invention discloses an equipment operation behavior identification method and system based on a relationship between a posture and a spatial position, and is used for solving the technical problem that the current equipment operation behavior identification method ignores the influence of environment context information and excessively depends on the precision of a posture detector, so that the identification precision is poor. The method comprises the following steps: acquiring equipment non-operation and operation video clips, and preprocessing to output equipment non-operation static images and equipment operation image frames; then detecting the device by using a YOLOv11 target detection model, and outputting a non-operation detection result of the device; generating a plurality of target human hand key point coordinates on the basis of an mmpose attitude estimation model and a regression function; determining a hand posture feature vector; and finally, a double-layer heterogeneous model stacking integrated structure is adopted, the hand posture feature vector and an equipment position feature vector in an equipment non-operation detection result are fused for recognition, and an equipment operation behavior recognition result is output.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning technology, and in particular to a method and system for recognizing device operation behavior based on the relationship between posture and spatial position. Background Technology

[0002] In the fields of computer vision and machine learning, human-object interaction (HOI) detection has always been an important and complex research area. Traditional methods typically treat people as a whole for identification, relying on appearance features and coarse spatial information to infer the interaction between people and objects. While this approach can provide some results in simple scenarios, it falls short when faced with complex interactions. Specifically, traditional methods often indiscriminately detect all objects in a scene, ignoring precise interaction information between people and objects. This not only increases the computational burden but also significantly reduces system efficiency. Furthermore, these methods over-rely on human appearance features, neglecting skeletal details related to specific actions, resulting in low recognition accuracy and difficulty in handling fine-grained interaction scenarios.

[0003] In practical applications, simply recognizing people and objects is insufficient for a comprehensive understanding of a scene's meaning. To achieve a deep understanding of complex environments, it is crucial to identify the interaction relationships between people and objects. As a high-level visual understanding task, behavior recognition requires systems not only to accurately locate instances of people and objects but also to understand the specific actions and interaction relationships between them. The core challenge of this task lies in how to efficiently identify the interactions between people and objects and correctly analyze the details of those actions.

[0004] In the current field of device operation behavior recognition, motion recognition methods based on human skeletons are widely used. However, this method ignores the influence of environmental context information. In real-world scenarios, the same set of limb movements often correspond to completely different meanings of device operation. Simply relying on human skeleton data cannot capture such contextual differences that are strongly associated with the environment. At the same time, this method is overly dependent on the accuracy of the posture detector. If the quality of human key points is poor or missing, it will seriously affect the recognition performance and lead to poor recognition accuracy. Summary of the Invention

[0005] This invention provides a device operation behavior recognition method and system based on the relationship between posture and spatial position, which solves the technical problem that current device operation behavior recognition methods ignore the influence of environmental context information and rely too much on the accuracy of posture detectors, resulting in poor recognition accuracy.

[0006] The first aspect of this invention provides a method for recognizing device operation behavior based on the relationship between posture and spatial position, comprising:

[0007] Acquire video clips of the device not being operated and video clips of the device being operated, preprocess the video clips of the device not being operated and the video clips of the device being operated, and output static images of the device not being operated and image frames of the device being operated.

[0008] The YOLOv11 target detection model is used to detect static images of the device when it is not in operation, and the detection results of the device not being in operation are output.

[0009] The mmpose pose estimation model and regression function are used to generate the coordinates of multiple key points of the target human hand based on the image frames of the device operation;

[0010] Based on the coordinates of the key points of the target human hand, the hand posture feature vector is determined;

[0011] A dual-layer heterogeneous model stacking and integration structure is adopted to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device non-operation detection result, and output the device operation behavior identification result.

[0012] Optionally, the preprocessing of the video clips showing the device not being operated and the video clips showing the device being operated, to output static images of the device not being operated and image frames of the device being operated, includes:

[0013] The video clips of the device not being operated are extracted and labeled to generate a static image of the device not being operated;

[0014] The device operation video clips are converted into an image frame sequence, and the image frame sequence is filtered and denoised to generate device operation image frames.

[0015] Optionally, the step of generating multiple target human hand key point coordinates based on the device operation image frame using the mmpose pose estimation model and regression function includes:

[0016] The mmpose pose estimation model is used to estimate the pose of the device operation image frame, and outputs multiple initial human hand key point coordinates and confidence scores corresponding to each initial human hand key point coordinate.

[0017] Each confidence score is compared with a preset confidence threshold.

[0018] Use the initial coordinates of the human hand key points corresponding to any confidence score greater than the preset confidence threshold as the coordinates of the target human hand key points.

[0019] The regression function is used to correct the initial human hand key point coordinates corresponding to any confidence score less than or equal to the preset confidence threshold, and the target human hand key point coordinates are output.

[0020] Optionally, determining the hand pose feature vector based on the coordinates of key points of each target human hand includes:

[0021] The motion trend factor corresponding to the coordinates of the key points of the hand of each target human body is calculated using the average velocity vector corresponding to the coordinates of the key points of the hand of each target human body.

[0022] Based on the motion trend factor corresponding to the coordinates of the key points of the hand of each target human body, calculate the dynamic weight corresponding to the coordinates of the key points of the hand of each target human body.

[0023] The hand posture feature vector is output by performing a weighted average calculation based on the coordinates of the key points of the target human hand and the dynamic weights corresponding to the coordinates of the key points of the target human hand.

[0024] Optionally, the dual-layer heterogeneous model stacked ensemble structure includes a meta-learner and multiple base learners; the dual-layer heterogeneous model stacked ensemble structure identifies the device operation behavior based on the hand pose feature vector and the device position feature vector in the device non-operation detection result, and outputs the device operation behavior recognition result, including:

[0025] The hand posture feature vector and the device position feature vector are concatenated to output a fused feature matrix;

[0026] The fused feature matrix is ​​used as the input to each meta-learner, and multiple prediction probability vectors are output.

[0027] The predicted probability vectors are concatenated to output the meta-feature vector;

[0028] The meta-feature vector is used as the input to the meta-learner to generate device operation behavior recognition results.

[0029] Optionally, the training process of the YOLOv11 object detection model is as follows:

[0030] Acquire a video clip of the device not being operated for model training, preprocess the video clip of the device not being operated for model training, and output a static image of the device not being operated for model training.

[0031] The initial detection model of YOLOv11 is used to perform detection based on the static images of no device operation used for model training, and outputs the detection results of no device operation used for model training.

[0032] The YOLOv11 initial detection model is trained using a preset loss function based on the device inactivity detection results used for model training, and the trained YOLOv11 object detection model is output.

[0033] The preset loss function is specifically as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] in, The loss value corresponding to the preset loss function; The weight coefficients corresponding to the loss values ​​of the bounding box are located; Locate the loss value for the bounding box; The weight coefficient corresponding to the dynamic weight loss value of the center point; The dynamic weight loss value for the center point; The focusing coefficient is non-monotonic; The base distance loss value; The intersection-union ratio (IUU) is the ratio of the predicted device bounding box to the ground truth device bounding box in the device inactivity detection results used for model training. It measures the degree of overlap between the two boxes. It is the natural logarithm function; The x-coordinate of the center coordinate of the device prediction box is the x-coordinate of the device position feature vector in the device inactive detection result used for model training. The x-coordinate of the center of the actual frame of the device; The ordinate of the center coordinate of the device prediction box is the ordinate of the device position feature vector in the device inactive detection result used for model training. The ordinate of the center coordinate of the actual frame of the device; , These are the width and height of the smallest bounding box that simultaneously contains both the predicted and ground truth boxes; As a quality assessment metric for dynamic bounding boxes; This is a hyperparameter used to control the degree of non-monotonic focusing; for; The total number of static images not processed by the device used for model training; and the total number of positive samples. Let be the prediction confidence in the detection result of device inaction for model training corresponding to the i-th static image of device inaction used for model training. These are the parameters used for dynamic decay weights; Let x be the x-coordinate of the device location feature vector corresponding to the i-th static image that the device did not operate on during model training; Let x be the x-coordinate of the center of the real frame of the i-th device; Let y be the ordinate of the device location feature vector corresponding to the i-th static image that is not operated by the device used for model training; The ordinate of the center coordinate of the i-th device's real frame; , Let be the standard deviation of the Gaussian kernel function. , , These are the width and height of the actual device frame, respectively. This is a preset scaling factor.

[0040] A second aspect of the present invention provides a device operation behavior recognition system based on the relationship between posture and spatial position, comprising:

[0041] The acquisition module is used to acquire video clips of the device not being operated and video clips of the device being operated, and to preprocess the video clips of the device not being operated and video clips of the device being operated, and to output static images of the device not being operated and image frames of the device being operated.

[0042] The detection module is used to detect static images of the device when it is not being operated on using the YOLOv11 target detection model, and output the detection result of the device not being operated.

[0043] The generation module is used to generate multiple target human hand key point coordinates based on the device operation image frames using the mmpose pose estimation model and regression function;

[0044] The determination module is used to determine the hand posture feature vector based on the coordinates of key points of the hand of each target human body;

[0045] The recognition module is used to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device non-operation detection result using a two-layer heterogeneous model stacked integrated structure, and output the device operation behavior recognition result.

[0046] A computer device provided in a third aspect of the present invention includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the preceding claims.

[0047] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the preceding claims.

[0048] The fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the preceding claims.

[0049] As can be seen from the above technical solutions, the present invention has the following advantages:

[0050] The above-mentioned technical solution of the present invention provides a device operation behavior recognition method based on the relationship between posture and spatial position. It acquires video clips of device inactivity and video clips of device operation, preprocesses these clips, and outputs static images of device inactivity and image frames of device operation. A YOLOv11 target detection model is used to detect the static images of device inactivity, outputting the detection result. An mmpose posture estimation model and regression function are used to generate coordinates of multiple target human hand key points based on the device operation image frames. Based on the coordinates of each target human hand key point, a hand posture feature vector is determined. A two-layer heterogeneous model stacking and integration structure is used to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device inactivity detection result, outputting the device operation behavior recognition result. Based on the above solution, the present invention incorporates the environmental context information of device spatial position into the recognition framework, compensating for the neglect of environmental information in traditional methods. Simultaneously, the regression function supplements potentially missing hand key points, reducing the over-reliance on the accuracy of the posture detector. The synergistic effect of both effectively improves the recognition accuracy of device operation behavior in complex scenarios. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the steps of a device operation behavior recognition method based on the relationship between posture and spatial position, as provided in Embodiment 1 of the present invention.

[0053] Figure 2 This is an overall framework diagram of a device operation behavior recognition method based on the relationship between posture and spatial position provided in Embodiment 1 of the present invention;

[0054] Figure 3This is a flowchart illustrating the training process of the YOLOv11 object detection model provided in Embodiment 2 of the present invention.

[0055] Figure 4 This is a structural block diagram of a device operation behavior recognition system based on the relationship between posture and spatial position, provided in Embodiment 3 of the present invention. Detailed Implementation

[0056] This invention provides a method and system for recognizing device operation behavior based on the relationship between posture and spatial position, which solves the technical problem that current device operation behavior recognition methods ignore the influence of environmental context information and rely too much on the accuracy of posture detectors, resulting in poor recognition accuracy.

[0057] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0058] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a device operation behavior recognition method based on the relationship between posture and spatial position, as provided in Embodiment 1 of the present invention.

[0059] This invention provides a method for recognizing device operation behavior based on the relationship between posture and spatial position, comprising:

[0060] Step 101: Obtain video clips of the device not being operated and video clips of the device being operated, and preprocess the video clips of the device not being operated and video clips of the device being operated to output static images of the device not being operated and image frames of the device being operated.

[0061] Video footage of equipment inactive: This refers to video footage captured by a camera when the equipment is idle, without any personnel contact or operation, and is used to extract the equipment's reference spatial position.

[0062] Equipment operation video clips: These are video clips taken by a camera when an operator performs specific actions on the equipment (such as debugging or maintenance), used to capture dynamic information of the human operation process.

[0063] Static images of devices not in operation: These are clear single-frame images extracted from video clips of devices not in operation. They contain the complete static form of the device and are used for training or detection of device localization models.

[0064] Device operation image frames: These refer to a continuous sequence of image frames converted from video clips of device operation. They record the dynamic process of a human operating the device and are used to extract human posture features.

[0065] It should be noted that fixed cameras are deployed in the designated testing area (such as a high-speed rail maintenance workshop). Video clips of the equipment inactive state are captured when it is idle (no personnel contact or operation), while video clips of the equipment in operation are captured when personnel perform specific actions on the equipment (such as debugging, maintenance, start-up, or shutdown). Both types of videos must cover different lighting conditions, different equipment states, and different operating scenarios to ensure data diversity. The acquired video clips undergo targeted preprocessing to obtain static images of the equipment inactive state and image frames of the equipment in operation.

[0066] Specifically, step 101 may include the following sub-steps:

[0067] S11. Extract and annotate video clips of the device not being operated to generate a static image of the device not being operated;

[0068] S12. Convert the device operation video clips into an image frame sequence, and perform filtering and noise reduction processing on the image frame sequence to generate device operation image frames.

[0069] It should be noted that for video clips where the device is not in operation, frame analysis tools are first used to filter out stable clips where the device is unobstructed and the image is free of motion blur. Then, FFmpeg is used to extract single-frame images from these clips as base material. Subsequently, Label Studio is used to draw bounding boxes on the device targets in the images, annotating the device's category information and its position coordinates in the image coordinate system. This generates static images of the device in operation that include spatial location annotations. For video clips where the device is in operation, FFmpeg is first used to convert them into a continuous sequence of image frames at a preset frame rate (e.g., 25 frames / second) to ensure a complete record of the dynamic process of the operator's interaction with the device. Then, the image frame sequence is processed using a Gaussian filtering algorithm to smooth out abrupt changes in pixel grayscale and weaken environmental noise (such as uneven lighting and background noise). Finally, clear images of the device in operation that clearly present the details of human operation are generated, laying the data foundation for subsequent posture feature extraction.

[0070] In this embodiment, a fixed camera is installed at a predetermined testing location (e.g., a high-speed rail maintenance workshop). Video clips are recorded when the operator is not operating the equipment. These video clips are then extracted into still images, and the target equipment in the images is precisely bounded by a bounding box using Label Studio to obtain the equipment's precise spatial location information, ultimately resulting in a still image of the equipment not being operated. The video clips of equipment operation are preprocessed by transcoding the video using FFmpeg (Fast Forward Moving Picture Experts Group) and converting it into a fixed frame rate sequence of images. This ensures that each frame in the video stream has sufficient information for subsequent processing. The image frames are then filtered and denoised to obtain the equipment operation image frames.

[0071] Step 102: Use the YOLOv11 target detection model to detect static images of devices that have not been operated, and output the detection results of devices not being operated.

[0072] YOLOv11 target detection model: refers to the 11th generation target detection algorithm model in the You Only Look Once series. It is characterized by single-stage detection and strong real-time performance, and can quickly identify targets in images and output their location, category and confidence information.

[0073] The results of the equipment not being operated include the equipment location feature vector (i.e., the center coordinates of the equipment prediction box), confidence score, equipment prediction box, and equipment category label.

[0074] It should be noted that for device frame-level target detection and extraction of device spatial coordinate features, this invention utilizes the YOLOv11 target detection model to process pre-processed image frames to identify and locate specific devices. Specifically, the pre-processed static image of a device not in operation is input into a pre-trained YOLOv11 target detection model. The model extracts device features from the image through the backbone network, fuses multi-scale features through the neck network, and outputs predicted device target information from the head network. This includes the bounding box coordinates of the device in the image (i.e., the center coordinates of the predicted device box), the device category label, and the corresponding confidence score. This forms a device in-operation detection result containing the device's spatial location (bounding box center coordinates) and category attributes, providing basic spatial information about the device environment for subsequent fusion with human pose features.

[0075] Step 103: Using the mmpose pose estimation model and regression function, generate the coordinates of multiple target human hand key points based on the device operation image frames.

[0076] mmpose pose estimation model (OpenMMLab Pose Estimation): refers to a pose estimation tool library based on an open-source deep learning framework. It can accurately identify the position coordinates of key human body parts in an image and output the confidence level of the recognition results. It is suitable for human pose feature extraction scenarios.

[0077] Regression function: Here, it specifically refers to the multinomial regression function, which is used to infer the coordinates of missing hand key points in the current frame based on the complete key point data of the preceding and following frames through temporal correlation, so as to make up for the shortcomings of single-frame detection.

[0078] The coordinates of multiple target human hand key points include the target image coordinate system positions of the left wrist, right wrist, left elbow, and right elbow.

[0079] It should be noted that this invention inputs the preprocessed device operation image frames frame by frame into the mmpose pose estimation model. The model extracts features and locates key points, outputting the coordinates of four core hand key points of the target human body in each frame, including the image coordinate system positions of the left wrist, right wrist, left elbow, and right elbow, as well as the confidence score corresponding to each key point. The output results are then verified. If the confidence score of a certain key point is lower than a preset threshold (e.g., 0.65), it is determined that the key point is missing. At this time, a multinomial regression function is called to infer the coordinates of the missing point based on the complete hand key point data in the adjacent frames before and after the frame (i.e., key points with qualified confidence and reliable coordinates) through temporal continuity, and finally generates a set of coordinates of multiple target human body hand key points.

[0080] Specifically, step 103 may include the following sub-steps:

[0081] S31. The mmpose pose estimation model is used to estimate the pose of the device operation image frame, and outputs multiple initial human hand key point coordinates and confidence scores corresponding to each initial human hand key point coordinate.

[0082] S32. Compare each confidence score with the preset confidence threshold;

[0083] S33. Take the initial human hand key point coordinates corresponding to any confidence score greater than the preset confidence threshold as the target human hand key point coordinates.

[0084] S34. Use a regression function to correct the initial human hand key point coordinates corresponding to any confidence score less than or equal to the preset confidence threshold, and output the target human hand key point coordinates.

[0085] The initial coordinates of multiple key points of the human hand include the initial image coordinate system positions of the left wrist, right wrist, left elbow, and right elbow.

[0086] It should be noted that this invention uses the mmpose pose estimation model to identify hand keypoints in images. If missing keypoints of the human hand (left and right wrists, left and right elbows) are detected in certain frames, i.e., the confidence score of an initial human hand keypoint coordinate is lower than a preset confidence threshold (0.65), then the keypoint is considered unreliable or missing. The system will use a regression function to predict the missing keypoints. The prediction process uses the temporal relationship between consecutive frames to infer the most likely location of the missing keypoint. The formula for the regression function is:

[0087] ;

[0088] in, The predicted coordinates of key points of the human hand are the coordinates of the target human hand key points output by the regression function. The complete skeletal keypoint data vector for the (i-1)th frame refers to the set of vectors composed of the coordinate information of human skeletal keypoints (such as left and right wrists, left and right elbows, etc.) when the keypoints are completely identified and the confidence (score) is higher than a preset threshold (0.65) in a certain frame of the image. For the first A complete skeleton keypoint data vector; function f is based on multinomial regression.

[0089] Furthermore, if the confidence score of a certain initial human hand key point coordinate is greater than the preset confidence threshold, then it is used as the target human hand key point coordinate.

[0090] It's worth noting that the mmpose pose estimation model was trained by acquiring video clips of different device operations, recording video data of operators interacting with different types of devices. This video data served as the training dataset for the model to learn the interaction behavior between humans and specific devices. To ensure data quality, FFmpeg was used to convert the video stream into a sequence of image frames at a fixed frame rate, and the image frames were filtered and denoised to obtain a preprocessed image frame dataset. Finally, the preprocessed image frame dataset was used to train the mmpose pose estimation model.

[0091] Step 104: Determine the hand posture feature vector based on the coordinates of key points of each target human hand.

[0092] It should be noted that, based on the generated coordinates of the complete hand key points of each target human body (including the image coordinate system positions of the left wrist, right wrist, left elbow, and right elbow), the Temporal Factor Dynamic Weight Allocation (TD-DWA) module is introduced to calculate the dynamic weight of each key point. Using the basic weight of the key point (0.3 for the left and right wrists, 0.2 for the left and right elbows) as a benchmark, and combining it with the real-time motion trend (velocity, acceleration), a dynamic adjustment coefficient is obtained, and finally the dynamic weight of each key point is determined. Then, the coordinates of the hand center point are calculated by weighted averaging. This coordinate serves as the core parameter for quantitatively representing the human hand posture, which is the hand posture feature vector, used for subsequent fusion processing with device position features.

[0093] Specifically, step 104 may include the following sub-steps:

[0094] S41. Using the average velocity vector corresponding to the coordinates of the key points of the hand of each target human body, calculate the motion trend factor corresponding to the coordinates of the key points of the hand of each target human body.

[0095] S42. Calculate the dynamic weights corresponding to the key point coordinates of the hand of each target human body based on the motion trend factors corresponding to the key point coordinates of each target human body.

[0096] S43. Calculate the weighted average based on the coordinates of the key points of the hand of each target human body and the dynamic weights corresponding to the coordinates of the key points of the hand of each target human body, and output the hand posture feature vector.

[0097] It should be noted that this invention assigns dynamic weights to the coordinates of key points on each target human hand to calculate the hand center point (P). center The process introduces a temporal factor dynamic weight allocation (T-DWA) module, which can assign coordinates to the left wrist (P) based on the operator's real-time motion state and spatial focus. lw ), right wrist (P rw ), left elbow (P le ) and right elbow (P re These four key points are used to calculate and assign dynamic weights to ensure P center It can more accurately reflect the operator's current focus and intention. The TD-DWA module calculates the dynamic weights W. dynamic,k The basic formula is:

[0098] ;

[0099] in, Dynamic weights representing the target image coordinate system positions of four key points (i.e., left wrist, right wrist, left elbow, and right elbow); These are the initial base weights for the four key points, with the initial weights for the left and right wrists set to 0.3. = =0.3), and the initial weights of the left and right elbows are set to 0.2 ( = =0.2); For movement trend factors; This is a preset coefficient used to adjust the degree of influence of motion trends; it is set here. It is 0.5.

[0100] Furthermore, the movement trend factor The introduction of this feature is to capture the real-time motion speed and intent of each keypoint. Specifically, the system analyzes each keypoint P... k Calculate the average velocity vector of the keypoint itself from the most recent N frames of temporal data (N=10). and acceleration scalar This factor It is a normalized linear combination of velocity and acceleration, which reflects the intensity of the action at key point k. The larger the value, the closer the key point is to the critical moment of the operation.

[0101] ;

[0102] in, The magnitude of the average velocity vector at key point k; This is the normalization function; The acceleration scalar represents the acceleration at the key point k; It is a preset coefficient used to balance the effects of velocity and acceleration.

[0103] Furthermore, after obtaining the dynamic weights of the left wrist, right wrist, left elbow, and right elbow, the final hand center point P is determined. center The coordinates are calculated using a weighted average:

[0104] ;

[0105] in, , , and These represent the left wrist, right wrist, left elbow, and right elbow, respectively. Dynamic weights representing four key points.

[0106] This dynamic weight allocation mechanism makes the calculated P... center It can adaptively shift towards the key points with the most intense movement and closest to the device, thereby greatly enhancing the ability and robustness of hand posture feature vectors to discriminate complex and fine-grained operational behaviors.

[0107] Furthermore, the calculated center point is used as the hand pose feature vector.

[0108] Step 105: Using a two-layer heterogeneous model stacking and integration structure, the device operation behavior recognition result is output based on the hand posture feature vector and the device position feature vector in the device non-operation detection result.

[0109] The two-layer heterogeneous model stacked ensemble structure includes a meta-learner and multiple base learners. It refers to an ensemble learning framework composed of two layers of models. The first layer consists of multiple heterogeneous base learners (classification models with different algorithmic principles), and the second layer is a meta-learner. Recognition accuracy is improved by integrating the outputs of the base learners. "Heterogeneous" reflects the algorithmic differences among the base learners, and "stacked ensemble" reflects the collaborative, sequential mode of the two-layer models.

[0110] It should be noted that this invention concatenates the hand posture feature vector (coordinates of the hand center point) with the device position feature vector (coordinates of the device bounding box center) from the device inactivity detection result to form a unified feature matrix that integrates human operation posture and device spatial position. This feature matrix is ​​then input into a two-layer heterogeneous model stacked structure. The first layer is a base learner layer containing four heterogeneous classifiers (base learners): Random Forest, Support Vector Machine, Multilayer Perceptron, and Lightweight XGBoost. Each classifier independently processes the feature matrix and outputs a category probability vector for the device operation behavior. These four probability vectors are then horizontally concatenated into a meta-feature vector, which is input into the full-featured XGBoost meta-learner in the second layer. By learning the decision bias and complementarity of the base learners, the final behavior category probability distribution is output. The category with the highest probability is selected as the device operation behavior recognition result, achieving accurate judgment of fine-grained operation behaviors (such as "adjusting instruments" or "maintaining valves").

[0111] Specifically, step 105 may include the following sub-steps:

[0112] S51. Concatenate the hand pose feature vector and the device position feature vector to output a fused feature matrix;

[0113] S52. Use the fused feature matrix as the input to each meta-learner and output multiple prediction probability vectors.

[0114] S53. Concatenate the predicted probability vectors to output the meta-feature vector;

[0115] S54. Use the meta-feature vector as input to the meta-learner to generate device operation behavior recognition results.

[0116] It should be noted that the extracted human hand pose feature vector and the device position feature vector are concatenated to form a fusion feature matrix V containing multiple pieces of information. fused The multi-information fusion feature matrix V is formed through feature concatenation. fused The input is fed into a two-layer heterogeneous model stacking ensemble structure to achieve high-precision recognition of complex and fine-grained device operation behaviors. The core of this ensemble structure lies in employing a two-layer learner mechanism to maximize the mining of V... fused The potential for collaborative discrimination of human hand position and equipment position information.

[0117] The first layer of this structure is the base learner layer (Level-0), which consists of four independent classifiers with different modeling principles. These four base learners include a Random Forest (MRF) model, a Support Vector Machine (MSVM) model, a Multilayer Perceptron (MMLP) model, and a Lightweight Gradient Boosting (MXGB) model. fused The input will be simultaneously fed into these four base learners for training, with each model M... i Each will output its predicted probability vector P for each behavior category. i :

[0118] ;

[0119] Where C represents the total number of behavior categories; This indicates that the probability vector is a real number vector containing C elements, where each element corresponds to the predicted probability of a behavior category. Let represent the predicted probability vector output by the i-th base learner.

[0120] Then we proceed to the second layer, the meta-learner layer (Level-1). First, the predicted probability vectors of all base learners in the first layer... They are horizontally concatenated to form a new meta-feature vector V. meta The meta-eigenvector V meta =[P RF ,P SVM ,P MLP ,P XGBThis includes an integrated understanding of the original classification problem from four heterogeneous models. Ultimately, V... meta The input is fed into the meta-learner, which is a full-featured XGBoost (Extreme Gradient Boosting) model M. Meta The system then performs secondary training and makes the final classification decision.

[0121] Final classification and recognition results For meta-learner M Meta The index of the class with the highest probability value in the output probability vector:

[0122] ;

[0123] Here, argmax is used to select the category with the highest output probability as the recognition result. This represents the behavior category predicted by the model, i.e., the result of the device operation behavior recognition.

[0124] Furthermore, this two-layer heterogeneous integrated structure introduces multiple base learners with complementary advantages and utilizes a meta-learner M. Meta Adaptively learning how to optimize the combination of predictions from these heterogeneous models greatly enhances the model's ability to fit training data and its generalization performance. When dealing with challenging tasks such as fine-grained action recognition, this architecture effectively integrates the different models' understanding of features, thereby significantly improving recognition accuracy and system robustness.

[0125] It's worth noting that by training the feature matrix, the XGBoost model can learn the details of different device operations, accurately identify complex and fine-grained device operation behaviors, and capture the key features of each operation behavior. After training, when the system inputs a video, the XGBoost model will perform multi-class classification judgments on the behaviors in the video, and select the category with the highest probability value as the final recognition result based on the calculated probability value, and display the recognition result through visualization.

[0126] For comparison of technical effectiveness, existing technologies can be used as a reference. Currently, action recognition methods based on human skeletons are a common approach. However, the main limitation of these methods is that they ignore the influence of environmental context information. In real-world scenarios, the same action may represent completely different meanings in different environments and contexts, and relying solely on skeleton information often fails to fully capture these contextual differences. Furthermore, these methods typically depend on the accuracy of the pose detector; if the quality of human keypoint acquisition is poor or missing, it will severely impact recognition performance. Therefore, how to combine skeleton features with environmental context information to improve the accuracy and robustness of human interaction detection has become a critical problem that urgently needs to be solved in this field.

[0127] To address the aforementioned problems, this invention provides a method for recognizing equipment operation behavior based on the relationship between posture and spatial position. By deeply fusing human posture data and equipment target data, it successfully overcomes the dual challenges of "lack of environmental context" and "insufficient granularity" faced by existing technologies when handling specific scenarios such as high-speed rail equipment operation. This method not only effectively solves the recognition challenges posed by factors such as lighting, background, and occlusion in complex industrial environments, but more importantly, it transforms the general behavior recognition problem into a feasible relationship reasoning problem specific to a particular scenario, achieving accurate association between operational behavior and specific equipment. This gives this invention enormous commercial and application potential in fields such as high-speed rail operation and maintenance, industrial automation monitoring, personnel training and assessment, and safety production management.

[0128] Specifically, please refer to Figure 2 The process involves acquiring labeled device data and short videos of human operation; using a target detection model to identify the device, and accurately extracting the device's center point coordinate features based on the Enhanced Localization and Center Point Loss (ELCL-Loss) function. Simultaneously, a posture detection model is used to identify hand keypoints, and missing keypoints are supplemented through temporal prediction based on regression functions. A Temporal Factor Dynamic Weight Allocation (T-DWA) module is introduced to calculate the hand center point, extracting a refined hand posture feature vector. These two feature vectors are then concatenated and fused, and input into a two-layer heterogeneous model stacking ensemble structure to achieve high-precision classification decisions and visualization operations.

[0129] Based on the above, this invention, by fusing multiple information sources such as human posture and the spatial relationship between devices, can accurately identify the interaction between people and specific devices in videos. Simultaneously, by capturing high-level semantic information from complex video visual scenes, it successfully discovers, locates, and classifies human-object pairs and their interaction relationships within the video. It demonstrates high robustness and accuracy in behavior recognition in complex scenes. This invention is primarily applied in the field of artificial intelligence technology.

[0130] In summary, this invention ensures the effective capture of key information in complex scenarios, significantly improving model performance and computational efficiency. This invention can effectively identify fine-grained interaction relationships between people and specific targets, and achieve high-precision behavior recognition using limited training data, thus demonstrating extremely high practical value in the specific industrial scenario of high-speed rail operation and maintenance.

[0131] In this embodiment of the invention, a method for recognizing device operation behavior based on the relationship between posture and spatial position is provided. The method acquires video clips of the device not being operated and video clips of the device being operated, preprocesses these clips, and outputs static images of the device not being operated and image frames of the device being operated. A YOLOv11 target detection model is used to detect the static images of the device not being operated, and the detection result is output. An mmpose posture estimation model and regression function are used to generate coordinates of multiple target human hand key points based on the image frames of the device being operated. Based on the coordinates of each target human hand key point, a hand posture feature vector is determined. A two-layer heterogeneous model stacking and integration structure is used to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device not being operated detection result, outputting the device operation behavior recognition result. Based on the above scheme, this invention incorporates the environmental context information of the device's spatial position into the recognition framework, compensating for the neglect of environmental information in traditional methods. Simultaneously, the regression function supplements potentially missing hand key points, reducing the over-reliance on the accuracy of the posture detector. The synergistic effect of both effectively improves the recognition accuracy of device operation behavior in complex scenarios.

[0132] For better explanation, refer to Figure 3 The diagram illustrates the steps of training the YOLOv11 object detection model provided in Embodiment 2 of the present invention. This process may include the following steps:

[0133] Step 301: Obtain the device inactive video clips for model training, preprocess the device inactive video clips for model training, and output the device inactive static images for model training.

[0134] Step 302: Use the YOLOv11 initial detection model to perform detection based on the device-inactive static images used for model training, and output the device-inactive detection results used for model training.

[0135] Step 303: Using a preset loss function, train the initial YOLOv11 detection model based on the device inactivity detection results used for model training, and output the trained YOLOv11 object detection model.

[0136] It should be noted that, in order to more accurately and robustly identify and locate specific equipment in complex industrial scenarios, this invention designs an Enhanced Localization and Center Point Loss (ELCL-Loss) function as the training objective of the model. ELCL-Loss consists of bounding box localization loss (L... loc ) and center point dynamic weight loss (L center It consists of two parts, and the total loss function (i.e., the preset loss function) is defined as follows:

[0137] ;

[0138] For bounding box localization loss (L loc The calculation aims to optimize the regression accuracy of predicted bounding boxes relative to the true bounding boxes, and is used to calculate the localization loss of bounding boxes in the object detection model. This invention uses robust Wise-IoU (WIoU) as the localization loss for bounding boxes.

[0139] WIoU introduces a dynamic non-monotonic focusing mechanism that automatically distinguishes between high-quality (normal) and low-quality (abnormal) samples in the training dataset. Its core function is to dynamically reduce the gradient contribution of low-quality samples (such as occluded or blurred devices) to the model training, thereby suppressing the impact of abnormal data on model regression and improving the model's generalization ability and localization robustness in complex industrial scenarios.

[0140] The loss function L loc The definition of is:

[0141] ;

[0142] The formula consists of two parts: base distance loss. Non-monotonic focusing coefficient .

[0143] As the base distance loss, it is used to calculate the geometric differences of the bounding boxes. It is defined as:

[0144] ;

[0145] IoU (Intersection over Union) is the intersection-union ratio between the predicted box and the ground truth box, used to measure the degree of overlap between the two boxes. and These are the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. It is the width and height of the smallest bounding box that simultaneously contains the predicted box and the ground truth box.

[0146] The (non-monotonic focusing coefficient) is a dynamic, non-monotonic gradient weight used to adjust the weights of the loss based on the current training quality. It is defined as:

[0147] ;

[0148] Where IoU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box. It is a dynamic bounding box quality evaluation metric that assesses the regression difficulty of bounding boxes by calculating the ratio of the center distance between the minimum bounding box and the center distance between the predicted box and the ground truth box. It is a hyperparameter used to control the degree of non-monotonic focusing.

[0149] Through this dynamic weight design, high-quality bounding boxes (IoU close to 1) are achieved. This will result in smaller weights, allowing for a focus on optimizing challenging outlier samples, thus enhancing the model's robustness.

[0150] Furthermore, regarding the dynamic weight loss of the center point ( Calculation: This loss function aims to directly optimize the device center point prediction accuracy output by the object detection model (YOLOv11). It effectively avoids the limitations of traditional Euclidean distance metrics in fine-grained localization optimization by designing a dynamic penalty mechanism based on a Gaussian kernel function, and achieves dynamic optimization by combining spatial penalty, target size information, and prediction confidence. During the calculation process... The model's prediction center point coordinates are needed. Prediction confidence (C) i (and the true center point coordinates from the training labels) and the actual bounding box size As input, it ultimately outputs a scalar loss value. This is used to guide the model in updating weights. The loss formula is defined as:

[0151] ;

[0152] in, This represents the summation over all positive samples (where the intersection-over-union (IoU) ratio between the predicted bounding box and the actual device bounding box is greater than a preset threshold). The loss value corresponding to the preset loss function; The weight coefficients corresponding to the loss values ​​of the bounding box are located; Locate the loss value for the bounding box; The weight coefficient corresponding to the dynamic weight loss value of the center point; The dynamic weight loss value for the center point; The focusing coefficient is non-monotonic; The base distance loss value; The intersection-union ratio (IUU) is the ratio of the predicted device bounding box to the ground truth device bounding box in the device inactivity detection results used for model training. It measures the degree of overlap between the two boxes. It is the natural logarithm function; The x-coordinate of the center coordinate of the device prediction box is the x-coordinate of the device position feature vector in the device inactive detection result used for model training. The x-coordinate of the center of the actual frame of the device; The ordinate of the center coordinate of the device prediction box is the ordinate of the device position feature vector in the device inactive detection result used for model training. The ordinate of the center coordinate of the actual frame of the device; , These are the width and height of the smallest bounding box that simultaneously contains both the predicted and ground truth boxes; As a quality assessment metric for dynamic bounding boxes; This is a hyperparameter used to control the degree of non-monotonic focusing; for; The total number of static images not processed by the device used for model training; and the total number of positive samples. Let be the prediction confidence in the detection result of device inaction for model training corresponding to the i-th static image of device inaction used for model training. These are the parameters used for dynamic decay weights; Let x be the x-coordinate of the device location feature vector corresponding to the i-th static image that the device did not operate on during model training; Let x be the x-coordinate of the center of the real frame of the i-th device; Let y be the ordinate of the device location feature vector corresponding to the i-th static image that is not operated by the device used for model training; The ordinate of the center coordinate of the i-th device's real frame; , Let be the standard deviation of the Gaussian kernel function. , , These are the width and height of the actual device frame, respectively. This is a preset scaling factor.

[0153] Furthermore, the initial YOLOv11 detection model is trained using the aforementioned loss function to obtain a trained YOLOv11 object detection model. Simultaneously, after the training phase, the model with the best validation performance is saved. Finally, in the application phase, this model performs inference on preprocessed image frames. For each identified device, the model outputs its center coordinates (x, y, y) in the image coordinate system. c ,y c ), namely the device location feature vector and a confidence score C. The center point coordinates (x, y) of these bounding boxes. c ,y c The extracted reference points are used as the spatial location features of the device for subsequent feature fusion and stitching.

[0154] In this embodiment of the invention, by acquiring and preprocessing video clips of devices not in operation for model training, static images with unobstructed and clear images of the device can be selected as training samples. This provides high-quality and targeted input data for the training of the YOLOv11 model, ensuring that the samples accurately reflect the static features of the device in different scenarios. Based on this, the YOLOv11 initial detection model is used to detect the static images used for training, generating initial detection results containing device bounding boxes, categories, and confidence levels, providing specific learning basis for model optimization. Then, the initial model is trained using a preset loss function, which quantifies the difference between the initial detection results and the real annotations, driving the model to iteratively optimize the parameters of the backbone network, neck network, and head network, ultimately obtaining a trained YOLOv11 object detection model. This series of steps enables the model to accurately identify the spatial location (bounding box coordinates) and category attributes of the device, significantly improving the confidence and positioning accuracy of device detection. This provides reliable and accurate device environmental context information for subsequent fusion of human posture features, laying the foundation for fine-grained device operation behavior recognition.

[0155] Please see Figure 4 , Figure 4 This is a structural block diagram of a device operation behavior recognition system based on the relationship between posture and spatial position, provided in Embodiment 3 of the present invention.

[0156] This invention provides a device operation behavior recognition system based on the relationship between posture and spatial position, comprising:

[0157] The acquisition module 401 is used to acquire video clips of the device not being operated and video clips of the device being operated, and to preprocess the video clips of the device not being operated and video clips of the device being operated, and to output static images of the device not being operated and image frames of the device being operated.

[0158] The detection module 402 is used to detect static images of devices that have not been operated using the YOLOv11 target detection model and output the detection results of devices not being operated.

[0159] The generation module 403 is used to generate the coordinates of multiple target human hand key points based on the device operation image frame using the mmpose pose estimation model and regression function.

[0160] The determination module 404 is used to determine the hand posture feature vector based on the coordinates of key points of the hand of each target human body;

[0161] The recognition module 405 is used to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device non-operation detection result using a two-layer heterogeneous model stacked integrated structure, and output the device operation behavior recognition result.

[0162] In one optional system embodiment, it further includes:

[0163] The first module is used to acquire video clips of the device not being operated for model training, preprocess the video clips of the device not being operated for model training, and output static images of the device not being operated for model training.

[0164] The second module is used to perform detection based on the device-inactive static images used for model training using the YOLOv11 initial detection model, and output the device-inactive detection results used for model training.

[0165] The third module is used to train the initial YOLOv11 detection model using a preset loss function based on the device inactivity detection results used for model training, and outputs the trained YOLOv11 object detection model.

[0166] The preset loss function is as follows:

[0167] ;

[0168] ;

[0169] ;

[0170] ;

[0171] ;

[0172] in, The loss value corresponding to the preset loss function; The weight coefficients corresponding to the loss values ​​of the bounding box are located; Locate the loss value for the bounding box; The weight coefficient corresponding to the dynamic weight loss value of the center point; The dynamic weight loss value for the center point; The focusing coefficient is non-monotonic; The base distance loss value; The intersection-union ratio (IUU) is the ratio of the predicted device bounding box to the ground truth device bounding box in the device inactivity detection results used for model training. It measures the degree of overlap between the two boxes. It is the natural logarithm function; The x-coordinate of the center coordinate of the device prediction box is the x-coordinate of the device position feature vector in the device inactive detection result used for model training. The x-coordinate of the center of the actual frame of the device; The ordinate of the center coordinate of the device prediction box is the ordinate of the device position feature vector in the device inactive detection result used for model training. The ordinate of the center coordinate of the actual frame of the device; , These are the width and height of the smallest bounding box that simultaneously contains both the predicted and ground truth boxes; As a quality assessment metric for dynamic bounding boxes; This is a hyperparameter used to control the degree of non-monotonic focusing; for; The total number of static images not processed by the device used for model training; and the total number of positive samples. Let be the prediction confidence in the detection result of device inaction for model training corresponding to the i-th static image of device inaction used for model training. These are the parameters used for dynamic decay weights; Let x be the x-coordinate of the device location feature vector corresponding to the i-th static image that the device did not operate on during model training; Let x be the x-coordinate of the center of the real frame of the i-th device; Let y be the ordinate of the device location feature vector corresponding to the i-th static image that is not operated by the device used for model training; The ordinate of the center coordinate of the i-th device's real frame; , Let be the standard deviation of the Gaussian kernel function. , , These are the width and height of the actual device frame, respectively. This is a preset scaling factor.

[0173] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0174] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the above embodiments.

[0175] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the above embodiments.

[0176] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any of the above embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing device operation behavior based on the relationship between posture and spatial position, characterized in that, include: Acquire video clips of the device not being operated and video clips of the device being operated, preprocess the video clips of the device not being operated and the video clips of the device being operated, and output static images of the device not being operated and image frames of the device being operated. The YOLOv11 target detection model is used to detect static images of the device when it is not in operation, and the detection results of the device not being in operation are output. The mmpose pose estimation model and regression function are used to generate the coordinates of multiple key points of the target human hand based on the image frames of the device operation; Based on the coordinates of the key points of the target human hand, the hand posture feature vector is determined; A dual-layer heterogeneous model stacking and integration structure is adopted to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device non-operation detection result, and output the device operation behavior identification result.

2. The device operation behavior recognition method based on attitude and spatial position relationship according to claim 1, characterized in that, The preprocessing of the video clips showing the device not in operation and the video clips showing the device in operation, to output static images of the device not in operation and image frames of the device in operation, includes: The video clips of the device not being operated are extracted and labeled to generate a static image of the device not being operated; The device operation video clips are converted into an image frame sequence, and the image frame sequence is filtered and denoised to generate device operation image frames.

3. The device operation behavior recognition method based on attitude and spatial position relationship according to claim 1, characterized in that, The method employs the mmpose pose estimation model and regression function to generate multiple target human hand key point coordinates based on the device operation image frames, including: The mmpose pose estimation model is used to estimate the pose of the device operation image frame, and outputs multiple initial human hand key point coordinates and confidence scores corresponding to each initial human hand key point coordinate. Each confidence score is compared with a preset confidence threshold. Use the initial coordinates of the human hand key points corresponding to any confidence score greater than the preset confidence threshold as the coordinates of the target human hand key points. The regression function is used to correct the initial human hand key point coordinates corresponding to any confidence score less than or equal to the preset confidence threshold, and the target human hand key point coordinates are output.

4. The device operation behavior recognition method based on attitude and spatial position relationship according to claim 1, characterized in that, The step of determining the hand posture feature vector based on the coordinates of key points of each target human hand includes: The motion trend factor corresponding to the coordinates of the key points of the hand of each target human body is calculated using the average velocity vector corresponding to the coordinates of the key points of the hand of each target human body. Based on the motion trend factor corresponding to the coordinates of the key points of the hand of each target human body, calculate the dynamic weight corresponding to the coordinates of the key points of the hand of each target human body. The hand posture feature vector is output by performing a weighted average calculation based on the coordinates of the key points of the target human hand and the dynamic weights corresponding to the coordinates of the key points of the target human hand.

5. The device operation behavior recognition method based on attitude and spatial position relationship according to claim 1, characterized in that, The dual-layer heterogeneous model stacked ensemble structure includes a meta-learner and multiple base learners; the dual-layer heterogeneous model stacked ensemble structure identifies the device operation behavior based on the hand pose feature vector and the device position feature vector in the device inactive detection result, and outputs the device operation behavior recognition result, including: The hand posture feature vector and the device position feature vector are concatenated to output a fused feature matrix; The fused feature matrix is ​​used as the input to each meta-learner, and multiple prediction probability vectors are output. The predicted probability vectors are concatenated to output the meta-feature vector; The meta-feature vector is used as the input to the meta-learner to generate device operation behavior recognition results.

6. The device operation behavior recognition method based on attitude and spatial position relationship according to claim 1, characterized in that, The training process of the YOLOv11 object detection model is as follows: Acquire a video clip of the device not being operated for model training, preprocess the video clip of the device not being operated for model training, and output a static image of the device not being operated for model training. The initial detection model of YOLOv11 is used to perform detection based on the static images of no device operation used for model training, and outputs the detection results of no device operation used for model training. The YOLOv11 initial detection model is trained using a preset loss function based on the device inactivity detection results used for model training, and the trained YOLOv11 object detection model is output. The preset loss function is specifically as follows: ； ； ； ； ； in, The loss value corresponding to the preset loss function; The weight coefficients corresponding to the loss values ​​of the bounding box are located; Locate the loss value for the bounding box; The weight coefficient corresponding to the dynamic weight loss value of the center point; The dynamic weight loss value for the center point; It is a non-monotonic focusing coefficient; The base distance loss value; The intersection-union ratio (IUU) is the ratio of the predicted device bounding box to the ground truth device bounding box in the device inactivity detection results used for model training. It measures the degree of overlap between the two boxes. It is the natural logarithm function; The x-coordinate of the center coordinate of the device prediction box is the x-coordinate of the device position feature vector in the device inactive detection result used for model training. The x-coordinate of the center of the actual frame of the device; The ordinate of the center coordinate of the device prediction box is the ordinate of the device position feature vector in the device inactive detection result used for model training. The ordinate of the center coordinate of the actual frame of the device; , These are the width and height of the smallest bounding box that simultaneously contains both the predicted and ground truth boxes; As a quality assessment metric for dynamic bounding boxes; This is a hyperparameter used to control the degree of non-monotonic focusing; for; The total number of static images not processed by the device used for model training; and the total number of positive samples. Let be the prediction confidence in the detection result of device inaction for model training corresponding to the i-th static image of device inaction used for model training. These are the parameters used for dynamically decaying weights; Let x be the x-coordinate of the device location feature vector corresponding to the i-th static image that the device did not operate on during model training; Let x be the x-coordinate of the center of the real frame of the i-th device; Let y be the ordinate of the device location feature vector corresponding to the i-th static image that is not operated by the device used for model training; The ordinate of the center coordinate of the i-th device's real frame; , Let be the standard deviation of the Gaussian kernel function. , , These are the width and height of the actual device frame, respectively. This is a preset scaling factor.

7. A device operation behavior recognition system based on the relationship between posture and spatial position, characterized in that, include: The acquisition module is used to acquire video clips of the device not being operated and video clips of the device being operated, and to preprocess the video clips of the device not being operated and video clips of the device being operated, and to output static images of the device not being operated and image frames of the device being operated. The detection module is used to detect static images of the device when it is not being operated on using the YOLOv11 target detection model, and output the detection result of the device not being operated. The generation module is used to generate multiple target human hand key point coordinates based on the device operation image frames using the mmpose pose estimation model and regression function; The determination module is used to determine the hand posture feature vector based on the coordinates of key points of the hand of each target human body; The recognition module is used to identify the device operation behavior based on the hand posture feature vector and the device position feature vector in the device non-operation detection result using a two-layer heterogeneous model stacked integrated structure, and output the device operation behavior recognition result.

8. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the device operation behavior recognition method based on attitude and spatial position relationship as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the device operation behavior recognition method based on the relationship between posture and spatial position as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the device operation behavior recognition method based on attitude and spatial position relationship as described in any one of claims 1-6.

Images