Method, device and medium for intelligent analysis of operating behavior of device operator

Abstract

The embodiment of the application discloses a kind of intelligent analysis methods for the operation behavior of equipment operator, comprising: obtaining the operation video section of equipment operator;For each time window of the video section: S1, human skeleton point recognition is carried out to each frame image of current time window, and the basic motion feature, statistical feature and timing feature of skeleton point are extracted;S2, the feature set of current time window is respectively input into the identification model of each behavior category, and the first confidence of each behavior category in current time window is obtained respectively;According to the first confidence time sequence of each behavior category in multiple time windows, the second confidence that the video section belongs to each behavior category is determined;The behavior category of second confidence highest is regarded as the behavior category that the video section finally belongs to.This embodiment can accurately intelligently identify and judge equipment operation behavior, to realize the human factor detection of control equipment, or automatic training / assessment etc..

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, device, and medium for intelligent analysis of the operational behavior of equipment operators. Background Technology

[0002] Currently, many control systems focus primarily on the needs of the control equipment and the operational content, rarely specifying requirements for monitoring and recording the actions of equipment operators. Typical recording is limited to video recording, with post-incident analysis conducted after an accident to determine whether drone pilot error caused the crash, or whether misoperation of industrial equipment led to the accident. Simultaneously, standardized operating procedures are crucial. Users may want to optimize training, shortening the time it takes for novices to become experts; or, by analyzing the posture data of skilled operators, establish standards to help others improve.

[0003] Therefore, there is an urgent need for an intelligent analysis method for the operational behavior of equipment operators, which can accurately and intelligently identify and judge the operational behavior of equipment, so as to realize human factor detection of equipment operation, or automated training / assessment, etc. Summary of the Invention

[0004] This invention provides a method, device, and medium for intelligent analysis of operator behavior to solve at least one of the above-mentioned problems.

[0005] In a first aspect, embodiments of the present invention provide an intelligent analysis method for the operational behavior of equipment operators, including: Acquire video clips of equipment operators performing their tasks; For each time window of the video segment: S1. Perform human skeleton point recognition on each frame of the current time window, and extract the basic motion features, statistical features and temporal features of the skeleton points; S2. Input the feature set of the current time window into the recognition model of each behavior category respectively, and obtain the first confidence of each behavior category in the current time window respectively; Based on the time sequence of the first confidence level of each behavior category across multiple time windows, the second confidence level of the video segment belonging to each behavior category is determined; the behavior category with the highest second confidence level is taken as the final behavior category to which the video segment belongs.

[0006] In a second aspect, embodiments of the present invention provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent analysis method for the operational behavior of equipment operators as described in any embodiment.

[0007] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent analysis method for the operational behavior of equipment operators as described in any embodiment.

[0008] In summary, the embodiments of the present invention provide an intelligent analysis method for the operational behavior of equipment operators. Based on AI image analysis and human skeleton recognition, and through machine learning methods, an automated process for dynamically recognizing operational behavior is constructed. This realizes the matching of human actions and behaviors with a behavior recognition database and the output of results, which helps in the human factor detection of equipment operation, or automated training / assessment, etc.

[0009] In particular, when the recognition models for each behavior category are trained separately, the confidence scores output by each model may not be directly comparable, affecting the accuracy of recognition and judgment. Therefore, this embodiment corrects the confidence scores directly output by the models based on the recognition difficulty, category imbalance, and confidence score stability of each action category. This corrected confidence scores can be directly compared to reflect the true category situation, improving the accuracy of behavior category recognition and ensuring more accurate human factor detection for equipment operation, or automated training / assessment, etc.

[0010] More specifically, this embodiment uses class imbalance as a static main gating and temporal instability as a dynamic adjustment factor during the correction process. Through nonlinear coupling, it achieves the fusion of recognition difficulty, class balance, and temporal stability, while reflecting the influence of the three types of factors on the reliability of confidence. It avoids mutual cancellation among the three types of factors, making the correction value more consistent with the recognition difficulty of the action category itself and the training status of the model. Thus, when using unified real labels for supervised training, it is easier to obtain reasonable convergent training parameters, which promotes the improvement of behavior recognition accuracy and the realization of human factor detection of control equipment, or automated training / assessment, etc. Attached Figure Description

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1This is a flowchart of an intelligent analysis method for the operational behavior of equipment operators provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of human upper body recognition points provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the training of a random forest model provided in an embodiment of the present invention; Figure 4 This is a flowchart of an intelligent analysis method for the operational behavior of equipment operators provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating a movement determination within each time window provided by an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0014] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0015] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0016] This invention provides an intelligent analysis method for the operational behavior of equipment operators. To implement this method, this embodiment uses a common camera to capture images of the front of the human body. The head and skeletal structure of the human body are identified from these images. A weighting strategy is then used to simplify key points of the human structure into multiple key parameters for machine learning algorithms. Next, a positive and negative sample library is established based on preset operational behavior (or operational behavior) categories. A dedicated recognition model is trained for each behavior category to identify whether an operator belongs to that category. The trained model is then used to judge the operator's operational behavior, achieving the purpose of operational monitoring. The entire process is as follows: Figure 1 As shown below, the specific recognition models used in this method will be explained in detail first.

[0017] In one specific implementation, the aforementioned camera can be a relatively common monocular RGB camera with a frame rate of not less than 30 Hz. All cameras with a resolution of 720p or higher are suitable for this method. Optionally, for devices primarily operated by hand, the camera can be aimed at the upper front of the operator, ensuring the image includes the operator's head, shoulders, complete hands, and the area above the hips. The image capture rate can be at least 30 frames per second (based on a camera with a rated frame rate of 30 Hz or higher).

[0018] Based on the real-time images transmitted from the camera, the MediaPipe algorithm can be used to determine 25 skeletal points of the body. Optionally, these skeletal points include... Figure 2 As shown in Table 1: Table 1

[0019] Furthermore, to balance the accuracy and efficiency of action recognition, multiple skeletal points whose relative positions remain unchanged during action execution can be grouped together, and then weighted and merged into a single skeletal point based on the distance of each skeletal point within the group to the group center. For example, combining... Figure 2As shown in Table 1, the relative positions of the 11 points on the head remain unchanged during the execution of the action. Therefore, these 11 points can be grouped together, and the position coordinates of the group center can be calculated based on the position coordinates of each point within the group. Then, based on the distance from each point to the group center, the position coordinates of each point within the group are weighted and summed (the closer the distance, the higher the weight), resulting in a new position coordinate. This position coordinate is the merged equivalent skeletal point. This method does not discard any skeletal points but uses a weighted summation to obtain an equivalent skeletal point, making the calculated coordinate positions more stable. For example, even if the nose point is not recognized, a relatively stable equivalent head point can still be obtained based on other points. Similarly, the four points on the upper torso maintain their relative displacement during the execution of the action, so these four points can be grouped together and ultimately merged into one equivalent skeletal point. The remaining 13 skeletal points may experience relative displacement during the execution of the action, therefore they cannot be merged. The final merged skeletal points (hereinafter referred to as key points for ease of distinction and description) are shown in Table 2: Table 2

[0020] After the key points are determined, at least one of the following is extracted: basic motion features, statistical features, and temporal features. Optionally, the basic motion features include at least one of the following: position features, distance features, angle features, and direction features. For example, the final extracted feature parameters are shown in Table 3. Table 3

[0021] The extracted feature parameters can effectively describe human posture information, especially operational behavior information. Taking the feature parameters listed in the table above as an example, there are 23 groups and 427 parameters in total, with the unit for each parameter varying according to the attribute. Of course, the selection of human key points and feature parameters is not limited to the above embodiment; the general principle is to extract elements associated with human body position and elements associated with spatial movement.

[0022] After obtaining the above features, an operation behavior recognition model can be constructed using these features. Optionally, for each behavior category, the following operations are performed: construct a positive and negative sample set for the current behavior category, where positive samples refer to behavior samples belonging to the current behavior category, and negative samples refer to behavior samples not belonging to the current behavior category; train a dedicated recognition model for the current behavior category using the positive and negative sample set, where the input of the dedicated recognition model is the corresponding feature set of each sample, and the output is the confidence score of each sample belonging to the current behavior category.

[0023] Optionally, the action recognition model can employ the random forest algorithm. Random forests can handle high-dimensional features, so all the above feature parameters can be concatenated into a feature vector, and feature selection can be performed using the feature importance of the random forest to achieve better recognition results.

[0024] In one specific implementation, a camera can continuously capture each frame of image data from the operator, perform skeletal point recognition, calculate feature parameters, and label the actual behavior category to obtain a positive and negative sample set for each behavior category. For example, the operator can perform the behavior categories shown in Table 4 below. The camera is connected to data acquisition and processing software to maintain data acquisition and processing. Before each action, the operator sets the current behavior category, then performs the action of that category, and records the execution time according to the table. If a timeout occurs, the software automatically truncates the specified duration. Of course, positive and negative samples can also be collected during actual operation, resulting in more realistic samples.

[0025] Table 4

[0026] For each behavior category, a new random forest model is trained multiple times using the positive and negative sample sets of that category; the more times, the better. Based on actual test results, training more than 5 times is suitable, with each training iteration generating a set of model parameters. The optimal parameter set can be selected as the final model; alternatively, a nested two-level random forest model approach can be used, merging the parameter sets of each model into a large dataset, and then using this dataset to train a new random forest model as the final action recognition model. Figure 3 As shown. Ultimately, each behavior category corresponds to a dedicated random forest model for judging new video images.

[0027] Based on the above models Figure 4 This is a flowchart of an intelligent analysis method for the operational behavior of equipment operators, provided in an embodiment of the present invention. The method is executed by an electronic device. Figure 4 As shown, the method specifically includes: S110. Obtain video clips of equipment operators' operations.

[0028] In this embodiment, it is assumed that the video segment includes only a single behavior category. Optionally, the video capture duration can be twice the maximum duration of all behavior categories, and a video segment of that duration can be continuously captured after motion data is detected by the camera. For example, according to the data in Table 4, the captured video segment duration is 10 seconds.

[0029] S120. Divide the video segment into multiple time windows and determine the confidence level of each time window belonging to each behavior category in turn.

[0030] Optionally, the time window length can be 2 seconds, and each time window can be a sliding time window, moving and judging over 10 seconds of data, such as... Figure 5 As shown. In one specific implementation, the following operations can be performed sequentially within each time window: S1. Perform human skeleton point recognition on each frame of the current time window and extract at least one of the basic motion features, statistical features and temporal features of the skeleton points.

[0031] Optionally, for each frame of the current time window, the skeletal points of the operator's head and upper body can be identified; multiple skeletal points whose relative positions remain unchanged when performing the action can be grouped together, and the multiple skeletal points can be weighted and merged into one skeletal point according to the distance of each skeletal point in the group to the center of the group.

[0032] Optionally, the statistical features include at least one of the following: the central moment features of the skeleton point distribution, the eigenvalues ​​and / or eigenvectors of the covariance matrix of the skeleton point distribution, the area and / or perimeter of the convex hull formed by the skeleton points, and the aspect ratio and / or orientation of the minimum bounding rectangle of the skeleton points. The temporal features include at least one of the following: the movement velocity of each skeleton point, the acceleration of each skeleton point, and the frequency of the periodic motion of each skeleton point.

[0033] For further details, please refer to the above embodiments.

[0034] S2. Input the feature set of the current time window into the recognition model of each behavior category respectively, and obtain the confidence of each behavior category in the current time window.

[0035] Taking any behavior category as an example, all features of the image in any frame (e.g., the first frame) within the current time window can be input into a dedicated recognition model for that behavior category. The model will then output the confidence score that the action within the current time window belongs to that behavior category. Alternatively, the above operation can be performed on every frame within the current time window, and one of the confidence scores from each frame can be selected as the confidence score for the current time window.

[0036] S130. Based on the time sequence of the first confidence scores of each behavior category across multiple time windows, determine the confidence score of the video segment belonging to each behavior category. For ease of distinction and description, this embodiment refers to the confidence score of the time window as the first confidence score and the confidence score of the video segment as the second confidence score. The behavior category with the highest second confidence score is taken as the final behavior category to which the video segment belongs.

[0037] Furthermore, based on whether the confidence levels directly given by each dedicated recognition model are directly comparable, this embodiment can also provide the following two optional implementation methods to determine the second confidence level of the video segment belonging to each behavior category.

[0038] In the first optional implementation, the confidence scores provided by each model are directly comparable, meaning the magnitudes of the confidence scores can be directly compared. In this case, the following operations can be performed for each behavior category: a time series is constructed from the first confidence scores of the current behavior category across multiple time windows; the maximum value in the time series is taken as the second confidence score of the video segment belonging to the current behavior category. Of course, this maximum value must also be greater than a minimum confidence threshold. Since the first confidence scores of each model can be directly compared, the second confidence scores of each model can also be directly compared, and the final behavior category to which the video segment belongs can be determined based on the highest second confidence score.

[0039] In the second optional implementation, the confidence scores directly provided by each model are not directly comparable. Specifically, in the aforementioned embodiments, since each model is trained independently using positive and negative sample sets for its respective behavior category, and each model only identifies one behavior category, it is possible that the confidence score of a single model meets the recognition requirements, but when multiple models are compared jointly, the benchmarks between the confidence scores are different, leading to inaccurate direct comparisons. To address this, this embodiment provides a method to correct each first confidence score based on the recognition difficulty, category balance, and confidence score stability of each behavior category, making the corrected first confidence scores of different behavior categories comparable. Then, a time series is constructed from the corrected first confidence scores of the same behavior category across multiple time windows, and the maximum value in the time series of each behavior category is taken as the second confidence score of the video segment belonging to each behavior category.

[0040] Optionally, this correction can first scale the initial confidence score output by the model based on the inherent difficulty of recognizing each behavior category. Then, considering the balance of behavior categories during model training and the stability of the confidence scores of each model, the scaled confidence score can be gating-adjusted to obtain a corrected confidence score value that more closely approximates reality. The inherent difficulty of recognizing a behavior category can be reflected in the discriminative power of the confidence scores directly provided by the model. Large differences in directly provided confidence scores indicate that the behavior category is easy to recognize; small differences indicate that the behavior category is difficult to recognize. During the scaling process, the greater the recognition difficulty, the more the distribution range of the initial confidence score should be expanded (or narrowed) less. The balance of behavior categories can be reflected in the sample ratio. A higher sample ratio indicates that the behavior is more common or easier to standardize, thus the initial confidence score is more reliable, and the correction should be moderate (as close to the original value as possible). The stability of the confidence level of a behavior category can be reflected in the temporal stability of the model output. Higher stability indicates more thorough training, resulting in a more reliable initial confidence level, and the calibration should be moderate (as close to the original value as possible). Of course, all calibration processes aim at a unified goal: to reflect the true behavior category. Through calibration, the confidence levels of various models are compared at roughly the same level of recognition difficulty, sample proportion, and stability, making it easier to approximate the true behavior category.

[0041] Based on the above correction approach, in one specific embodiment, the correction process may include the following steps: Step 1: Based on each behavior category sample proportion and gating strength Determine the main gating for each behavior category : (1) in, This represents the sigmoid function, which transforms data distributed between positive and negative infinity to the range [0,1] while preserving the order of the data. Indicates behavior category The number of samples, This represents the total number of samples across all behavior categories. Optionally, the number of samples here can be the number of positive samples from the model training process described above. This represents a logarithmic function used to compress extreme values. Used to reflect class balance This is used to control the gating strength.

[0042] Step 2: Based on each behavior category The first confidence time series in the current time window Determine the categories of behavior Confidence stability: (2) in, Represents the standard deviation function; Indicates behavior category The confidence stability coefficient The larger the value, the lower the timing stability.

[0043] Step 3: Based on each behavior category Time stability coefficient and main gate Determine the sample-temporal joint gating for each behavior category. : (3) in, This is used to compress the time series standard deviation to the range of [0,1]. The ultimate goal of joint gating is to: the larger the sample proportion, the higher the confidence stability, the stronger the gating constraint on scaling, and to avoid overscaling.

[0044] Step 4: Based on each behavior category Joint gating and identification difficulty coefficient First confidence level for each behavioral category Perform correction to obtain the correction value. : (4) Furthermore, in order to make the correction value The components are relatively evenly distributed within the [0,1] interval. Alternatively, you can first... Transform into intermediate data that are uniformly distributed in the interval from negative infinity to positive infinity. : (5) Then As in formula (4) Substituting this into formula (4), the final comprehensive correction formula is: (6) In the above formula, each behavior category gating strength and identification difficulty coefficient This is a user-defined parameter; the other parameters are fixed. and It can be trained in the following way: First, construct a full sample set including all behavior categories. Optionally, positive samples from each model training phase can be reused, using the set of positive samples for each behavior category as the full sample set. Each sample corresponds to a behavior category vector label, which serves as the supervision label in this training. The vector element corresponding to the actual behavior category is 1, and the vector elements corresponding to other behavior categories are 0.

[0045] Then, the computational parameters within each model are frozen unchanged, and the following loss function is used with the full sample set. Gating strength for each behavior category and identification difficulty coefficient Conduct training: (7) in, Indicates sample Does it belong to a behavioral category? The true label (0 or 1). Indicates behavior category The samples output by the dedicated recognition model Does it belong to a behavioral category? The correction value for the first confidence level, This represents the hyperparameter weights. The values ​​in formula (4) or (6) are used to represent the hyperparameter weights. and Replace with and (behavioral category) The samples output by the dedicated recognition model Does it belong to a behavioral category? Substituting the first confidence level into formula (7), we can obtain the complete information about... and The loss function; through Minimize or converge, and you can determine each and The value of , where, This ensures that the corrected confidence levels are comparable and can reflect the true behavioral categories; Used to constrain each fine-tuning amplitude ( Try to get as close to 1 as possible, and avoid overcorrection. Optional, each The same value can also be used.

[0046] In summary, this invention provides an intelligent analysis method for the operational behavior of equipment operators. Based on AI image analysis and human skeleton recognition, and through machine learning methods, it constructs an automated process for dynamically recognizing operational behaviors, achieving matching of human actions with a behavior recognition database and outputting results. Specifically, the application value of this embodiment includes: 1. Safety precautions and accident tracing: from passive response to proactive early warning, especially when operating high-risk or expensive equipment (such as drones, construction machinery, and flight simulators).

[0047] Real-time risk warnings: The system can identify and warn of non-standard, fatigued, or dangerous postures that may lead to operational errors or equipment damage. Examples include drivers frequently looking down at their phones, pilots' improper operation in emergency situations, and workers placing their hands in hazardous machinery areas. These warnings provide immediate alerts to prevent accidents.

[0048] Objective accident investigation: In the event of an accident or emergency, complete posture and movement records (combined with operation logs) provide irrefutable "black box" data. Investigators can accurately reconstruct the operator's hand position, body posture, line of sight, and reaction sequence before the incident, thus clearly distinguishing whether it was equipment failure, design flaw, or human error, providing solid evidence for determining responsibility and making improvements.

[0049] 2. Standardized Operations and Skills Training: From experience transfer to scientific quantification, especially for operations requiring high skills (such as surgical robots, precision instruments, and complex process control), it can improve the overall operational level.

[0050] Establish standard operating model: By recording the standardized, efficient, and robust posture sequences of top experts (such as ace pilots and senior surgeons) in specific tasks, best practice templates can be constructed.

[0051] Quantitative assessment and precision training: Trainees' operating postures can be quantified in real time and compared with the "gold standard." The system can provide objective feedback, such as "elbow angle deviation of 15 degrees," "excessive eye movement," and "excessive redundancy in operating trajectory," enabling personalized, data-driven precision training, greatly shortening the training cycle and improving training quality.

[0052] In particular, since the recognition models for each behavior category in this embodiment are trained separately, there may be situations where the confidence scores directly output by each model cannot be directly compared, affecting the accuracy of recognition and judgment. Therefore, this embodiment corrects the confidence scores directly output by the models based on the recognition difficulty, category imbalance, and confidence score stability of each action category. This allows the corrected confidence scores to be directly compared, reflecting the true category situation, improving the accuracy of behavior category recognition, and ensuring more accurate human factor detection for equipment operation, or automated training / assessment, etc.

[0053] More specifically, this embodiment uses class imbalance as a static main gating and temporal instability as a dynamic adjustment factor during the correction process. Through nonlinear coupling, it achieves the fusion of recognition difficulty, class balance, and temporal stability, while reflecting the influence of the three types of factors on the reliability of confidence. It avoids mutual cancellation among the three types of factors, making the correction value more consistent with the recognition difficulty of the action category itself and the training status of the model. Thus, when using unified real labels for supervised training, it is easier to obtain reasonable convergent training parameters, which promotes the improvement of behavior recognition accuracy and the realization of human factor detection of control equipment, or automated training / assessment, etc.

[0054] It should be noted that the data of the operators involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0055] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 6 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0056] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the intelligent analysis method of operator behavior in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned intelligent analysis method of operator behavior.

[0057] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0058] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0059] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent analysis method for operator behavior of any embodiment.

[0060] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0061] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0062] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0063] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Python, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent analysis of the operational behavior of equipment operators, characterized in that, include: Acquire video clips of equipment operators performing their tasks; For each time window of the video segment: S1. Perform human skeleton point recognition on each frame of the current time window, and extract the basic motion features, statistical features and temporal features of the skeleton points; S2. Input the feature set of the current time window into the recognition model of each behavior category respectively, and obtain the first confidence of each behavior category in the current time window respectively; Based on the first confidence time sequence of each behavior category across multiple time windows, the second confidence of the video segment belonging to each behavior category is determined. The behavior category with the highest second confidence level is taken as the final behavior category to which the video segment belongs.

2. The method according to claim 1, characterized in that, The step of performing human skeleton point recognition on each frame of the current time window includes: For each frame of the current time window, identify the skeletal points of the operator's head and upper body; Multiple skeletal points whose relative positions remain unchanged during the execution of an action are grouped together, and the multiple skeletal points are weighted and merged into one skeletal point based on the distance of each skeletal point in the group to the center of the group.

3. The method according to claim 1, characterized in that, The extraction of basic motion features, statistical features, and temporal features of skeletal points includes: Extract at least one of the following statistical features: the central moment feature of the skeleton point distribution, the eigenvalues ​​and / or eigenvectors of the covariance matrix of the skeleton point distribution, the area and / or perimeter of the convex hull formed by the skeleton points, and the aspect ratio and / or orientation of the minimum bounding rectangle of the skeleton points.

4. The method according to claim 1, characterized in that, The extraction of basic motion features, statistical features, and temporal features of skeletal points includes: Extract at least one of the following temporal features: the movement velocity of each bone point, the acceleration of each bone point, and the frequency of the periodic motion of each bone point.

5. The method according to claim 1, characterized in that, The step of determining the second confidence level of the video segment belonging to each behavior category based on the first confidence time series of each behavior category across multiple time windows includes: A time series is formed by the first confidence level of the same behavior category across multiple time windows; The maximum value in the time sequence of each behavior category is taken as the second confidence score of the video segment belonging to each behavior category.

6. The method according to claim 1, characterized in that, Before S2, it also includes: for each behavior category: using the positive and negative sample set of the current behavior category, train a special recognition model for the current behavior category, the special recognition model is used to output the confidence score of belonging to the current behavior category; Accordingly, determining the second confidence level of the video segment belonging to each behavior category based on the first confidence time series of each behavior category across multiple time windows includes: Based on the difficulty of identification, class balance, and confidence stability of each behavior category, the first confidence level is corrected to make the corrected first confidence levels of different behavior categories comparable. The time series is composed of the first confidence scores of the same behavior category over multiple time windows after correction; The maximum value in the time sequence of each behavior category is taken as the second confidence score of the video segment belonging to each behavior category.

7. The method according to claim 6, characterized in that, The step of correcting each first confidence level based on the recognition difficulty, class balance, and confidence stability of each behavior category includes: The first confidence level is corrected according to the following formula: in, Indicates behavior category The first confidence level, express The correction value, express The time series; Indicates the difficulty coefficient for recognition. Indicates the gating strength; Indicates behavior category The number of samples, This represents the total number of samples across all behavior categories; Represents the standard deviation function. Represents the hyperbolic tangent function. Represents the logarithmic function. This represents the sigmoid function.

8. The method according to claim 7, characterized in that, Before correcting each first confidence level according to the following formula, the method further includes: Construct a full sample set including all behavioral categories; Using the full sample set and the following loss function Gating strength for each behavior category and identification difficulty coefficient Conduct training: in, Indicates sample Does it belong to a behavioral category? The true label, Indicates behavior category The samples output by the dedicated recognition model Does it belong to a behavioral category? The correction value for the first confidence level, This represents the hyperparameter weights.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent analysis method for the operational behavior of equipment operators as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the intelligent analysis method for the operational behavior of equipment operators as described in any one of claims 1-8.

Images