Intelligent internet-of-things evaluation method and system for operation behavior process based on interaction reliability

Abstract

The invention discloses an intelligent internet-of-things evaluation method and system for an operation behavior process based on interaction credibility, and the method comprises the steps: firstly determining a personnel operation process, dividing operation behavior steps, and defining an operation article for interaction between the operation behavior steps and personnel; secondly, constructing a reliability detection model of the personnel and the object target, identifying the hand of the operator and the operation object based on the operation video frame, and obtaining the intersection-union ratio of the hand area of the operator and the operation object according to the identification result; and then, constructing a figure interaction comprehensive reliability index, determining an index threshold, and judging whether an operation behavior step occurs or not. And finally, obtaining an operation behavior step identification result of the determined operation behavior according to a time sequence, performing process consistency determination on the operation behavior step identification result and the operation behavior step, and outputting an evaluation result of operation process execution. According to the invention, stable and explainable behavior recognition is realized, and the management and control efficiency of the operation process is improved.

Description

Technical Field

[0001] This invention belongs to the field of behavior recognition technology, and in particular relates to an intelligent IoT evaluation method and system for work behavior processes based on interaction reliability. Background Technology

[0002] In recent years, industrial enterprises have increasingly stringent safety and quality control requirements for special operations such as welding, temporary electrical work, and hot work. The evaluation of related personnel's work behaviors and processes is gradually moving towards digitalization, informatization, and intelligentization. However, currently, most work behavior and process evaluations still rely primarily on surveillance cameras for manual judgment of work quality, recording of processes, and subsequent re-inspection through random checks. This method is not only highly dependent on human resources but also easily influenced by subjective human factors, resulting in inconsistent work quality assessment standards, low efficiency in work process control, and difficulty in objectively and continuously recording and tracing the entire work process.

[0003] Action recognition technology based on object detection provides a feasible path for computers to understand complex human behaviors. However, during operations, human body parts (hands) often interact with tools and objects, increasing the complexity and dynamism of action step recognition and workflow evaluation. To address these issues, a highly reliable method is needed to recognize action steps based on the physical interaction between the worker and the operated object, thus meeting the requirements of intelligent workflow evaluation. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an intelligent IoT evaluation method and system for work behavior processes based on interaction reliability. This method can improve the accuracy of recognizing the work behavior steps of workers by considering the interaction relationship between the hand and the object detection frame in the spatial dimension and by integrating the reliability of multiple frames in the temporal dimension.

[0005] In a first aspect, the present invention provides an intelligent IoT evaluation method for work behavior processes based on interaction reliability, comprising the following steps:

[0006] S1, Determine the personnel work process, divide the work process into work behavior steps according to the work content, and define the operation items for the work behavior steps and personnel interaction.

[0007] S2, construct a reliability detection model for personnel and object targets, and identify the hands of workers and the objects they are handling based on the operation video frames.

[0008] S3, based on the hand and object recognition results, obtain the intersection-over-union ratio of the operator's hand area and the object being operated.

[0009] S4 integrates the results of intersection and union comparison, hand and object recognition, and jointly constructs a comprehensive reliability index for human interaction, and determines the threshold of the comprehensive reliability index for human interaction.

[0010] S5 compares the overall reliability index of human interaction with the threshold of the overall reliability index of human interaction to judge whether the task behavior steps have occurred.

[0011] S6: Obtain the operation behavior step identification results in chronological order of the determined operation behavior, and then perform process consistency determination with the operation behavior steps in step S1, and output the evaluation results of the operation process execution.

[0012] Preferably, step S1 includes:

[0013] S11. Based on the work operation standards, formulate the preliminary work process and divide the process into work action steps. .

[0014] S12, Determine the steps of the work action. The items are manipulated by personnel and defined according to the work behavior step index. .

[0015] Preferably, step S2 includes:

[0016] S21, Construct a deep learning detection model for target reliability. ,Model This is used to perform target detection and classification prediction on operation video frames, and output the bounding box of the operated item, the corresponding confidence level and the uncertainty of the confidence level; among them, the target confidence deep learning detection model is implemented by introducing an uncertainty modeling mechanism into the target detection network.

[0017] S22, Install fixed sensing cameras in the work area to collect work actions and steps in real time. middle Time-based operation video frames And use the detection model deployed on the edge server. For the video frames of the operation Hands of workers and items handled Perform recognition and obtain the hand recognition frames of the workers respectively. and the identification box for the manipulated items ,in , , , , , , , Representing the recognition box and The x-coordinate of the top left corner, the y-coordinate of the top left corner, the width of the recognition box, and the height of the recognition box.

[0018] S23, Detection Model Simultaneously output the confidence level of each bounding box. , And the uncertainty of the corresponding reliability. and .

[0019] Preferably, the uncertainty modeling mechanism includes:

[0020] The head classification branch of the object detection algorithm is structurally modified so that it no longer outputs a score, but instead outputs the result after the activation function. Activated evidence Subsequently, based on the theory of evidence deep learning, evidence Mapped to Dirichlet Distribution parameters Then through parameters With all parameters Calculate the confidence level by the sum of the proportions and through With all parameters The uncertainty of reliability is calculated based on the total percentage. , This represents the total number of item categories being manipulated, and an evidence-based multi-classification prediction framework is constructed for subsequent reliability decisions.

[0021] Preferably, step S4 includes:

[0022] S41, a sliding window is used to smooth the temporal characteristics of consecutive video frames, and the width of the sliding window is set to... , The value is the product of the video frame rate and the smoothing time.

[0023] S42, based on intersection-union ratio Combining hand reliability and uncertainty with item reliability and uncertainty, the sliding time window width is... Within the scope of the data, a comprehensive reliability index for character interaction is constructed. , used to characterize The reliability of the interaction between the hand and the object being manipulated at any given moment.

[0024] Preferably, the overall reliability index of the character interaction is obtained by weighted summation of the intersection-over-union ratio, reliability, and uncertainty of multiple frames within the window; its calculation is as follows:

[0025] Current frame Index minus time offset Implement backtracking from the current frame Frame, denoted as ;calculate Corresponding crossover ratio .

[0026] pass calculate The corresponding degree of certainty, multiplied by The confidence level of the hand confidence level was obtained, and the confidence level of the manipulated object confidence level was calculated in the same way.

[0027] Within a sliding window, multiply the confidence scores of the hand confidence score by the confidence scores of the manipulated object confidence score at all times, and then perform an intersection-union comparison. And sum and take the average. .

[0028] Preferably, the determination of the threshold for the comprehensive reliability index of character interaction is specifically implemented as follows:

[0029] According to the settings threshold Confidence threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes Uncertainty threshold for hand confidence Threshold for uncertainty of reliability of the manipulated items .

[0030] pass Calculate the certainty threshold for hand reliability by... Calculate the certainty threshold for operational item reliability.

[0031] Will threshold Confidence threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes The certainty thresholds for hand-related reliability and object-related reliability are multiplied to construct a comprehensive reliability index threshold for human-human interaction. .

[0032] Preferably, step S5 includes:

[0033] S51, the hand recognition box obtained by the edge server. and the identification box for the manipulated items and the reliability of each bounding box , and the uncertainty of its reliability. and Uploaded to the cloud.

[0034] S52 will jointly construct a comprehensive reliability index for human interaction in the cloud from the bounding boxes obtained from the edge server, along with the reliability and uncertainty of the bounding boxes. ,when Greater than the threshold of the overall reliability index of character interaction At that time, it can be judged that Detecting work behavior steps at all times It happened; otherwise, it did not happen.

[0035] Preferably, step S6 includes:

[0036] S61, the action sequence finite state machine is constructed based on the sequence of work behaviors and the corresponding action logic in the work process; the sequence of work behavior steps obtained in step S52 according to the time sequence detection is formed. The input is fed into the finite state machine of the action sequence, and the process consistency is determined with the work behavior steps in step S1. If the consistency determination is passed, the work behavior sequence can reach the completion state along the compliant state transition path of the work process, and the work process is determined to be executed correctly; if the input work behavior sequence enters a non-compliant state or cannot reach the completion state, the work process is determined to be executed incorrectly.

[0037] S62, according to the state definition of the action sequence finite state machine described in step S61, it includes an initial state for representing the start of the work process, a compliant state that conforms to the path planning, an intermediate state for representing each key work action step, and a completion state for representing the completion of the work process; when the action sequence finite state machine receives a work action step input that does not meet the process transition conditions in any intermediate state, it automatically transitions to the non-compliant state.

[0038] Secondly, this invention provides an intelligent IoT evaluation system for work behavior processes based on interaction reliability, comprising the following modules:

[0039] The module for determining work behaviors and manipulated items is used to determine the personnel's work process. Based on the work content, the work process is divided into work behavior steps, and the manipulated items for personnel interaction are defined for each work behavior step.

[0040] The recognition module is used to build a reliability detection model for personnel and object targets, and to recognize the hands of workers and the objects they are handling based on the video frames of the operation.

[0041] The intersection-union ratio (IUGR) calculation module is used to obtain the IUGR of the operator's hand area and the operated items based on the hand and operated item recognition results.

[0042] The Human Interaction Comprehensive Reliability Index module is used to integrate the results of intersection-union comparison, hand recognition, and object manipulation recognition to jointly construct the Human Interaction Comprehensive Reliability Index and determine the threshold of the Human Interaction Comprehensive Reliability Index.

[0043] The task behavior evaluation module compares the overall reliability index of human interaction with the threshold of the overall reliability index of human interaction to determine whether the task behavior steps have occurred.

[0044] The task behavior evaluation result output module is used to obtain the task behavior step identification results in chronological order of the determined task behavior, then perform process consistency judgment with the task behavior steps, and output the evaluation results of the task process execution.

[0045] The beneficial effects of this invention are as follows:

[0046] This invention first acquires image data of personnel's work behavior, then uses an object detection model to identify people and objects in the images, outputting the reliability and uncertainty of the recognition boxes. The overlap ratio of the interaction between the person and object recognition boxes is calculated frame by frame. Using the overlap ratio, the reliability of the person and object recognition boxes, and the uncertainty, a comprehensive reliability index for human-object interaction is constructed to continuously determine the type of personnel's work behavior, thereby obtaining a sequence of work behaviors representing the work process. A finite state machine is used to evaluate the work behavior sequence to determine whether the sequence is consistent with the standardized work process. This invention combines object detection with a human-object recognition box interaction mechanism to explore the physical and spatiotemporal relationship between the worker and the operated object to achieve stable and interpretable behavior recognition. Simultaneously, this invention uses a finite state machine to comprehensively evaluate the work behavior process, thereby improving the efficiency of work process control. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate the invention and, together with the descriptions, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart of the application process;

[0049] Figure 2 This is a schematic diagram of the structure of an embodiment of this application;

[0050] Figure 3 This is a result diagram of the intelligent IoT evaluation method for work behavior processes based on interaction reliability;

[0051] Figure 4 The assessment results are illustrated in the diagram of the completed assignment.

[0052] Figure 5 This is a diagram illustrating the errors in the assessment results. Detailed Implementation

[0053] The present invention will be described in detail below with reference to the accompanying drawings and examples of welding operation evaluation. It should be noted that the described embodiments are only some, not all, of the embodiments of the technical solution of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0054] like Figure 1 As shown in the embodiments of this application, the intelligent IoT evaluation method for work behavior processes based on interaction reliability includes the following steps:

[0055] 101. Using a fixed-installation sensing camera to acquire image data of the workers, and based on the image data, acquiring recognition frames for the workers' hands and the objects they are handling, including: acquiring video data of the workers and the relevant environment, including different work scenarios and types; selecting a certain number of representative sample videos, and acquiring the work behavior steps corresponding to at least one object in each sample video, wherein the work behavior steps include... - Check the fire extinguishers, -Inspect the gasoline cans, -Inspect the welding machine, -Inspect the welding torch, - Check the grounding clamp, - Turn on the wall power. - Turn on the welding machine power. - Adjust the welding machine voltage, -Grounding clamp grounding, - Clean the weld beads, - Protective work, - Welding operations - Turn off the welding machine power. - Turn off the wall power. -Reset welding torch, -Reset grounding clamp, - Clean solder lines and - Cleaning, the items to be handled according to the corresponding work steps include: -Fire extinguisher, -Gasoline cans, -welder, - Welding torch, -Grounding clamp, - Wall power supply, - Welding machine power supply -Welding machine voltage regulator, -Grounding clamp, -Just refreshed -face mask, - Welding torch, - Welding machine power supply - Wall power supply, - Welding torch, -Grounding clamp, -Just refreshed and - Broom; By locating the recognition box position of each operation image frame in the sample video, the recognition box information of the operation person and object is obtained.

[0056] 102. Target Reliability Deep Learning Detection Model The worker's hand is identified in each frame of each sample video, and the hand region recognition box is detected. The system identifies the objects being manipulated in each frame of each sample video and detects the corresponding bounding boxes. And obtain the confidence level of each recognition box based on the model output. and and the uncertainty of its reliability and ,in , Indicates the total number of categories of items being manipulated; , representing the numbers of the left-hand and right-hand recognition boxes respectively.

[0057] The process of constructing the target reliability deep learning detection model is as follows:

[0058] Will By incorporating deep learning theory of evidence, we can modify the output to achieve category confidence and its uncertainty. Specifically, we... The classification branch of the head module in the structure has been structurally modified so that the model no longer outputs a score, but instead outputs the result after activation. Activated evidence Subsequently, based on the theory of evidence deep learning, evidence Mapped to Dirichlet Distribution parameters Then calculate the confidence level. And calculate the uncertainty of the reliability. ,in , This represents the total number of item categories being operated on, thus constructing an evidence-based multi-class prediction framework for subsequent reliability decisions.

[0059] 103. Subsequently, the image frames captured by the sensing camera are transmitted to the edge server, where the target detection model... Recognition frames for the left and right hands of workers during welding scene inspection And the identification box for the work operation items The ratio of the overlapping area to the union area is calculated to characterize the degree of spatial overlap between the hand and the object. The calculation formula is as follows:

[0060]

[0061] Among them, superscript Indicates the hand ( ), superscript Indicates items ( ); Indicates the frame number. Indicates the first Index of hand detection bounding boxes Indicates the first Index of item detection boxes. Indicates the first Hand in frame Recognition box area, Indicates the first Items in a frame Recognition box area, Indicates the first Hand in frame Recognition box With items Recognition box The overlapping area. In each frame image, for object recognition boxes of the same category, only the area between the object and any hand recognition box is retained. The pair with the highest value is used to calculate the overall reliability index of subsequent interpersonal interactions.

[0062] 104. A sliding window method is used to smooth the temporal characteristics of continuously processed video frames to eliminate instantaneous detection fluctuations; the sliding window width is set to... , The value is determined by the video frame rate to ensure the stability and real-time performance of time smoothing. With video frame rate The relationship can be expressed as ;in, This indicates the smoothing time, and its value ranges from 0.3 to 1.0 seconds.

[0063] Based on the obtained worker's hand area recognition box and the interactive object recognition box. and the reliability obtained , and the uncertainty of the corresponding reliability , Construct a comprehensive reliability index for character interaction The comprehensive reliability index of the character interaction is obtained by comprehensively analyzing the intersection-over-union ratio, reliability, and uncertainty of multiple frames within the window; its calculation expression is as follows:

[0064]

[0065] In this formula, Indicates the index of the current frame; This is the time offset, representing the backward movement relative to the current frame. frame.

[0066] According to the settings threshold Reliability threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes and the confidence uncertainty threshold , Constructing a comprehensive reliability index threshold for character interaction .

[0067] pass Calculate the certainty threshold for hand reliability by... Calculate the certainty threshold for operational item reliability, and... threshold Confidence threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes The certainty thresholds for hand-related reliability and object-related reliability are multiplied to construct a comprehensive reliability index threshold for human-human interaction. The calculation formula is as follows:

[0068]

[0069] 105. Continuous The comprehensive reliability index of human interaction constructed from Zhang's image frame information. Reliability index threshold for interaction with characters To make a comparison, when Greater than At that time, it can be judged that The corresponding operation action step is detected at all times; otherwise, it has not occurred.

[0070] 106. The identified work behavior steps are transmitted to the evaluation module in chronological order. The core of the evaluation module is the action sequence finite state machine. The action sequence finite state machine can be constructed based on the work behavior sequence and corresponding action logic of the work process. The work behavior sequence obtained by chronological detection is input into the action sequence finite state machine. In step S1, the work behavior steps are judged for process consistency. When the consistency judgment is passed, the work behavior sequence can reach the completion state along the compliant state transition path of the process, and the work process is judged to be executed correctly. If the input work behavior sequence enters a non-compliant state or cannot reach the completion state, the work process is judged to be executed incorrectly.

[0071] According to the state definition of the action sequence finite state machine, it includes an initial state to indicate the start of the process, a compliant state that conforms to the path planning, an intermediate state to indicate each key step, and a completion state to indicate the completion of the process; when the action sequence finite state machine receives an input of a work behavior step that does not meet the process transition conditions in any intermediate state, it automatically transitions to a non-compliant state.

[0072] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures or software modules for executing each function. It should be noted that, based on the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in a combination of hardware and computer software. Those skilled in the art can use different execution methods to implement a certain function for each specific application according to the corresponding requirements, but such implementation should not be considered to exceed the scope of this application.

[0073] The work behavior recognition system in the embodiments of this application will be described in detail below. Figure 2 This is a schematic diagram of one embodiment of the job behavior recognition system provided in this application. Figure 2As shown, the work behavior recognition device may include a work behavior and operation item determination module, a recognition module, an intersection-union calculation module, a human-human interaction comprehensive reliability index module, a work behavior evaluation module, and a work behavior evaluation result output module. The system comprises the following modules: a task behavior and operation item determination module, used to determine the personnel's work process, dividing the work process into task behavior steps according to the task content, and defining the operation items that the task behavior steps involve personnel interaction with; an identification module, used to build a reliability detection model for personnel and operation items, and to identify the personnel's hands and operation items based on the task video frames; an intersection-union (IU) calculation module, used to obtain the IU of the personnel's hand area and the operation items based on the hand and operation item identification results; a comprehensive reliability index module for human-person interaction, used to combine the IU, hand and operation item identification results to jointly construct a comprehensive reliability index for human-person interaction, and to determine the threshold of the comprehensive reliability index for human-person interaction; a task behavior evaluation module, used to compare the comprehensive reliability index for human-person interaction with the threshold of the comprehensive reliability index for human-person interaction, and to evaluate whether the task behavior steps have occurred; and a task behavior evaluation result output module, used to obtain the task behavior step identification results in chronological order of the determined task behaviors, and then perform a process consistency judgment with the task behavior steps to output the evaluation results of the task process execution.

[0074] like Figure 3 As shown, when the status is RUN, check the welding machine steps. (As indicated...) Figure 4 As shown, the job is complete when the status is DONE. Figure 5 As shown, when the status is "error", the job is faulty.

[0075] It should be noted that the above modules can be implemented either as software functional modules or as hardware. For the latter, this can be achieved by having all the above modules reside in the same processor; or by having the above modules reside in different processors in any combination.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still modify or replace the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

[0077] Experimental verification:

[0078] To verify the practical effectiveness of the intelligent IoT evaluation method for work behavior processes based on interaction reliability proposed in this invention, experimental verification was conducted in a welding operation scenario. The experiment mainly evaluated the method of this invention from aspects such as the recognition effect of work behavior steps, the ability to determine compliance of work processes, and the ability to identify abnormal processes.

[0079] (I) Data Introduction

[0080] The experimental data originated from video data of real welding operation scenarios, which were acquired by fixedly installed sensing cameras. The experimental videos covered the complete welding operation process and included abnormal operation scenarios such as missing steps. The videos included various operating items such as operators, welding machines, welding torches, grounding clamps, wall power supplies, masks, and brooms, reflecting typical interactions between personnel and items during the welding operation.

[0081] The collected video data uses resolution and frame rate parameters common in industrial settings, accurately reflecting the spatiotemporal characteristics of the work environment. This data can be used to verify the applicability and stability of the method described in this invention in real-world working environments.

[0082] (II) Brief Introduction to Experimental Methods and Evaluation Indicators

[0083] During the experiment, the video footage was processed frame by frame according to the method described in this manual. First, the target reliability detection model was used to identify the worker's hands and the objects being handled in the video frames, obtaining the corresponding bounding boxes, reliability, and reliability uncertainty. Then, the intersection-over-union ratio (IoU) between the worker's hand bounding box and the object's bounding box was calculated, and combined with the bounding box reliability and reliability uncertainty, a comprehensive reliability index for human interaction was constructed. A sliding window was used to smooth the continuous video frames over time. When the comprehensive reliability index for human interaction exceeded a set threshold, the corresponding work action step was considered complete.

[0084] After obtaining the sequence of work action steps arranged in chronological order, the sequence of work action steps is input into the action sequence finite state machine to perform a compliance assessment of the work process and determine whether the work process meets the preset work specification requirements.

[0085] To evaluate the experimental results, the following evaluation indicators were used:

[0086] 1. Step recognition accuracy: refers to the proportion of the task behavior steps determined by the comprehensive reliability index of human interaction that are consistent with the actual task behavior steps.

[0087] 2. Accuracy rate of process compliance assessment: This refers to the percentage of complete work process compliance assessments that are consistent with the actual situation.

[0088] 3. Abnormal process identification rate: refers to the percentage of successful identification of abnormal work processes with missing steps.

[0089] The above evaluation indicators can comprehensively assess the technical effectiveness of the method of the present invention from two aspects: operation behavior identification and operation process evaluation.

[0090] (III) Experimental Results

[0091] To demonstrate the technical effectiveness of the method of this invention, a comparative experiment was conducted with a job behavior recognition method that does not incorporate a process constraint mechanism. In the experiment, the completion criterion for job behavior steps was determined by the overall reliability index of the human-human interaction exceeding a preset threshold after smoothing through a sliding window. The resulting sequence of job behavior steps was then input into an action sequence finite state machine for process compliance evaluation. The experimental results are shown in Table 1.

[0092] Table 1 Comparison of experimental results from different methods

[0093]

[0094] Among them, the step recognition accuracy rate is based on the recognition results of all operational steps in the experimental data; the process compliance judgment accuracy rate is based on the compliance / non-compliance judgment results of process instances; and the abnormal process recognition rate is based on the recognition results of abnormal process instances.

[0095] The experimental results show that, compared with methods that do not introduce process constraints, the method of the present invention shows improvements in step identification accuracy, process compliance judgment accuracy, and abnormal process identification rate, which can verify the effectiveness of the method of the present invention for work behavior process evaluation.

[0096] (iv) Analysis and explanation of experimental results

[0097] Experimental results show that determining work steps based on the comprehensive reliability index of human-human interaction can maintain the stability of step recognition results even in the presence of short-term occlusion or rapid tool movement, thus providing reliable input for subsequent compliance assessment of work processes. This is mainly due to the fact that this invention models the spatial interaction relationship between people and objects through the comprehensive reliability index of human-human interaction and combines it with a sliding window mechanism in the time dimension, effectively reducing the impact of instantaneous detection errors on the work behavior recognition results.

[0098] Meanwhile, by introducing a finite state machine for action sequences to dynamically constrain the sequence of work behavior steps, this invention can correctly complete the process determination when multiple legal work processes coexist, and effectively identify abnormal work processes with missing steps, thereby verifying the practical effectiveness and engineering feasibility of the method of this invention in intelligent evaluation of work behavior processes.

Claims

1. A smart IoT evaluation method for work behavior processes based on interaction reliability, characterized in that, Includes the following steps: S1, Determine the personnel work process, divide the work process into work behavior steps according to the work content, and define the operation items for the work behavior steps and personnel interaction; S2, Construct a reliability detection model for personnel and object targets, and identify the hands of workers and the objects they are handling based on the operation video frames; S3, based on the hand and object recognition results, obtain the crossover ratio of the operator's hands and the objects being operated; S4 integrates the results of cross-comparison, hand and object recognition to jointly construct a comprehensive reliability index for human interaction and determine the threshold of the comprehensive reliability index for human interaction. S5 compares the overall reliability index of human interaction with the threshold of the overall reliability index of human interaction to judge whether the task behavior steps have occurred. S6 identifies the identified work steps in chronological order based on the determined work actions, then performs a process consistency determination with the work action steps, and outputs the evaluation results of the work process execution.

2. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: S11. Based on the work operation standards, formulate the preliminary work process and divide the process into work action steps. ; S12, Determine the steps of the work action. The items are manipulated by personnel and defined according to the work behavior step index. .

3. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 2, characterized in that, The specific implementation process of step S2 is as follows: S21, Construct a deep learning detection model for target reliability. ,Model The model performs object detection and classification prediction on the video frames of the operation, outputting the bounding box of the operation item, the corresponding confidence level, and the uncertainty of the confidence level. This is achieved by introducing an uncertainty modeling mechanism into the target detection network; S22, Install fixed sensing cameras in the work area to collect work actions and steps in real time. middle Time-based operation video frames And use the detection model deployed on the edge server. For the video frames of the operation Hands of workers and items handled Perform recognition and obtain the hand recognition frames of the workers respectively. and the identification box for the manipulated items ,in , , , , , , , Representing the recognition box and The x-coordinate of the top left corner, the y-coordinate of the top left corner, the width of the recognition box, and the height of the recognition box; S23, Detection Model Simultaneously output the reliability of the hand recognition bounding box and the object manipulation recognition bounding box. , And the uncertainty of the corresponding reliability. and .

4. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 3, characterized in that, The specific implementation process of the uncertainty modeling mechanism is as follows: The head classification branch of the object detection algorithm is structurally modified so that it no longer outputs a score, but instead outputs the result after the activation function. Activated evidence ; Subsequently, based on the theory of evidence deep learning, evidence Mapped to Dirichlet Distribution parameters Then through parameters With all parameters Calculate the confidence level by the sum of the proportions and through With all parameters The uncertainty of reliability is calculated based on the total percentage. , This represents the total number of categories of work items, and an evidence-based multi-classification prediction framework is constructed for subsequent reliability decisions.

5. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 4, characterized in that, The specific implementation process of jointly constructing the comprehensive reliability index of character interaction is as follows: S41, a sliding window is used to smooth the temporal characteristics of consecutive video frames, and the width of the sliding window is set to... , The value is the product of the video frame rate and the smoothing time. S42, based on intersection-union ratio Combining hand reliability and uncertainty with item reliability and uncertainty, the sliding time window width is... Within the scope of the data, a comprehensive reliability index for character interaction is constructed. , used to characterize The reliability of the interaction between the hand and the object being manipulated at any given moment.

6. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 5, characterized in that, The overall reliability index of the character interaction is obtained by weighting and combining the intersection-over-union ratio, reliability and uncertainty of multiple frames within the window; The calculation is as follows: Current frame Index minus time offset Implement backtracking from the current frame Frame, denoted as ;calculate Corresponding crossover ratio ; pass calculate The corresponding degree of certainty, multiplied by The certainty of the hand reliability was obtained, and the certainty of the operational item reliability was calculated in the same way; Within a sliding window, multiply the certainty of hand reliability by the certainty of item reliability at all times, and then perform an intersection-comparison (OCC) algorithm. And sum and take the average. .

7. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 6, characterized in that, The specific implementation of determining the threshold for the comprehensive reliability index of character interaction is as follows: According to the settings threshold Confidence threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes Uncertainty threshold for hand confidence Threshold for uncertainty of reliability of the manipulated items ; pass Calculate the certainty threshold for hand reliability by... Calculate the certainty threshold for operational item reliability; Will threshold Confidence threshold for personnel hand recognition bounding boxes , Confidence threshold for object recognition boxes The certainty thresholds for hand reliability and object reliability are multiplied to construct a comprehensive reliability index threshold for human-human interaction. .

8. The intelligent IoT evaluation method for work behavior processes based on interaction reliability according to claim 7, characterized in that, The specific implementation process of step S5 is as follows: S51, the hand recognition box obtained by the edge server. and the identification box for the manipulated items and the reliability of each bounding box , and the uncertainty of its reliability. and Upload to the cloud; S52 will jointly construct a comprehensive reliability index for human interaction in the cloud from the bounding boxes obtained from the edge server, along with the reliability and uncertainty of the bounding boxes. ,when Greater than the indicator threshold At that time, it can be judged that Detecting work behavior steps at all times It happened; otherwise, it did not happen.

9. The IoT-based evaluation method for work behavior processes based on interaction reliability according to claim 8, characterized in that, The specific implementation process of step S6 is as follows: S61, the action sequence finite state machine is constructed based on the sequence of work behaviors and the corresponding action logic in the work process; the sequence of work behavior steps obtained in step S52 according to the time sequence detection is formed. The input is fed into the finite state machine of the action sequence, and a process consistency check is performed with the work behavior steps. If the consistency check passes, the work behavior sequence can reach the completion state along the compliant state transition path of the work process, and the work process is judged to be executed correctly. If the input work behavior sequence enters a non-compliant state or cannot reach the completion state, the work process is judged to be executed incorrectly. S62, according to the state definition of the action sequence finite state machine, includes the initial state for representing the start of the work process, the compliance state for conforming to the path planning, the intermediate state for representing each key work action step, and the completion state for representing the completion of the work process. When the action sequence finite state machine receives a job action step input that does not meet the process transition conditions in any intermediate state, it automatically transitions to the non-compliant state.

10. A work behavior workflow intelligent IoT evaluation system based on interaction reliability, used to implement the work behavior workflow intelligent IoT evaluation method according to any one of claims 1 to 9, characterized in that, Includes the following modules: The module for determining work behaviors and manipulated items is used to determine the personnel's work process. Based on the work content, the work process is divided into work behavior steps, and the manipulated items that the personnel interact with are defined for each work behavior step. The recognition module is used to build a reliability detection model for personnel and object targets, and to recognize the hands of workers and the objects they are handling based on the video frames of the operation. The intersection-union ratio (IUU) calculation module is used to obtain the IUU of the operator's hand area and the operated items based on the hand and operated item recognition results; The Human Interaction Comprehensive Reliability Index module is used to integrate the results of intersection-union comparison, hand recognition, and object manipulation recognition to jointly construct the Human Interaction Comprehensive Reliability Index and determine the threshold of the Human Interaction Comprehensive Reliability Index. The task behavior evaluation module is used to compare the comprehensive reliability index of human interaction with the threshold of the comprehensive reliability index of human interaction to evaluate whether the task behavior steps have occurred. The task behavior evaluation result output module is used to obtain the task behavior step identification results in chronological order of the determined task behavior, then perform process consistency judgment with the task behavior steps, and output the evaluation results of the task process execution.

Images