Operating personnel behavior analysis method and system based on trajectory analysis

Abstract

The invention relates to the field of industrial vision, in particular to an operator behavior analysis method and system based on trajectory analysis. The method comprises the following steps: acquiring an action video of an operator, performing action feature extraction on a picture of the action video, performing fitting processing on an action feature point set, and setting a paragraph step length, the action track curve is divided into paragraphs according to the paragraph step length, the space distance between action track points in the action track paragraphs and standard track points in the standard operation track is calculated, the dynamic parameter difference is analyzed according to the action track paragraphs and the standard track paragraphs matched with the action track paragraphs, and a behavior monitoring result is obtained according to the dynamic parameter difference. The system comprises a video acquisition device, a video identification model, a curve fitting module, a paragraph matching module and an anomaly detection module. The method is lower in cost, cannot be interfered by an industrial field environment, and is wider in applicability. And the track is dynamically divided into track paragraphs, so that the applicability of the method is improved.

Description

Technical Field

[0001] The present application relates to the field of industrial vision, and in particular to a method and system for analyzing operator behavior based on trajectory analysis. Background Art

[0002] In industrial production, it is necessary to analyze the behavior of operators. Currently, this can usually be achieved using technologies such as manual observation, wearable device technology, and video analysis technology. Among them, manual observation involves managers or technical analysts observing and recording the behavior of operators with their naked eyes, which is inefficient and prone to omissions. Wearable device technology records behavior through sensors worn on operators, but the cost of wearable devices is too high, and the parameters that sensors can collect are limited. Video analysis technology collects two-dimensional action videos of operators through cameras, and then analyzes the behavior of operators through image content recognition. However, image content recognition relies on a powerful image content recognition model, which has a high training cost, and the image content recognition model has high requirements for image quality. Factors such as dust occlusion and workpiece reflection may interfere with the effect of image content recognition at industrial production sites.

[0003] In summary, existing technologies for analyzing operator behavior have significant defects. Summary of the invention

[0004] The technical problem to be solved by this application is that it is difficult to analyze the behavior of operators.

[0005] In a first aspect, the present application provides a method for analyzing operator behavior based on trajectory analysis, comprising the following steps: Obtaining an action video of the operator; Extracting action features from the action video to obtain an action feature point set; Performing fitting processing on the action feature point set to obtain an action trajectory curve; Setting a paragraph step length, dividing the motion trajectory curve into paragraphs according to the paragraph step length, and obtaining motion trajectory paragraphs; Calculating the spatial distance between the action trajectory point in the action trajectory segment and the standard trajectory point in the standard operation trajectory to determine the standard trajectory segment that matches the action trajectory segment; Dynamic parameter differences are analyzed according to the action trajectory segment and the standard trajectory segment that matches it, and behavior monitoring results are obtained according to the dynamic parameter differences. The dynamic parameters include: trajectory curvature, trajectory point speed and trajectory point spatial distance. The behavior trajectory analysis result includes: whether there are abnormal trajectory points and the abnormal reasons of the abnormal trajectory points.

[0006] Specifically, the action video includes a picture and distance data, the coordinates of the action feature points in the action feature point set include the distance data, and the step of extracting action features from the picture of the action video includes: Performing action area recognition on the picture to obtain an operation area; Extracting features from the picture to obtain a candidate action feature point set; The candidate action feature points in the candidate action feature point set are screened according to the operation area and the distance data to obtain the action feature point set.

[0007] Specifically, the step of performing fitting processing on the action feature point set includes: Arranging the action feature points in the action feature point set in time sequence to obtain time sequence action feature points, and adjusting the discreteness of the time sequence action feature points to obtain optimized time sequence action feature points; The motion trajectory curve is obtained by fitting the optimized time sequence motion feature points.

[0008] Specifically, the step of adjusting the discreteness of the action timing feature points includes: Correcting the temporal action feature points whose offset is greater than the offset threshold to obtain corrected feature points; Interpolation processing is performed on the corrected feature points whose point spacing is greater than the spacing threshold to obtain the optimized time sequence action feature points.

[0009] Specifically, the step of calculating the spatial distance between the action track point in the action track segment and the standard track point in the standard operation track includes: Calculate the spatial distance between the action trajectory points in the action trajectory segment and the standard trajectory points in the standard operation trajectory, mark the action trajectory points and the standard trajectory points whose spatial distance is less than a determination threshold as the first paragraph point and the second paragraph point, intercept the standard trajectory segment according to the paragraph step length of the action trajectory segment, and the position of the first paragraph point in the action trajectory segment is the same as the position of the second paragraph point in the action trajectory segment.

[0010] In the second aspect, the present application proposes a system for analyzing operator behavior based on trajectory analysis, comprising: A video acquisition device, used to obtain the action video of the operator; A video recognition model is used to extract action features from the action video to obtain an action feature point set; A curve fitting module, used for fitting the action feature point set to obtain an action trajectory curve; A paragraph matching module is used to set a paragraph step length, divide the motion trajectory curve into paragraphs according to the paragraph step length, and obtain motion trajectory paragraphs; calculate the spatial distance between the motion trajectory points in the motion trajectory paragraphs and the standard trajectory points in the standard operation trajectory, so as to determine the standard trajectory paragraph that matches the motion trajectory paragraph; The anomaly detection module is used to analyze the dynamic parameter difference between the action trajectory segment and the standard trajectory segment that matches it, and obtain the behavior monitoring result according to the dynamic parameter difference, wherein the dynamic parameters include: trajectory curvature, trajectory point speed and trajectory point spatial distance, and the behavior trajectory analysis result includes: whether there is an abnormal trajectory point and the abnormal reason of the abnormal trajectory point.

[0011] Specifically, the video acquisition device includes an infrared distance measurement module, the action video includes a picture and distance data, and the coordinates of the action feature points in the action feature point set include the distance data.

[0012] Specifically, a time sequence alignment submodule is provided in the curve fitting module, which is used to arrange the action feature points in the action feature point set in time sequence to obtain time sequence action feature points.

[0013] Specifically, the curve fitting module is also provided with a curve optimization submodule for correcting the timing action feature points whose offset is greater than the offset threshold to obtain corrected feature points; and interpolating the corrected feature points whose point spacing is greater than the spacing threshold to obtain optimized timing action feature points.

[0014] As a preferred solution, the abnormality detection module further includes a display submodule and a voice submodule, which is used for when the behavior trajectory analysis result contains the abnormal trajectory point, the voice submodule plays an abnormal alarm signal, and the display submodule displays the behavior trajectory analysis result.

[0015] This application has the following technical effects: A method for analyzing the behavior of operators through trajectory analysis is proposed. This method does not require wearable devices and image content recognition models, has lower costs, is not affected by industrial site environments, and has wider applicability. By dynamically dividing the trajectory into segments during trajectory analysis, the method is made more flexible, thereby improving the applicability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-restrictive manner, and the same or corresponding numbers represent the same or corresponding parts.

[0017] Figure 1 It is a flow chart of a method for analyzing operator behavior based on trajectory analysis in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0019] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0020] In industrial sites, it is necessary to analyze the behavior of operators, but existing technologies such as manual observation, wearable device technology, video analysis technology, etc. have been found to have significant defects in the implementation process, so it is difficult for existing technologies to analyze the behavior of operators. The inventors of this application have found through research that when operators perform operating actions, their motion trajectories are relatively fixed and easy to capture. Compared with recognizing image content, it is obviously easier to implement to recognize motion trajectories. Based on this, the inventors collected sample videos of operator actions at industrial sites, used the action sample videos to train a video recognition model that can identify the operating area and action feature points from the action videos, established a standard action trajectory library, and then further proposed a method and system for analyzing operator behavior based on trajectory analysis, and gave an embodiment.

[0021] In this embodiment, a three-dimensional camera is installed at an appropriate position of the workstation to capture the operator's working action video in real time. The three-dimensional camera has a built-in infrared sensor, so the captured action video contains not only the picture but also the distance data. The analysis result obtained by analyzing the video picture with distance data has better accuracy and reliability.

[0022] After capturing the action video, follow the steps below: Figure 1 The steps shown in this article are to conduct an operator behavior analysis, including: S1. Obtain the action video of the operator. The video shot by the 3D camera has a high frame rate and richer details.

[0023] S2, extracting action features from the action video to obtain an action feature point set. Because the present embodiment uses a three-dimensional camera, the action video not only has a picture but also distance data, and the coordinates of the action feature points in the action feature point set contain distance data. If a two-dimensional camera is used for shooting, the action video only has a picture, and the coordinates of the action feature points in the action feature point set do not contain distance data. The model used for action feature extraction in this embodiment is trained based on YoloV5, because if YoloV5 is directly used to detect key points of the human body, it will be found that it is difficult to ensure a stable recognition rate in an industrial environment. Therefore, the hand annotation is re-performed on the operation video sample, and then the model is trained based on the hand annotation, so that the model has a more prominent effect of extracting hand feature points, and the processing efficiency of the model can reach 20FPS, the inference time is usually around ten milliseconds, the computational complexity is within 10G, and the timeliness of the result is high. Since the initial video contains a lot of frames, the data processing volume will be relatively large, so the feature points can be clustered in chronological order, and timestamps are set for the feature points after clustering. According to the identified action area, the action feature points that are not in the operation area are removed. At the same time, the noise points and interference points are determined according to the distance data, and the feature points are further screened to reduce the amount of data that can be processed while also improving the accuracy of subsequent calculations. If a two-dimensional camera is used for shooting, such screening cannot be performed. Due to the excessive number of action feature points and excessive noise in the action feature point concentration, the accuracy of the obtained trajectory will be reduced, and in the subsequent distance calculation, the accuracy of the calculation result will also decrease due to the lack of distance coordinates.

[0024] S3, fitting the action feature point set to obtain the action trajectory curve. Specifically, it includes obtaining the time-series action feature points according to the time sequence of the points based on the timestamp, and the time-series action feature points at this time are coordinate points regularly arranged in three-dimensional space. The time-series action feature points with excessive offset are corrected to obtain the corrected feature points. For example, these points with large offset can be fine-tuned according to the distance data. In addition, if the spatial distance between two points is too large, the missing points are filled based on spatial interpolation methods such as Bezier curves or linear interpolation. Linear interpolation is simple and fast, suitable for simple transition effects, but cannot handle complex curve shapes. Although Bezier curve interpolation has a high computational complexity, it can generate smooth and complex curve shapes. According to the curvature of the curve, the Bezier curve and linear interpolation method are selected adaptively. When smooth transitions and complex shapes are required, Bezier curve interpolation is more suitable, and when simple transition effects need to be generated quickly, linear interpolation is more suitable to make the fitted curve smooth and coherent.

[0025] S4. Set the paragraph step length, divide the action trajectory curve into paragraphs according to the paragraph step length, and obtain the action trajectory paragraphs. In this embodiment, the paragraph step length is set to 10 points, and the length of the paragraph step length can be dynamically adjusted according to different operation scenarios. Compared with analyzing the entire track, analyzing the paragraphs after dividing them can more easily match the corresponding paragraphs in the standard trajectory library, which improves the applicability and avoids the inability to match and analyze the trajectory due to partial deviation or loss of the trajectory.

[0026] S5. Calculate the spatial distance between the action trajectory points in the action trajectory segment and the standard trajectory points in the standard operation trajectory to determine the standard trajectory segment that matches the action trajectory segment. When the distance between any point in the action trajectory segment and the spatial distance between any point in the standard operation trajectory is less than a threshold, the segment is successfully matched. Take the points with the number of paragraph steps in the standard operation trajectory as the paragraph that matches the action trajectory segment. By dynamically adjusting the paragraph step, the trajectory is divided into paragraphs, which improves the flexibility of trajectory matching.

[0027] S6. Analyze the dynamic parameter differences based on the action trajectory segment and the standard trajectory segment that matches it, and obtain the behavior monitoring results based on the dynamic parameter differences. Calculate the spatial distance between each point in the action trajectory segment and the corresponding point in the standard segment trajectory point by point to determine whether the point is an abnormal point. The speed of the trajectory point can be calculated by the distance and timestamp between the two points before and after the point. In addition, the curvature of the entire trajectory segment can be calculated to compare with the standard trajectory segment. The behavior trajectory analysis results include: whether there are abnormal trajectory points and the abnormal reasons for the abnormal trajectory points.

[0028] The analysis method of this embodiment requires a corresponding analysis system, including a three-dimensional camera, for obtaining an action video of an operator. The three-dimensional camera includes an infrared ranging module, the action video includes a picture and distance data, and the coordinates of the action feature points in the action feature point set include the distance data.

[0029] The video recognition model is used to extract action features from action video images to obtain action feature point sets; The curve fitting module is used to fit the action feature point set to obtain the action trajectory curve. The curve fitting module is provided with a time sequence alignment submodule, which is used to arrange the action feature points in the action feature point set in time sequence to obtain time sequence action feature points. The curve fitting module is also provided with a curve optimization submodule, which is used to correct the time sequence action feature points whose offset is greater than the offset threshold to obtain the corrected feature points; and to interpolate the corrected feature points whose point spacing is greater than the spacing threshold to obtain the optimized time sequence action feature points.

[0030] The paragraph matching module is used to set the paragraph step length, divide the action trajectory curve into paragraphs according to the paragraph step length, and obtain the action trajectory paragraph; calculate the spatial distance between the action trajectory point in the action trajectory paragraph and the standard trajectory point in the standard operation trajectory to match the standard trajectory paragraph corresponding to the action trajectory paragraph.

[0031] The anomaly detection module is used to calculate the judgment parameters based on the action trajectory segments and the standard trajectory segments, and obtain the behavior trajectory analysis results based on the judgment parameters. The judgment parameters include: trajectory curvature, trajectory point speed and trajectory point spatial distance. The behavior trajectory analysis results include: whether there are abnormal trajectory points and the abnormal reasons for the abnormal trajectory points. The anomaly detection module also includes a display submodule and a voice submodule, which are used to provide timely voice feedback to operators or managers when there are abnormal trajectory points in the behavior trajectory analysis results, and display the abnormal information on the screen. The obtained action trajectory segments and abnormal conditions are stored in the database, and process analysts or technical R&D personnel can use these action trajectory segments to perform behavior improvement design and process flow optimization design.

[0032] Although this specification has shown and described a plurality of embodiments of the present application, it is obvious to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein may be adopted.

[0033] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for analyzing operator behavior based on trajectory analysis, characterized in that: The following steps are involved: Obtaining an action video of the operator; Extracting action features from the action video to obtain an action feature point set; Performing fitting processing on the action feature point set to obtain an action trajectory curve; Setting a paragraph step length, dividing the motion trajectory curve into paragraphs according to the paragraph step length, and obtaining motion trajectory paragraphs; Calculating the spatial distance between the action trajectory point in the action trajectory segment and the standard trajectory point in the standard operation trajectory to determine the standard trajectory segment that matches the action trajectory segment; Dynamic parameter differences are analyzed according to the action trajectory segment and the standard trajectory segment that matches it, and behavior monitoring results are obtained according to the dynamic parameter differences. The dynamic parameters include: trajectory curvature, trajectory point speed and trajectory point spatial distance. The behavior trajectory analysis result includes: whether there are abnormal trajectory points and the abnormal reasons of the abnormal trajectory points.

2. The operator behavior analysis method according to claim 1, characterized in that: The action video includes a picture and distance data, the coordinates of the action feature points in the action feature point set include the distance data, and the step of extracting action features from the picture of the action video includes: Performing action area recognition on the picture to obtain an operation area; Extracting features from the picture to obtain a candidate action feature point set; The candidate action feature points in the candidate action feature point set are screened according to the operation area and the distance data to obtain the action feature point set.

3. The operator behavior analysis method according to claim 1, characterized in that: The step of performing fitting processing on the action feature point set comprises: Arranging the action feature points in the action feature point set in time sequence to obtain time sequence action feature points, and adjusting the discreteness of the time sequence action feature points to obtain optimized time sequence action feature points; The motion trajectory curve is obtained by fitting the optimized time sequence motion feature points.

4. The operator behavior analysis method according to claim 3, characterized in that: The step of adjusting the discreteness of the action timing feature points comprises: Correcting the temporal action feature points whose offset is greater than the offset threshold to obtain corrected feature points; Interpolation processing is performed on the corrected feature points whose point spacing is greater than the spacing threshold to obtain the optimized time sequence action feature points.

5. The operator behavior analysis method according to claim 1, characterized in that: The step of calculating the spatial distance between the action track point in the action track segment and the standard track point in the standard operation track comprises: Calculate the spatial distance between the action trajectory points in the action trajectory segment and the standard trajectory points in the standard operation trajectory, mark the action trajectory points and the standard trajectory points whose spatial distance is less than a determination threshold as the first paragraph point and the second paragraph point, intercept the standard trajectory segment according to the paragraph step length of the action trajectory segment, and the position of the first paragraph point in the action trajectory segment is the same as the position of the second paragraph point in the action trajectory segment.

6. A worker behavior analysis system based on trajectory analysis, operated using the method according to any one of claims 1 to 5, characterized in that: include: A video acquisition device, used to obtain the action video of the operator; A video recognition model is used to extract action features from the action video to obtain an action feature point set; A curve fitting module, used for fitting the action feature point set to obtain an action trajectory curve; A paragraph matching module is used to set a paragraph step length, divide the motion trajectory curve into paragraphs according to the paragraph step length, and obtain motion trajectory paragraphs; calculate the spatial distance between the motion trajectory points in the motion trajectory paragraphs and the standard trajectory points in the standard operation trajectory, so as to determine the standard trajectory paragraph that matches the motion trajectory paragraph; The anomaly detection module is used to analyze the dynamic parameter difference between the action trajectory segment and the standard trajectory segment that matches it, and obtain the behavior monitoring result according to the dynamic parameter difference, wherein the dynamic parameters include: trajectory curvature, trajectory point speed and trajectory point spatial distance, and the behavior trajectory analysis result includes: whether there is an abnormal trajectory point and the abnormal reason of the abnormal trajectory point.

7. The operator behavior analysis system according to claim 6, characterized in that: The video acquisition device includes an infrared distance measurement module, the action video includes a picture and distance data, and the coordinates of the action feature points in the action feature point set include the distance data.

8. The operator behavior analysis system according to claim 6, characterized in that: The curve fitting module is provided with a time sequence alignment submodule, which is used to arrange the action feature points in the action feature point set according to time sequence to obtain time sequence action feature points.

9. The operator behavior analysis system according to claim 8, characterized in that: The curve fitting module is also provided with a curve optimization submodule, which is used to correct the timing action feature points whose offset degree is greater than the offset threshold to obtain the corrected feature points; and to interpolate the corrected feature points whose point spacing is greater than the spacing threshold to obtain the optimized timing action feature points.

10. The operator behavior analysis system according to claim 6, characterized in that: The abnormality detection module also includes a display submodule and a voice submodule, which is used for when the behavior trajectory analysis result has the abnormal trajectory point, the voice submodule plays an abnormal alarm signal, and the display submodule displays the behavior trajectory analysis result.

Images