Process operation behavior detection method and device and electronic equipment

Abstract

The invention discloses a process operation behavior detection method and device and electronic equipment, and relates to the technical field of action normalization detection.The method comprises the steps that a video sequence in a process operation scene is obtained, and a target edge corresponding to process operation of a display panel area in an image is determined; and determining a target process operation area in the image according to the target edge. For each frame of image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; determining operation track information according to a target key point corresponding to the operation gesture in each frame of image; according to the intersection relation between the operation track information and each sampling state line and the gesture motion direction corresponding to the operation track information, the sampling state value of each sampling state line is determined, and finally according to the sampling state value of each sampling state line, the process operation behavior detection result is determined. Therefore, an efficient and accurate procedure operation behavior detection scheme is provided.

Description

Technical Field

[0001] The present application relates to the technical field of action normativeness detection, and in particular to a method, device and electronic equipment for detecting process operation behavior. Background Art

[0002] The factory worker operation behavior recognition technology integrates cutting-edge technologies such as computer vision, machine learning, and deep learning, aiming to improve production safety and efficiency. By deploying high-precision cameras on the factory site to capture video streams, these systems can analyze worker behavior in real time to determine whether they follow standard operating procedures (SOPs). In the assembly process of electronic devices, there is a process of attaching tape to the edge of the display panel of the electronic device. When workers do not operate in a standardized manner, it will cause quality problems in the electronic equipment, such as the pressure-loss operation after the panel tape is attached, which may cause the backlight tape to debond, light leakage, and other quality accidents.

[0003] Related technologies have proposed a solution to detect the standardization of workers' actions based on deep learning algorithms. Generally, images of workers working are collected, and then features are extracted from the images based on action recognition models to output prediction results of whether the actions are compliant. However, the tape application process is generally a series of actions, and video sequences need to be collected to determine the compliance of the process. Processing video sequences based on action recognition models requires a large amount of data processing, high resource consumption, and low processing efficiency. In addition, the action recognition model has low processing performance for video sequences, resulting in poor accuracy in identifying process operation behaviors. Summary of the invention

[0004] The present application provides a method, device and electronic equipment for detecting process operation behavior, so as to solve the problem of low efficiency and poor accuracy of behavior detection in the prior art.

[0005] In a first aspect, the present application provides a method for detecting process operation behavior, the method comprising:

[0006] Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area;

[0007] For each frame of the image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; and determining operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0008] According to the intersection relationship between the operation trajectory information and the sampling state lines and the gesture movement direction corresponding to the operation trajectory information, the sampling state values ​​of the sampling state lines are determined; according to the sampling state values ​​of the sampling state lines, the process operation behavior detection result is determined.

[0009] In a second aspect, the present application provides a device for detecting process operation behavior, the device comprising:

[0010] A first determination module is used to obtain a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area;

[0011] A second determination module is used to detect, for each frame of the video sequence, a target key point corresponding to the operation gesture in the image; and determine the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0012] The behavior detection module is used to determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; and determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0013] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0014] Memory, used to store computer programs;

[0015] The processor is used to implement the method steps when executing the program stored in the memory.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described are implemented.

[0017] The above technical solution has the following advantages or beneficial effects:

[0018] During the operation of the tape attaching process at the edge of the display panel, the operation edge of the display panel area will be operated. The present application first obtains a video sequence in the process operation scene, and for any image in the video sequence, determines the target edge corresponding to the process operation of the display panel area in the image, and determines the target process operation area according to the target edge. Perform process operation behavior detection in the target process operation area to avoid environmental interference outside the target process operation area and improve the accuracy of process operation behavior detection. In addition, each sampling state line in the target process operation area is determined, and the process operation behavior detection result is determined according to the sampling state value of each sampling state line, without processing the entire trajectory point, thereby improving the efficiency of process operation behavior detection. Specifically, for each frame of the video sequence, the target key point corresponding to the operation gesture in the image is detected; according to the target key point corresponding to the operation gesture in each frame of the image, the operation trajectory information is determined; and then according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information, the sampling state value of each sampling state line is determined, and finally, according to the sampling state value of each sampling state line, the process operation behavior detection result is determined. Thus, an efficient and accurate process operation behavior detection scheme is provided. The present invention solves the problems in related technologies of processing video sequences based on action recognition models, large data processing volume, high resource consumption, low processing efficiency, and low processing performance of action recognition models for video sequences, resulting in poor accuracy in identifying process operation behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the process of the first process operation behavior detection provided in this application;

[0021] Figure 2 A schematic diagram of a process for determining a target edge of a display panel area provided in the present application;

[0022] Figure 3 This is an example segmentation effect diagram of a TV panel provided in this application;

[0023] Figure 4 A schematic diagram of each vertex for determining the display panel area provided in this application;

[0024] Figure 5 A schematic diagram of a target edge for determining a display panel area provided in this application;

[0025] Figure 6 A schematic diagram of the second process operation behavior detection process provided by this application;

[0026] Figure 7 A schematic diagram of dividing each sampling state line in the target process operation area provided in this application;

[0027] Figure 8 A schematic diagram of the third process operation behavior detection process provided by this application;

[0028] Fig. 9 A schematic diagram of the process of detecting target key points corresponding to operation gestures in an image provided by the present application;

[0029] Fig.10 A schematic diagram of the fourth process operation behavior detection process provided by this application;

[0030] Fig.11 A schematic diagram of the fifth process operation behavior detection process provided by this application;

[0031] Fig.12 A schematic diagram of the sixth process operation behavior detection process provided by this application;

[0032] Fig.13 The process operation behavior detection flow chart provided for this application;

[0033] Fig.14 Schematic diagram of the operation area division provided for this application;

[0034] Fig.15 Schematic diagram of the operation trajectory information provided for this application;

[0035] Fig.16 This is the recognition effect diagram corresponding to action 1 provided in this application;

[0036] Fig.17 A schematic diagram of the structure of the device for detecting process operation behavior provided in this application;

[0037] Fig.18 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0038] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0039] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0040] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0041] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0042] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0044] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0045] Figure 1 This is a schematic diagram of the process of the first process operation behavior detection provided in this application, which includes the following steps:

[0046] S101: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area;

[0047] S102: for each frame of the image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; and determining operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0048] S103: Determine sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0049] The method for detecting process operation behavior provided in the present application is applied to an electronic device, which may be a terminal device such as a PC or a computer, or a server.

[0050] In the process operation scene, a camera installed in a fixed position is generally arranged, and the field of view of the camera needs to cover the scene of the worker's process operation. Specifically, it is necessary to cover the area where the workers performing the process operation are located and the area where the operated display panel is located. The camera collects the video sequence of the worker performing the process operation and sends the video sequence to the electronic device. It should be noted that the process operation in this application includes the process operation of taping the edge of the display panel. When the tape is taped to the edge of the display panel, the display panel remains stationary, and the worker generally performs the tape taping operation on the edge of the display panel by holding a tool. Display panels include but are not limited to display panels of televisions, display panels of computers, and display panels of other smart devices.

[0051] The electronic device obtains a video sequence in a process operation scene. Since the position of the display panel and the position of the camera are fixed, the position of the display panel area in each frame of the video sequence is the same. Based on the above considerations, the present application determines the target edge corresponding to the process operation of the display panel area in the image for any image in the video sequence. Optionally, each edge in the image is detected by an edge detection operator, and the length of each edge is determined. The electronic device can pre-save the length range of the edge corresponding to the process operation of the display panel area, and determine the edge whose length belongs to the pre-saved length range among the edges detected by the edge detection operator as the target edge corresponding to the process operation of the display panel area in the image. Edge detection operators include but are not limited to Sobel, Roberts, Prewitt, Canny, Laplacian and other operators.

[0052] After the electronic device determines the target edge, the target process operation area is determined according to the target edge. Optionally, the electronic device can save the number of extended pixels in advance, and extend the target edge in the four directions of up, down, left, and right in the image plane by the number of extended pixels respectively to obtain the target process operation area. It should be noted that the number of extended pixels corresponding to different directions can be the same or different. For example, the number of extended pixels corresponding to the up and down direction is 100, the number of extended pixels corresponding to the up and left directions is 80, and so on.

[0053] After the electronic device determines the target process operation area, it determines each sampling state line in the target process operation area. Optionally, a preset number of line segments can be divided in any direction in the target process operation area, and the line segments intersecting with the target edge in the target process operation area are used as sampling state lines.

[0054] The electronic device detects the target key points corresponding to the operation gesture in the image for each frame of the image in the video sequence. The process is divided into two steps: step one is to detect the operation gesture in the image, and step two is to detect the target key points corresponding to the operation gesture. For the above step one, the electronic device can pre-deploy a trained operation gesture detection model. The operation gesture detection model is used to detect whether the gesture in the image is an operation gesture or a non-operation gesture. When the operation gesture detection model is trained, it is trained based on the operation gesture images, non-operation gesture images and their corresponding label information in the training set. The operation gesture detection model can output the probability value that the gesture in the image is an operation gesture, and when the probability value is greater than the preset probability threshold, the gesture in the image is determined to be an operation gesture. The preset probability threshold is, for example, 0.8, 0.85, etc. For the above step two, the key points in the operation gesture can be detected by a key point detection algorithm, and the key points include the index finger key point, the middle finger key point, the ring finger key point, the little finger key point, the thumb key point, the back of the hand key point, etc. This application can use any key point in the operation gesture as the target key point corresponding to the operation gesture. The operation trajectory information is determined according to the target key points corresponding to the operation gesture in each frame of the image. In other words, according to the video sequence, the target key point sequence corresponding to the operation gesture can be determined, and the trajectory formed by the target key point sequence is the operation trajectory information.

[0055] According to the intersection relationship between the operation trajectory information and each sampling state line, and the gesture movement direction corresponding to the operation trajectory information, the sampling state value of each sampling state line is determined. Optionally, for each sampling state line, if the sampling state line intersects with the operation trajectory information, it is determined that the sampling state line has a sampling state value, and if the sampling state line does not intersect with the operation trajectory information, it is determined that the sampling state line does not have a sampling state value. For the case where there is a sampling state value, different sampling state values ​​corresponding to different gesture movement directions can be pre-set. The sampling state value can be a symbol value such as a number, a letter, or a character. For example, the two ends of the target process operation area are respectively called the first end and the second end. If the gesture movement direction is from the first end to the second end, the corresponding sampling state value is 1, and if the gesture movement direction is from the second end to the first end, the corresponding sampling state value is 2. In this way, the sampling state value of each sampling state line can be determined. Finally, according to the sampling state value of each sampling state line, the process operation behavior detection result is determined. Optionally, each sampling state value and its corresponding quantity when the process operation behavior is normal can be pre-saved. For example, when the process operation behavior is normal, the number corresponding to the sampling state value 1 is 5, and the number corresponding to the sampling state value 2 is 4. After determining the sampling state values ​​of each sampling state line, count each sampling state value and each corresponding number. If the counted sampling state values ​​and each corresponding number are consistent with the previously saved sampling state values ​​and each corresponding number when the process operation behavior is normal, then the process operation behavior detection result is determined to be normal, otherwise the process operation behavior detection result is determined to be abnormal.

[0056] During the operation of the tape attaching process at the edge of the display panel, the operation edge of the display panel area will be operated. The present application first obtains a video sequence in the process operation scene, and for any image in the video sequence, determines the target edge corresponding to the process operation of the display panel area in the image, and determines the target process operation area according to the target edge. Perform process operation behavior detection in the target process operation area to avoid environmental interference outside the target process operation area and improve the accuracy of process operation behavior detection. In addition, each sampling state line in the target process operation area is determined, and the process operation behavior detection result is determined according to the sampling state value of each sampling state line, without processing the entire trajectory point, thereby improving the efficiency of process operation behavior detection. Specifically, for each frame of the video sequence, the target key point corresponding to the operation gesture in the image is detected; according to the target key point corresponding to the operation gesture in each frame of the image, the operation trajectory information is determined; and then according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information, the sampling state value of each sampling state line is determined, and finally, according to the sampling state value of each sampling state line, the process operation behavior detection result is determined. Thus, an efficient and accurate process operation behavior detection scheme is provided. The present invention solves the problems in related technologies of processing video sequences based on action recognition models, large data processing volume, high resource consumption, low processing efficiency, and low processing performance of action recognition models for video sequences, resulting in poor accuracy in identifying process operation behaviors.

[0057] In order to improve the accuracy of determining the edge of the target, Figure 2 The process schematic diagram of determining the target edge of the display panel area provided in the present application, wherein determining the target edge corresponding to the process operation of the display panel area in the image comprises the following steps:

[0058] S201: Determine a display panel area in the image by using an instance segmentation algorithm;

[0059] S202: determining each outer contour point of the display panel area by a boundary tracking algorithm; determining each vertex of the display panel area according to coordinate information of each outer contour point;

[0060] S203: Determine the edges of the display panel area according to the vertices of the display panel area, and use the edge on the side where the process operation is performed as the target edge corresponding to the process operation.

[0061] After the electronic device acquires the video sequence in the process operation scene, for any image in the video sequence, the display panel area in the image is determined by the instance segmentation algorithm. Optionally, the SegAnyting large model algorithm, the RTMdet algorithm, etc. are used as instance segmentation algorithms to determine the display panel area in the image. After the display panel area is determined, the various outer contour points of the display panel area are determined by the boundary tracking algorithm. According to the coordinate information of each outer contour point, the various vertices of the display panel area are determined. Then, the various edges of the display panel area are determined according to each vertex. Optionally, each edge is a sky side edge, a ground side edge, a left side edge, and a right side edge. The electronic device can pre-save the process operation edge corresponding to each workstation for the workstation. For example, if the process operation edge is a sky side edge, the sky side edge of the display panel area determined according to each vertex is determined as the target edge.

[0062] Optionally, in the TV panel tape attaching operation process, the TV panel as the operation object is firstly image recognized and segmented. The RTMdet instance segmentation algorithm is trained by using factory data, and then the trained model is used to detect the segmented area of ​​the TV panel. Figure 3 This is an example segmentation effect diagram of the TV panel provided in this application.

[0063] The four edges of the TV display panel of the segmented image are obtained through a boundary tracing algorithm.

[0064] In digital binary image processing, the boundary tracking algorithm performs topological structure analysis on the image to extract a series of coordinate points of the boundary contour or the tracking of chain code boundaries. It is an important technology in the field of computer vision and image processing, mainly used to extract meaningful shape information from binary images. The core of this technology is to identify and track the boundaries between foreground objects and backgrounds in the image, and then analyze the topological structures of these objects, such as their connectivity, connectivity, number of holes and other characteristics.

[0065] The basic steps are as follows:

[0066] Initialization: Start with a known boundary pixel, usually a point on the edge of the image.

[0067] Tracking direction: Determine the next pixel to visit based on the neighborhood information of the current pixel, ensuring that it always moves along the boundary without crossing into or outside the object.

[0068] Contour construction: As the tracking process proceeds, the positions of boundary pixels are continuously recorded to form contours.

[0069] Complex structure processing: When encountering branches, end points or holes, these topological events are recorded through specific rules to ensure that the structural features in the image can be accurately reflected.

[0070] Termination condition: Boundary tracking ends when returning to the starting point or reaching the predetermined stop condition.

[0071] Finally, a series of outer contour points of the TV segmentation area are obtained through the boundary tracking algorithm, and x, y Find the corresponding upper left vertex, upper right vertex, lower left vertex and lower right vertex according to the coordinate size. The formula is as follows:

[0072] P 左上 =min x (min y (P1, P2, P3…, P n ))

[0073] P 左下 =min x (maax y (P1, P2, P3…, P n ))

[0074] P 右上 =max x (min y (P1, P2, P3…, P n ))

[0075] P 右下 =max x (max y (P1, P2, P3…, P n )).

[0076] The top left, bottom left, top right, and bottom right represent the four vertices of the TV panel, p1, p2, p3, and p n Represents all the contour points obtained, min x Indicates the smallest point in coordinate comparison, max x Indicates the smallest point in coordinate comparison.

[0077] Figure 4 A schematic diagram of various vertices for determining the display panel area provided in this application. Figure 5 A schematic diagram of a target edge for determining a display panel area provided in the present application.

[0078] In order to determine the target process operation area more accurately, in the present application, determining the target process operation area according to the target edge includes:

[0079] Detect an operation gesture detection frame in the image; determine edge extension size information based on size information of the operation gesture detection frame; and expand the target edge in a direction perpendicular to the process operation based on the edge extension size information to obtain the target process operation area.

[0080] In the present application, the operation gesture detection frame in the image can be detected by the operation gesture detection model to determine the size information of the operation gesture detection frame. The size information includes the length information and width information of the operation gesture detection frame. According to the size information of the operation gesture detection frame, the edge extension size information is determined. Optionally, the width information of the operation gesture detection frame is used as the edge extension size information. Width information such as 1.5 times and 2 times can also be used as edge extension size information. According to the edge extension size information, the target edge is extended in the direction perpendicular to the process operation to obtain the target process operation area. Optionally, in the direction perpendicular to the process operation, the edge extension size information is extended to both sides of the target edge respectively, and the obtained area is used as the target process operation area.

[0081] The electronic device obtains the four edges of the display panel area. For example, the side where the worker is operating in the image is named the sky side. The entire sky side process is completed on the sky side, including the tape attachment and pressing operations. Therefore, operations in other areas can be regarded as interference or unnecessary operations and filtered. After obtaining the sky side edge, the adaptive adjustment of the target process operation area is achieved by operating the size of the gesture detection box.

[0082] Figure 6 The schematic diagram of the second process operation behavior detection process provided in this application includes the following steps:

[0083] S301: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; according to the edge extension size information, extend the target edge in a direction perpendicular to the process operation to obtain the target process operation area; and determine each sampling state line in the target process operation area;

[0084] S302: for each frame of the image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; and determining operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0085] S303: Determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0086] In order to determine each sampling state line in the target process operation area more accurately, in the present application, determining each sampling state line in the target process operation area includes:

[0087] Divide the target process operation area equally according to the direction perpendicular to the process operation to obtain a preset number of sub-areas;

[0088] The edge lines of the preset number of sub-areas in the direction perpendicular to the process operation are used as sampling state lines.

[0089] Figure 7 A schematic diagram of dividing each sampling state line in the target process operation area provided in this application, such as Figure 7 As shown in FIG. 1 , the target process operation area is evenly divided according to the direction perpendicular to the process operation to obtain 7 sub-areas. It should be noted that Figure 7 The division into 7 sub-areas is only an example, and the division into 5 sub-areas, 6 sub-areas, etc. The edge lines of the preset number of sub-areas in the direction of the process operation are used as the sampling state lines. Figure 7 8 sampling status lines are obtained.

[0090] Figure 8 The third process operation behavior detection process diagram provided for this application includes the following steps:

[0091] S401: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; according to the edge extension size information, extend the target edge in a direction perpendicular to the process operation to obtain the target process operation area; divide the target process operation area equally according to the direction perpendicular to the process operation to obtain a preset number of sub-areas; and use edge lines of the preset number of sub-areas in the direction perpendicular to the process operation as sampling state lines;

[0092] S402: for each frame of the image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; and determining operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0093] S403: Determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0094] In order to determine the target key points corresponding to the operation gesture in the detection image more accurately, Fig. 9 The schematic diagram of the process of detecting the target key points corresponding to the operation gesture in the image provided by the present application includes the following steps:

[0095] S501: Determine an operation gesture area in the image by using an instance segmentation algorithm;

[0096] S502: determining gesture key points in the operation gesture area through a key point detection algorithm; the gesture key points include an index finger key point and a thumb key point;

[0097] S503: Taking the center point of the index finger key point and the thumb key point as the target key point corresponding to the operation gesture.

[0098] In this application, the operation gesture area in the image is determined by the SegAnyting large model algorithm, the RTMdet algorithm and other instance segmentation algorithms, and then the gesture key points in the operation gesture area are determined by the key point detection algorithm. Interference from areas outside the operation gesture area is avoided, and the accuracy of the determined gesture key points is improved. Gesture key points include but are not limited to index finger key points and thumb key points. The center point of the index finger key point and the thumb key point is then used as the target key point corresponding to the operation gesture. The present application can ensure that the operation trajectory information corresponding to the target key point is located within the target process operation area, thereby ensuring the accuracy of subsequent process operation behavior detection.

[0099] In the present application, in order to ensure the accuracy of process operation behavior detection, before determining the operation trajectory information according to the target key points corresponding to the operation gesture in each frame of the image, the method further includes:

[0100] Determine the operator region in the image by using an instance segmentation algorithm; determine the key points of each action of the operator in the operator region by using a key point detection algorithm;

[0101] If each action key point is located in the pre-defined personnel operation ROI area, and the target key point corresponding to the operation gesture is located in the target process operation area, the target key point is determined to be a valid key point, and the operation trajectory information is determined according to the target key point corresponding to the operation gesture in each frame image; otherwise, the target key point is determined to be an invalid key point.

[0102] In this application, the electronic device detects the target key points corresponding to the operation gestures in each frame of the video sequence, and then determines the operator area in the image by using the SegAnyting large model algorithm, RTMdet algorithm and other instance segmentation algorithms before determining the operation trajectory information based on the target key points corresponding to the operation gestures in each frame of the image. Then, the key points of each action performed by the operator in the operator area are determined by the key point detection algorithm. This avoids interference from areas outside the operator area and improves the accuracy of determining each action key point. The various action key points for the operator to perform an operation include elbow key points, wrist key points, arm key points, etc.

[0103] The electronic device stores a pre-defined personnel operation ROI area, determines whether each action key point is located in the pre-defined personnel operation ROI area, and determines whether the target key point corresponding to the operation gesture is located in the target process operation area. If both of the above judgment results are yes, the target key point is determined to be a valid key point, and the process of determining the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image is performed. If either of the above two judgment results is no, the target key point is determined to be an invalid key point, and the subsequent process of determining the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image is not performed.

[0104] Fig.10 The schematic diagram of the fourth process operation behavior detection provided in this application includes the following steps:

[0105] S601: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; according to the edge extension size information, extend the target edge in a direction perpendicular to the process operation to obtain the target process operation area; divide the target process operation area equally according to a direction perpendicular to the process operation to obtain a preset number of sub-areas; and use edge lines of the preset number of sub-areas in the direction perpendicular to the process operation as sampling state lines;

[0106] S602: for each frame of the video sequence, detecting the target key points corresponding to the operation gesture in the image; determining the operator region in the image by using an instance segmentation algorithm; determining each action key point of the operator's operation behavior in the operator region by using a key point detection algorithm; if each action key point is located in a pre-defined operator operation ROI region, and the target key point corresponding to the operation gesture is located in the target process operation region, determining the target key point as a valid key point; determining the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0107] S603: Determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0108] In the present application, in order to improve the accuracy of determining the sampling state value of each sampling state line, according to the intersection relationship between the operation trajectory information and the sampling state lines, and the gesture movement direction corresponding to the operation trajectory information, determining the sampling state value of each sampling state line includes:

[0109] For each of the sampling state lines, if the operation trajectory information intersects with the sampling state line, determine the positive and negative signs of the sampling state values ​​of the sampling state line according to the gesture movement direction corresponding to the operation trajectory information, and determine the numerical values ​​of the sampling state values ​​of the sampling state line according to the order in which the operation trajectory information passes through the sampling state lines; determine the sampling state value of the sampling state line according to the positive and negative signs and the numerical values.

[0110] For each sampling state line, if the operation trajectory information does not intersect with the sampling state line, it is determined that the sampling state line has no sampling state value. If the operation trajectory information intersects with the sampling state line, the gesture movement direction corresponding to the operation trajectory information is determined, and it is pre-defined that the sampling state values ​​of the sampling state lines in different directions have different positive and negative signs. For example, when the gesture movement direction is from left to right, the sampling state value of the sampling state line is positive; when the gesture movement direction is from right to left, the sampling state value of the sampling state line is negative. According to the above-mentioned pre-defined rules, the positive and negative signs of the sampling state values ​​of the sampling state line can be determined. In addition, the numerical value of the sampling state value of the sampling state line is determined according to the order in which the operation trajectory information passes through the sampling state line. Optionally, each sampling state line is numbered in advance according to the order of the sampling state lines. For example, there are 6 sampling state lines, numbered 1, 2, 3, 4, 5, 6 from left to right, or they can also be numbered 6, 5, 4, 3, 2, 1 from left to right. After numbering each sampling state line in the order of the sampling state line, when determining the numerical value of the sampling state value of the sampling state line according to the order in which the operation trajectory information passes through the sampling state line, the number of the sampling state line can be directly used as the numerical value of the sampling state value. The sampling state value of the sampling state line is determined according to the positive and negative signs and the numerical values.

[0111] Fig.11 The schematic diagram of the fifth process operation behavior detection provided in this application includes the following steps:

[0112] S701: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; according to the edge extension size information, extend the target edge in a direction perpendicular to the process operation to obtain the target process operation area; divide the target process operation area equally according to the direction perpendicular to the process operation to obtain a preset number of sub-areas; and use edge lines of the preset number of sub-areas in the direction perpendicular to the process operation as sampling state lines;

[0113] S702: for each frame of the video sequence, detecting the target key points corresponding to the operation gesture in the image; determining the operator region in the image by using an instance segmentation algorithm; determining each action key point of the operator's operation behavior in the operator region by using a key point detection algorithm; if each action key point is located in a pre-defined operator operation ROI region, and the target key point corresponding to the operation gesture is located in the target process operation region, determining the target key point as a valid key point; determining the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0114] S703: For each sampling state line, if the operation trajectory information intersects with the sampling state line, determine the positive and negative signs of the sampling state values ​​of the sampling state line according to the gesture movement direction corresponding to the operation trajectory information, and determine the numerical values ​​of the sampling state values ​​of the sampling state line according to the order in which the operation trajectory information passes through the sampling state lines; determine the sampling state value of the sampling state line according to the positive and negative signs and the numerical values; determine the process operation behavior detection results according to the sampling state values ​​of each sampling state line.

[0115] In the present application, in order to further improve the accuracy of determining the sampling state value of each sampling state line, determining the process operation behavior detection result according to the sampling state value of each sampling state line includes:

[0116] A sampling state value sequence is determined according to the sampling state values ​​of each sampling state line and the order in which the operation trajectory information passes through each sampling state line; and a process operation behavior detection result is determined according to a comparison result between the sampling state value sequence and a pre-saved target sampling state value sequence.

[0117] Taking the specific identification requirements of the operation of attaching the tape to the sky side of the TV panel as an example, the following five action steps need to be identified. Step 1: Press 2-5cm from the left end; Step 2: Paste the tape from left to right; Step 3: Press back within 5cm of the right end; Step 4: Press back the entire tape from right to left; Step 5: Press back within 5cm of the left end. Each action is pre-saved with a target sampling state value sequence. According to the sampling state values ​​of each sampling state line and the order in which the operation trajectory information passes through each sampling state line, after determining the sampling state value sequence, it is compared with the pre-saved target sampling state value sequence. If they are consistent, it is determined that the action is completed normally, otherwise it is determined that the action is not completed normally. If the five actions are all determined to be completed normally in order, the process operation behavior detection result is determined to be normal process operation. Otherwise, the process operation is determined to be abnormal.

[0118] Fig.12 The schematic diagram of the sixth process operation behavior detection provided in this application includes the following steps:

[0119] S801: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; according to the edge extension size information, extend the target edge in a direction perpendicular to the process operation to obtain the target process operation area; divide the target process operation area equally according to the direction perpendicular to the process operation to obtain a preset number of sub-areas; and use edge lines of the preset number of sub-areas in the direction perpendicular to the process operation as sampling state lines;

[0120] S802: for each frame of the video sequence, detecting the target key points corresponding to the operation gesture in the image; determining the operator region in the image by using an instance segmentation algorithm; determining each action key point of the operator's operation behavior in the operator region by using a key point detection algorithm; if each action key point is located in a pre-defined operator operation ROI region, and the target key point corresponding to the operation gesture is located in the target process operation region, determining the target key point as a valid key point; determining the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0121] S803: For each of the sampling state lines, if the operation trajectory information intersects with the sampling state line, determine the positive and negative signs of the sampling state values ​​of the sampling state line according to the gesture movement direction corresponding to the operation trajectory information, and determine the numerical values ​​of the sampling state values ​​of the sampling state line according to the order in which the operation trajectory information passes through the sampling state lines; determine the sampling state value of the sampling state line according to the positive and negative signs and the numerical values;

[0122] S804: Determine a sampling state value sequence according to the sampling state values ​​of each sampling state line and the order in which the operation trajectory information passes through each sampling state line; determine a process operation behavior detection result according to a comparison result between the sampling state value sequence and a pre-saved target sampling state value sequence.

[0123] The process of detecting the process operation behavior provided by the present application is described in detail below with reference to the accompanying drawings.

[0124] This application proposes a method for detecting irregular behaviors of workers in attaching TV tapes. It monitors the operation process in real time under the premise of ensuring the real-time positioning of the operator's activity area. This solution not only improves the existing end-to-end solution-based resource occupation and poor interpretability and debugging, but also alleviates the problem of large interference in the analysis of key human information. This further optimizes the efficiency of workers, discovers production bottlenecks, reduces resource waste, and ultimately achieves cost reduction, efficiency increase, and quality improvement in production and manufacturing.

[0125] Fig.13 The process operation behavior detection flow chart provided for this application is as follows: Fig.13 As shown, the present application proposes a method for detecting irregular behaviors of workers attaching TV tape. (1) The scheme first transforms the coordinates of the acquired monitoring image to eliminate the influence caused by distortion, and then performs TV panel detection and human body detection on the image respectively; (2) In the operation area positioning module, the RTMdet instance segmentation algorithm is used to segment the TV panel, and the vertex of the TV panel is located using the boundary tracking algorithm after the segmented area is obtained, so as to locate the TV panel worker operation area according to the conversion of the acquired vertices into four edges, and at the same time, the operation process detection zone is delineated to generate the key action information of the worker's work; (3) Then, unnecessary interference is eliminated through the prior knowledge of the factory environment working area, and then the located operation area and the located human body detection frame are logically combined and judged to determine the process operator. (4) Adaptive processing is performed based on the key personnel information obtained and the detection zone delineated by the operation area, so as to obtain the key action information of the personnel operation process, and finally the key information obtained is compared with the action logic library based on the prior knowledge, and the final judgment result of the action is output.

[0126] In the prior art, the worker action detection solution is mainly end-to-end. In the face of multi-person operation and interference problems under other workstation operations in complex factory scenes, the solution relies on increasing the amount of data training and computing resources, which is not conducive to reducing costs and increasing efficiency in factories. In addition, the stability and interpretability of the algorithm are poor, and there is a lack of flexible optimization means in the later stage. Based on this problem, this application proposes a method for locating the worker operation area based on prior experience and instance segmentation algorithm. Different from the method of using deep learning to locate the operator ID, the working area of ​​the operator is obtained through the instance segmentation algorithm in the overall solution, so as to determine the operator, avoiding the omission and false detection caused by the instability of the key action information of the worker due to the jump of the operator ID. In addition, through the division of the operation area, it deviates from the traditional solution of tracking and detecting with people as the main target, and focuses on the effective actions in the detection area, thereby alleviating the problem of waste of human resources caused by the invalidity and misjudgment of the algorithm under the interference of the complex environment of the factory.

[0127] In the present application, after locating the four sides of the TV display panel area, the four sides of the TV display panel area are obtained, taking the side (sky side) where the worker is operating in the image as an example, including the operation of attaching and pressing tape on the sky side. The entire sky side process is completed on the sky side, so operations in other areas can be regarded as interference or unnecessary operations and are filtered. Fig.14 This is a schematic diagram of the operation area division provided by this application. Fig.14 As shown, after obtaining the sky side edge, the adaptive adjustment of the target process operation area is achieved by operating the size of the gesture detection box. In addition, there are multiple line segments in the vertical direction. Different from the scheme of directly using the hand key point information for analysis, the present application can output the current valid trajectory information by the hand key point trajectory passing through the line segment.

[0128] In this application, we first use the factory's prior knowledge to delineate the ROI area in the image to determine the effective working range of the personnel, and then use the ray algorithm to determine whether the key points of the human body's movements are within the delineated ROI area. Then, we determine whether the key points of the hand are within the target process operation area, and finally determine the operator.

[0129] The formula based on the ray method is as follows:

[0130] Among them, p1 and p2 represent the two vertices of an edge of the polygon, and x0 and y0 represent the point.

[0131] Taking the specific identification requirements of the operation of attaching the tape to the sky side of the TV panel as an example, the following five action steps need to be identified (with the worker's direction as an indicator): Step 1: Press 2-5 cm from the left end; Step 2: Stick the tape from left to right; Step 3: Press back within 5 cm of the right end; Step 4: Press back the entire tape from right to left; Step 5: Press back within 5 cm of the left end.

[0132] In the existing worker motion detection scheme, the end-to-end method has poor generalization ability, low model interpretability and optimization, while the key point information + trajectory analysis method uses the motion trajectory matching method: it is necessary to manually enter the standard motion trajectory a in advance, and then when the worker operates, the worker's hand key points are detected in real time to generate the actual motion trajectory b of the hand; then the Euclidean distance between the actual motion trajectory b and the standard motion trajectory a is calculated to determine the matching degree of the two trajectories; the problem with this method is that the trajectory composed of multiple key points is easily interfered by invalid key points, and when the number of key point trajectories of the human body increases, the matching algorithm requirements for the standard motion trajectory and the actual motion trajectory are more difficult and complex.

[0133] Therefore, in order to further optimize worker efficiency, identify production bottlenecks, and reduce resource waste, this application uses the interactive behavior of operator trajectories and operation area divisions to perform trajectory adaptive processing, thereby obtaining key action information to compare with the self-built trajectory algorithm library and output the results. This method extracts effective action variables in trajectory information comparison, greatly reduces the amount of comparison, and optimizes algorithm efficiency.

[0134] The following describes how to obtain key action information status through operator trajectories and operation division areas, and how to extract valid trajectories and make judgments with the self-built action logic library.

[0135] This application uses a labeled factory dataset to train the RTMpose model to detect the key point information of the workers' TV panel attachment action, and filters out invalid points through the designated ROI area and operation area, and then connects the key points in the operation process to obtain the workers' real-time trajectory. In addition, the obtained operation division area is divided into multiple vertical line segments. When the real-time trajectory passes through a vertical line segment, the current operation is given an action state, and the previous action state is retained.

[0136] Take step 2 on the sky side, pasting the tape from left to right as an example. Fig.15 The operation track information diagram provided for this application is as follows: Fig.15 As shown in the figure, each divided vertical line segment corresponds to a state. When the motion trajectory passes through the "1" line segment from left to right, the current state is assigned to 1, and when it passes through the "1" line segment from right to left, the current state is assigned to -1. Some abnormal interference, such as some small movements or bad habits of operators, can be well filtered out, so as to obtain truly effective current motion information. Compared with the traditional solution of matching motion vectors based on real-time trajectories, this solution creates a new motion information state variable based on the acquisition of motion trajectories, which can not only effectively obtain the key information of the current worker's movement, but also reduce the waste of computing resources and instability caused by matching the full motion trajectory.

[0137] Fig.16 This is the recognition effect diagram corresponding to action 1 provided in this application.

[0138] Step 1 Identification: Fig.16 As shown in FIG. 1 , when the operator starts the operation, the hand trajectory stays in the black box for more than 2 seconds, which can be regarded as completing action 1.

[0139] Step 2 identification: When the operator's action trajectory state shows the continuous state variables [1, 2, 3, 4, 5, 6], the step 2 action can be considered completed.

[0140] Step 3 identification: When the operator's action trajectory state shows the continuous state variable [6, -6, 6, -6], step 3 action is considered to be completed.

[0141] Step 4 identification: When the operator's action trajectory state shows the continuous state variable [-6, -5, -4, -3, -2, -1], the step 4 action is considered to be completed.

[0142] Step 5 identification: When the operator's action trajectory state shows a continuous state variable of [1,-1,1,-1], step 5 is considered to be completed.

[0143] During the TV panel tape attachment process, there will be a corresponding action logic vector for each action. The action logic variable greatly simplifies the real-time matching judgment of the action trajectory, and can create vectors in real time according to different operating actions. To a certain extent, the replication and promotion of the algorithm is increased, providing an algorithm basis for subsequent applications in other workstations, production lines, and factories. It should be noted that in other similar scenarios such as primer attachment action and welding gun operation action detection, the technical solution proposed in this application can be quickly replicated and promoted, thereby reducing the algorithm development cycle and personnel costs.

[0144] Fig.17 The schematic diagram of the device structure for detecting process operation behavior provided in this application includes:

[0145] The first determination module 171 is used to obtain a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area;

[0146] The second determination module 172 is used to detect, for each frame of the video sequence, a target key point corresponding to the operation gesture in the image; and determine the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image;

[0147] The behavior detection module 173 is used to determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; and determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

[0148] The first determination module 171 is specifically used to determine the display panel area in the image through an instance segmentation algorithm; determine the various outer contour points of the display panel area through a boundary tracking algorithm; determine the various vertices of the display panel area according to the coordinate information of the various outer contour points; determine the various edges of the display panel area according to the various vertices of the display panel area, and use the side edge where the process operation is performed as the target edge corresponding to the process operation.

[0149] The first determination module 171 is specifically used to detect an operation gesture detection frame in the image; determine edge extension size information according to size information of the operation gesture detection frame; and expand the target edge in a direction perpendicular to the process operation according to the edge extension size information to obtain the target process operation area.

[0150] The first determination module 171 is specifically configured to equally divide the target process operation area in a direction perpendicular to the process operation to obtain a preset number of sub-areas; and use edge lines of the preset number of sub-areas in the direction perpendicular to the process operation as sampling state lines.

[0151] The second determination module 172 is specifically used to determine the operation gesture area in the image through an instance segmentation algorithm; determine the gesture key points in the operation gesture area through a key point detection algorithm; the gesture key points include the index finger key point and the thumb key point; and use the center point of the index finger key point and the thumb key point as the target key point corresponding to the operation gesture.

[0152] The second determination module 172 is also used to determine the operator area in the image through an instance segmentation algorithm; determine the various action key points of the operator's operation behavior in the operator area through a key point detection algorithm; if the various action key points are located in a pre-defined personnel operation ROI area, and the target key point corresponding to the operation gesture is located in the target process operation area, determine the target key point as a valid key point, and determine the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image; otherwise, determine the target key point as an invalid key point.

[0153] The behavior detection module 173 is specifically used to determine, for each of the sampling state lines, if the operation trajectory information intersects with the sampling state line, the positive and negative signs of the sampling state values ​​of the sampling state line according to the gesture movement direction corresponding to the operation trajectory information, and the numerical values ​​of the sampling state values ​​of the sampling state line according to the order in which the operation trajectory information passes through the sampling state line; and determine the sampling state value of the sampling state line according to the positive and negative signs and the numerical values.

[0154] The behavior detection module 173 is specifically used to determine the sampling state value sequence according to the sampling state values ​​of each sampling state line and the order in which the operation trajectory information passes through each sampling state line; and determine the process operation behavior detection result according to the comparison result between the sampling state value sequence and the pre-saved target sampling state value sequence.

[0155] The present application also provides an electronic device, such as Fig.18 As shown, it includes: a processor 181, a communication interface 182, a memory 183 and a communication bus 184, wherein the processor 181, the communication interface 182, and the memory 183 communicate with each other through the communication bus 184;

[0156] The memory 183 stores a computer program, and when the program is executed by the processor 181, the processor 181 executes any of the above method steps.

[0157] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0158] The communication interface 182 is used for communication between the above electronic device and other devices.

[0159] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0160] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0161] The present application also provides a computer storage readable storage medium, wherein the computer readable storage medium stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0162] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0163] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting process operation behavior, characterized in that: The method comprises: Acquire a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area; For each frame of the image in the video sequence, detecting a target key point corresponding to the operation gesture in the image; and determining operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image; According to the intersection relationship between the operation trajectory information and the sampling state lines and the gesture movement direction corresponding to the operation trajectory information, the sampling state values ​​of the sampling state lines are determined; according to the sampling state values ​​of the sampling state lines, the process operation behavior detection result is determined.

2. The method according to claim 1, characterized in that Determining a target edge corresponding to a process operation of a display panel area in the image includes: Determine the display panel area in the image by using an instance segmentation algorithm; Determine each outer contour point of the display panel area by a boundary tracking algorithm; determine each vertex of the display panel area according to the coordinate information of each outer contour point; According to each vertex of the display panel area, each edge of the display panel area is determined, and the edge on the side where the process operation is performed is used as the target edge corresponding to the process operation.

3. The method according to claim 1, characterized in that Determining a target process operation area according to the target edge includes: Detect an operation gesture detection frame in the image; determine edge extension size information based on size information of the operation gesture detection frame; and expand the target edge in a direction perpendicular to the process operation based on the edge extension size information to obtain the target process operation area.

4. The method according to claim 1, characterized in that Determining each sampling state line in the target process operation area includes: Divide the target process operation area equally according to the direction perpendicular to the process operation to obtain a preset number of sub-areas; The edge lines of the preset number of sub-areas in the direction perpendicular to the process operation are used as sampling state lines.

5. The method according to claim 1, characterized in that Detecting target key points corresponding to the operation gesture in the image includes: Determine the operation gesture area in the image by using an instance segmentation algorithm; Determine the gesture key points in the operation gesture area by a key point detection algorithm; the gesture key points include the index finger key point and the thumb key point; The center point of the index finger key point and the thumb key point is used as the target key point corresponding to the operation gesture.

6. The method according to claim 1, characterized in that Before determining the operation trajectory information according to the target key points corresponding to the operation gesture in each frame of the image, the method further includes: Determine the operator region in the image by using an instance segmentation algorithm; determine the key points of each action of the operator in the operator region by using a key point detection algorithm; If each action key point is located in the pre-defined personnel operation ROI area, and the target key point corresponding to the operation gesture is located in the target process operation area, the target key point is determined to be a valid key point, and the operation trajectory information is determined according to the target key point corresponding to the operation gesture in each frame image; otherwise, the target key point is determined to be an invalid key point.

7. The method according to claim 1, characterized in that Determining the sampling state value of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information includes: For each of the sampling state lines, if the operation trajectory information intersects with the sampling state line, determine the positive and negative signs of the sampling state values ​​of the sampling state line according to the gesture movement direction corresponding to the operation trajectory information, and determine the numerical values ​​of the sampling state values ​​of the sampling state line according to the order in which the operation trajectory information passes through the sampling state lines; determine the sampling state value of the sampling state line according to the positive and negative signs and the numerical values.

8. The method according to claim 1, characterized in that Determining the process operation behavior detection result according to the sampling state values ​​of each sampling state line includes: A sampling state value sequence is determined according to the sampling state values ​​of each sampling state line and the order in which the operation trajectory information passes through each sampling state line; and a process operation behavior detection result is determined according to a comparison result between the sampling state value sequence and a pre-saved target sampling state value sequence.

9. A device for detecting process operation behavior, characterized in that: The device comprises: A first determination module is used to obtain a video sequence in a process operation scene, and for any image in the video sequence, determine a target edge corresponding to a process operation in a display panel area in the image; determine a target process operation area according to the target edge; and determine each sampling state line in the target process operation area; A second determination module is used to detect, for each frame of the video sequence, a target key point corresponding to the operation gesture in the image; and determine the operation trajectory information according to the target key point corresponding to the operation gesture in each frame of the image; The behavior detection module is used to determine the sampling state values ​​of each sampling state line according to the intersection relationship between the operation trajectory information and each sampling state line and the gesture movement direction corresponding to the operation trajectory information; and determine the process operation behavior detection result according to the sampling state values ​​of each sampling state line.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

Images