Methods, apparatus, equipment, and storage media for paper tearing behavior recognition based on ontology.

Abstract

This invention discloses a method, apparatus, device, and storage medium for paper-tearing behavior recognition based on an ontology library. The method includes: constructing an ontology library of paper-tearing behavior; wherein, the paper-tearing behavior is described hierarchically in the ontology library, and the behavior is divided into three primitives according to chronological order, with keyframes representing the primitives, and keyframe rules established for each keyframe; acquiring a video of the paper-tearing behavior to be recognized, performing target detection on each frame of the video to obtain target candidate boxes for each frame; when it is determined that three frames sequentially satisfy the corresponding keyframe rules according to chronological order, the corresponding keyframes and their corresponding primitives are activated sequentially; when all primitives are activated, the paper-tearing behavior in the video is determined to have successfully occurred. This invention effectively solves the judgment error caused by unsatisfactory target detection results, achieving effective and accurate recognition of paper-tearing behavior.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a method, apparatus, device, and storage medium for paper tearing behavior recognition based on an ontology library. Background Technology

[0002] The industrial process of heat sink crimping includes four steps: placing the cover plate, removing the heat sink sticker, placing the heat sink, and crimping the heat sink. The success of removing the heat sink sticker directly affects the crimping quality. Currently, after removing the heat sink sticker, a target detection algorithm is used to detect the presence of the sticker on the heat sink to determine if the removal was successful, thus monitoring the sticker-removal process. However, due to limited camera positions and hand obstructions, accurate identification of the sticker removal during the operation is not possible. Summary of the Invention

[0003] The purpose of this invention is to provide a paper-tearing behavior recognition method, device, equipment, and storage medium based on an ontology library. By exploring the essence of paper-tearing behavior, the invention utilizes a dedicated ontology library to compensate for the deficiencies in target detection and hand detection in real-world scenarios. By combining target detection, hand detection, and behavior patterns, the invention effectively solves the judgment errors caused by unsatisfactory target detection results, achieving effective recognition of paper-tearing behavior with high accuracy.

[0004] To achieve the above objectives, embodiments of the present invention provide a paper-tearing behavior recognition method based on an ontology library, comprising:

[0005] Construct an ontology library for the paper-tearing behavior; wherein, the paper-tearing behavior is described hierarchically in the ontology library, the paper-tearing behavior is divided into three primitives according to the chronological order, the primitives are represented by keyframes, and keyframe rules are established for each keyframe.

[0006] A video of the paper-tearing behavior to be identified is acquired, and target detection is performed on each frame of the paper-tearing behavior video to obtain target candidate boxes for each frame; wherein, the target candidate boxes include: left-hand candidate boxes, right-hand candidate boxes, and heat sink candidate boxes;

[0007] The target candidate box is used to determine whether the image containing the target candidate box satisfies the corresponding key frame rules in chronological order. When it is determined that there are three frames that satisfy the corresponding key frame rules in chronological order, the corresponding key frame and the primitive corresponding to the key frame are activated in sequence.

[0008] When all the aforementioned primitives are activated, it is determined that the paper-tearing behavior in the video of the paper-tearing behavior has successfully occurred.

[0009] As an improvement to the above scheme, the ontology library includes: behavior level, primitive level, keyframe level, and entity level; wherein,

[0010] The behavioral levels include: paper tearing behavior;

[0011] The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive;

[0012] The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule;

[0013] The physical levels include: left hand, right hand, heat sink, and sticker.

[0014] As an improvement to the above solution, the keyframe rule for the keyframe of holding the heat sink includes one of the following:

[0015] The candidate boxes for the heat sink and the candidate boxes for the left hand intersect;

[0016] The candidate boxes for the heat sink and the right-hand candidate boxes have an intersection;

[0017] The keyframe rules for the two-handed interaction keyframes include one of the following:

[0018] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is less than a preset first distance threshold.

[0019] The left-hand candidate box and the right-hand candidate box have an intersection;

[0020] The left-hand candidate box and the right-hand candidate box intersect with the heat sink candidate box, and the overlap rate of the left-hand candidate box and the right-hand candidate box on the vertical axis is greater than the preset overlap rate.

[0021] The keyframe rule for the hand separation keyframe includes one of the following:

[0022] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is greater than a preset second distance threshold; wherein, the second distance threshold is greater than the first distance threshold;

[0023] The left-hand candidate box and the right-hand candidate box do not overlap.

[0024] As an improvement to the above scheme, the step of using the target candidate box to determine whether the images containing the target candidate box sequentially satisfy the corresponding keyframe rules in chronological order, and when it is determined that three images sequentially satisfy the corresponding keyframe rules in chronological order, sequentially activating the corresponding keyframes and the primitives corresponding to the keyframes, includes:

[0025] Determine whether the target candidate box of each frame image satisfies the keyframe rule of the hand holding the heat sink keyframe. When it is determined that there is a target candidate box of the first current frame image that satisfies the keyframe rule of the hand holding the heat sink keyframe, the first current frame image is used as the hand holding the heat sink keyframe and the hand holding the heat sink keyframe is activated.

[0026] When the keyframe of the hand holding the heat sink is activated, the hand holding the heat sink primitive is activated;

[0027] When the hand holding the heat sink primitive is activated, it continues to determine whether the target candidate box of each frame image after the hand holding the heat sink keyframe satisfies the keyframe rule of the two-hand interaction keyframe. When it is determined that there is a target candidate box of the second current frame image that satisfies the keyframe rule of the two-hand interaction keyframe, the second current frame image is used as the two-hand interaction keyframe, and the two-hand interaction keyframe is activated.

[0028] When the two-hand interaction keyframe is activated, the two-hand interaction primitive is activated;

[0029] When the two-hand interaction primitive is activated, it continues to determine whether the target candidate box of each frame image after the two-hand separation keyframe satisfies the keyframe rule of the two-hand separation keyframe. When it is determined that the target candidate box of the third current frame image satisfies the keyframe rule of the two-hand separation keyframe, the third current frame image is taken as the two-hand separation keyframe and the two-hand separation keyframe is activated.

[0030] When the hand separation keyframe is activated, the hand separation primitive is activated.

[0031] As an improvement to the above scheme, the target candidate box further includes a sticker candidate box. Therefore, when all the primitives are activated, the ontology-based paper-tearing behavior recognition method further includes:

[0032] For any frame image after the three frames, if it is determined that there is a first intersection between the heat sink candidate frame and the sticker candidate frame, and the relative size of the first intersection is greater than a preset relative size threshold, the paper tearing behavior in the video is determined to be unsuccessful.

[0033] As an improvement to the above scheme, the step of performing target detection on each frame of the video of the paper-tearing behavior to obtain the target candidate box for each frame includes:

[0034] Target detection is performed on each frame of the video of the paper-tearing behavior;

[0035] When multiple left hands are detected in any frame of an image, the distance between the center point of the candidate box of each left hand and the center point of the candidate box of the pre-set working area of ​​both hands is calculated, and the candidate box of the left hand with the shortest distance is taken as the detected target candidate box.

[0036] When multiple right hands are detected in any frame of an image, the distance between the center point of the candidate box of each right hand and the center point of the candidate box of the working area of ​​both hands is calculated, and the candidate box of the right hand with the shortest distance is taken as the detected target candidate box.

[0037] To achieve the above objectives, embodiments of the present invention also provide a paper-tearing behavior recognition device based on an ontology library, comprising:

[0038] An ontology library construction module is used to construct an ontology library for the paper-tearing behavior; wherein, the paper-tearing behavior is described hierarchically in the ontology library, the paper-tearing behavior is divided into three primitives according to the chronological order, the primitives are represented by keyframes, and keyframe rules are established for each keyframe.

[0039] The target detection module is used to acquire the paper-tearing behavior video to be identified, and to perform target detection on each frame of the paper-tearing behavior video to obtain the target candidate box of each frame; wherein, the target candidate box includes: left-hand candidate box, right-hand candidate box and heat sink candidate box;

[0040] The activation module is used to determine whether the images where the target candidate box is located satisfy the corresponding key frame rules in chronological order. When it is determined that there are three frames that satisfy the corresponding key frame rules in chronological order, the corresponding key frame and the primitives corresponding to the key frame are activated in sequence.

[0041] The determination module is used to determine that the paper-tearing behavior in the video of the paper-tearing behavior has successfully occurred when all the primitives are activated.

[0042] As an improvement to the above scheme, the ontology library includes: behavior level, primitive level, keyframe level, and entity level; wherein,

[0043] The behavioral levels include: paper tearing behavior;

[0044] The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive;

[0045] The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule;

[0046] The physical levels include: left hand, right hand, heat sink, and sticker.

[0047] To achieve the above objectives, embodiments of the present invention also provide an ontology-based paper tearing behavior recognition device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the ontology-based paper tearing behavior recognition method as described above.

[0048] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the paper tearing behavior recognition method based on the ontology library as described above.

[0049] Compared with the prior art, the paper tearing behavior recognition method, apparatus, device and storage medium based on ontology library provided by the embodiments of the present invention have the following beneficial effects:

[0050] (1) By analyzing paper-tearing behavior based on an ontology database, rather than relying excessively on target detection algorithms, the problem of false detection, misdetection, and missed detection caused by small or obscured targets during the paper-tearing process can be effectively solved. This invention enables reasonable and effective supervision and identification of paper-tearing behavior even when stickers cannot be effectively detected in real time.

[0051] (2) Based on cognition, the essence of paper tearing behavior is expressed, enabling the machine to deeply understand industrial production behavior, achieving an organic combination of cognition and perception, with strong adaptability, generalization and transferability; for the industrial production scenarios faced by paper tearing behavior, if a large amount of data collection and retraining are required every time the scenario changes, it will consume huge human, material and financial resources. However, the present invention significantly reduces the migration cost compared with the existing technology, and can effectively face complex and ever-changing scenarios outside the dataset. Attached Figure Description

[0052] Figure 1 This is a flowchart of a paper tearing behavior recognition method based on an ontology library provided in an embodiment of the present invention;

[0053] Figure 2 This is an essential expression of the paper-tearing behavior in the embodiments of the present invention;

[0054] Figure 3This is a schematic diagram of the structure of a paper tearing behavior recognition device based on an ontology library provided in an embodiment of the present invention;

[0055] Figure 4 This is a structural block diagram of a paper tearing behavior recognition device based on an ontology library provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] See Figure 1 , Figure 1 This is a flowchart of an ontology-based paper tearing behavior recognition method provided in an embodiment of the present invention. The ontology-based paper tearing behavior recognition method includes:

[0058] S1. Construct an ontology library for paper tearing behavior; wherein, the paper tearing behavior is described hierarchically in the ontology library, the paper tearing behavior is divided into three primitives according to the chronological order, the primitives are represented by keyframes, and keyframe rules are established for each keyframe.

[0059] S2. Acquire the video of the paper-tearing behavior to be identified, and perform target detection on each frame of the video to obtain the target candidate box of each frame; wherein, the target candidate box includes: left-hand candidate box, right-hand candidate box and heat sink candidate box;

[0060] S3. Use the target candidate box to determine whether the image where the target candidate box is located satisfies the corresponding key frame rule in chronological order. When it is determined that there are three frames that satisfy the corresponding key frame rule in chronological order, activate the corresponding key frame and the primitive corresponding to the key frame in sequence.

[0061] S4. When all the primitives are activated, it is determined that the paper-tearing behavior in the paper-tearing behavior video has successfully occurred.

[0062] Specifically, the ontology library includes: behavior-level, primitive-level, keyframe-level, and entity-level; wherein,

[0063] The behavioral levels include: paper tearing behavior;

[0064] The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive;

[0065] The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule;

[0066] The physical levels include: left hand, right hand, heat sink, and sticker.

[0067] It is understandable that, from a cognitive perspective, the act of peeling off the sticker on the heatsink must follow the following pattern: ① One hand holds the heatsink; ② The other hand approaches, and the two hands interact; ③ The hands separate, and the hand holding the heatsink moves towards the place where the heatsink is placed.

[0068] like Figure 2 As shown, based on the above-mentioned paper-tearing behavior patterns, the embodiments of the present invention provide an essential expression of paper-tearing behavior and construct an ontology library of paper-tearing behavior.

[0069] The ontology library comprises four levels:

[0070] (1) Action level: tearing paper;

[0071] (2) Primitive-level: The act of tearing paper is broken down into three primitives in chronological order, including: holding the heat sink primitive, hands interacting primitive, and hands separating primitive;

[0072] (3) Frame-level: Primitives are represented by keyframes, including: keyframes of holding a heat sink, keyframes of hands interacting, and keyframes of hands separating; each keyframe has a corresponding keyframe rule.

[0073] (4) Entity-level: The entities involved in the tearing of paper include the left hand, right hand, heat sink and sticker.

[0074] In an optional embodiment, the keyframe rule for the keyframe of the hand holding the heatsink includes one of the following:

[0075] The candidate boxes for the heat sink and the candidate boxes for the left hand intersect;

[0076] The candidate boxes for the heat sink and the candidate boxes for the right hand overlap.

[0077] It is understandable that holding the heatsink means starting to peel off the sticker on the heatsink. Therefore, when any frame of the image meets the keyframe rule of the hand holding the heatsink keyframe, that frame of the image is taken as the hand holding the heatsink keyframe and activated, thereby activating the hand holding the heatsink primitive corresponding to the hand holding the heatsink keyframe.

[0078] In an optional embodiment, the keyframe rules for the two-handed interaction keyframes include one of the following:

[0079] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is less than a preset first distance threshold.

[0080] The left-hand candidate box and the right-hand candidate box have an intersection;

[0081] The left-hand candidate box and the right-hand candidate box intersect with the heat sink candidate box, and the overlap rate of the left-hand candidate box and the right-hand candidate box on the vertical axis is greater than a preset overlap rate.

[0082] It is understandable that the two-handed interaction represents peeling off the sticker on the heat sink. Therefore, when any frame image after the keyframe of the hand holding the heat sink satisfies the keyframe rule of the two-handed interaction keyframe, that frame image is taken as the two-handed interaction keyframe and activated, thereby activating the two-handed interaction key primitive corresponding to the two-handed interaction keyframe.

[0083] It's understandable that when peeling off the sticker on the heatsink, there might be situations where the left-hand candidate box and the right-hand candidate box don't intersect, but they do intersect with the heatsink candidate box. Since both hands are peeling off the sticker, they are on the same horizontal plane, so the candidate boxes of both hands are almost identical in position on the vertical axis. For example, if we take the bottom left corner of the left-hand candidate box as the origin, then the vertical coordinate of the top left corner of the left-hand candidate box is 100, and the vertical coordinate of the top left corner of the right-hand candidate box is 90. Therefore, the overlap rate between the left-hand and right-hand candidate boxes on the vertical axis is (90 / 100)*100% = 90%.

[0084] In an optional embodiment, the keyframe rule for the hand separation keyframe includes one of the following:

[0085] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is greater than a preset second distance threshold; wherein, the second distance threshold is greater than the first distance threshold;

[0086] The left-hand candidate box and the right-hand candidate box do not overlap.

[0087] It is understandable that the separation of the hands indicates that the sticker has been removed. Therefore, when any frame image after the separation of the hands keyframe satisfies the keyframe rule of the separation of the hands keyframe, the frame image is taken as the separation of the hands keyframe and activated, thereby activating the separation of the hands primitive corresponding to the separation of the hands keyframe.

[0088] Specifically, the step of using the target candidate box to determine whether the image containing the target candidate box satisfies the corresponding keyframe rules in chronological order, and when it is determined that three images satisfy the corresponding keyframe rules in chronological order, sequentially activating the corresponding keyframes and the primitives corresponding to the keyframes, includes:

[0089] Determine whether the target candidate box of each frame image satisfies the keyframe rule of the hand holding the heat sink keyframe. When it is determined that there is a target candidate box of the first current frame image that satisfies the keyframe rule of the hand holding the heat sink keyframe, the first current frame image is used as the hand holding the heat sink keyframe and the hand holding the heat sink keyframe is activated.

[0090] When the keyframe of the hand holding the heat sink is activated, the hand holding the heat sink primitive is activated;

[0091] When the hand holding the heat sink primitive is activated, it continues to determine whether the target candidate box of each frame image after the hand holding the heat sink keyframe satisfies the keyframe rule of the two-hand interaction keyframe. When it is determined that there is a target candidate box of the second current frame image that satisfies the keyframe rule of the two-hand interaction keyframe, the second current frame image is used as the two-hand interaction keyframe, and the two-hand interaction keyframe is activated.

[0092] When the two-hand interaction keyframe is activated, the two-hand interaction primitive is activated;

[0093] When the two-hand interaction primitive is activated, it continues to determine whether the target candidate box of each frame image after the two-hand separation keyframe satisfies the keyframe rule of the two-hand separation keyframe. When it is determined that the target candidate box of the third current frame image satisfies the keyframe rule of the two-hand separation keyframe, the third current frame image is taken as the two-hand separation keyframe and the two-hand separation keyframe is activated.

[0094] When the hand separation keyframe is activated, the hand separation primitive is activated.

[0095] Understandably, since an ontology library describes the things themselves, the ontology library for the paper-tearing behavior has fixed patterns in both space and time. During testing, camera data (video of the paper-tearing behavior to be identified) is first read in real time. Spatially, each frame is processed using the YOLOv5 object detection algorithm to detect objects, which can detect the left hand and / or right hand and / or heat sink and / or sticker, thus obtaining candidate bounding boxes for the left hand and / or right hand and / or heat sink and / or sticker, and then determining the relationship between them. Temporally, each captured image frame is detected and judged. If it meets the established keyframe rules, the keyframe is activated, thereby activating the corresponding primitive. Primitive activation must also be sequential; if a preceding primitive is not activated, subsequent primitives cannot be activated. When the primitives are activated sequentially in time, the paper-tearing behavior successfully occurs, meaning the sticker is successfully torn from the heat sink.

[0096] In an optional embodiment, the target candidate box further includes a sticker candidate box. Then, when all the primitives are activated, the ontology-based paper tearing behavior recognition method further includes:

[0097] For any frame image after the three frames, if it is determined that there is a first intersection between the heat sink candidate frame and the sticker candidate frame, and the relative size of the first intersection is greater than a preset relative size threshold, the paper tearing behavior in the video is determined to be unsuccessful.

[0098] It is understandable that although the object detection algorithm cannot effectively capture the sticker in every frame of the image, the recognition method of the present invention can still be further improved by using the occasionally captured sticker candidate boxes. As long as the sticker is determined to be on the heat sink after the hand separation primitive is activated, the paper tearing behavior can be considered to have failed. In order to more accurately identify whether the paper tearing behavior has been successfully achieved, the present invention determines whether there is a first intersection between the heat sink candidate box and the sticker candidate box in any frame of the image after the three frames when all primitives are activated, and the first intersection is greater than a preset relative size threshold. If it exists, it means that the paper tearing behavior has failed and the sticker has not been successfully torn off.

[0099] In an optional embodiment, the step of performing target detection on each frame of the video of the paper-tearing behavior to obtain a target candidate box for each frame includes:

[0100] Target detection is performed on each frame of the video of the paper-tearing behavior;

[0101] When multiple left hands are detected in any frame of an image, the distance between the center point of the candidate box of each left hand and the center point of the candidate box of the pre-set working area of ​​both hands is calculated, and the candidate box of the left hand with the shortest distance is taken as the detected target candidate box.

[0102] When multiple right hands are detected in any frame of an image, the distance between the center point of the candidate box of each right hand and the center point of the candidate box of the working area of ​​both hands is calculated, and the candidate box of the right hand with the shortest distance is taken as the detected target candidate box.

[0103] Understandably, real factory environments are very complex. If the operating space is crowded, the camera may detect multiple hands simultaneously. If these hands are not properly filtered, it will affect the algorithm's operation. This embodiment of the invention could have used the idea of ​​"nearer objects appear larger and farther objects appear smaller" to filter out two hands that are currently working from multiple hands. However, since the absolute distance in the image is only a projection of the real three-dimensional space onto two-dimensional space and does not take into account depth, simply relying on the size of the candidate box is insufficient to filter out the hands that are currently working.

[0104] Because the camera position and angle are fixed, and the processing control panel is also fixed in the lower-middle area of ​​the image, the area where the hands are active during the operation is close to the lower-middle area of ​​the image. Therefore, this embodiment of the invention specifically sets up a two-hand operation area to address the problem of multiple hands. The lower-middle area of ​​the image is set in the two-hand operation area. When multiple hands appear in the image, the distance of each hand from the two-hand operation area is calculated, and the two hands closest to the two-hand operation area are selected.

[0105] See Figure 3 , Figure 3 This is a structural block diagram of an ontology-based paper tearing behavior recognition device 10 provided in an embodiment of the present invention. The ontology-based paper tearing behavior recognition device 10 includes:

[0106] The ontology library construction module 11 is used to construct an ontology library for the paper tearing behavior; wherein, the paper tearing behavior is described hierarchically in the ontology library, the paper tearing behavior is divided into three primitives according to the time sequence, the primitives are represented by keyframes, and keyframe rules are established for each keyframe.

[0107] The target detection module 12 is used to acquire the paper tearing behavior video to be identified, and to perform target detection on each frame of the paper tearing behavior video to obtain the target candidate box of each frame; wherein, the target candidate box includes: left-hand candidate box, right-hand candidate box and heat sink candidate box;

[0108] Activation module 13 is used to determine whether the image containing the target candidate box satisfies the corresponding key frame rules in chronological order using the target candidate box. When it is determined that there are three frames that satisfy the corresponding key frame rules in chronological order, the corresponding key frame and the primitive corresponding to the key frame are activated in sequence.

[0109] The determination module 14 is used to determine that the paper-tearing behavior in the paper-tearing behavior video has successfully occurred when all the primitives are activated.

[0110] Preferably, the ontology library includes: behavior-level, primitive-level, keyframe-level, and entity-level; wherein,

[0111] The behavioral levels include: paper tearing behavior;

[0112] The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive;

[0113] The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule;

[0114] The physical levels include: left hand, right hand, heat sink, and sticker.

[0115] Preferably, the keyframe rule for the keyframe of the hand holding the heat sink includes one of the following:

[0116] The candidate boxes for the heat sink and the candidate boxes for the left hand intersect;

[0117] The candidate boxes for the heat sink and the right-hand candidate boxes have an intersection;

[0118] The keyframe rules for the two-handed interaction keyframes include one of the following:

[0119] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is less than a preset first distance threshold.

[0120] The left-hand candidate box and the right-hand candidate box have an intersection;

[0121] The left-hand candidate box and the right-hand candidate box intersect with the heat sink candidate box, and the overlap rate of the left-hand candidate box and the right-hand candidate box on the vertical axis is greater than the preset overlap rate.

[0122] The keyframe rule for the hand separation keyframe includes one of the following:

[0123] The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is greater than a preset second distance threshold; wherein, the second distance threshold is greater than the first distance threshold;

[0124] The left-hand candidate box and the right-hand candidate box do not overlap.

[0125] Preferably, the step of using the target candidate box to determine whether the images containing the target candidate box sequentially satisfy the corresponding keyframe rules in chronological order, and when it is determined that three images sequentially satisfy the corresponding keyframe rules in chronological order, sequentially activating the corresponding keyframes and the primitives corresponding to the keyframes, includes:

[0126] Determine whether the target candidate box of each frame image satisfies the keyframe rule of the hand holding the heat sink keyframe. When it is determined that there is a target candidate box of the first current frame image that satisfies the keyframe rule of the hand holding the heat sink keyframe, the first current frame image is used as the hand holding the heat sink keyframe and the hand holding the heat sink keyframe is activated.

[0127] When the keyframe of the hand holding the heat sink is activated, the hand holding the heat sink primitive is activated;

[0128] When the hand holding the heat sink primitive is activated, it continues to determine whether the target candidate box of each frame image after the hand holding the heat sink keyframe satisfies the keyframe rule of the two-hand interaction keyframe. When it is determined that there is a target candidate box of the second current frame image that satisfies the keyframe rule of the two-hand interaction keyframe, the second current frame image is used as the two-hand interaction keyframe, and the two-hand interaction keyframe is activated.

[0129] When the two-hand interaction keyframe is activated, the two-hand interaction primitive is activated;

[0130] When the two-hand interaction primitive is activated, it continues to determine whether the target candidate box of each frame image after the two-hand separation keyframe satisfies the keyframe rule of the two-hand separation keyframe. When it is determined that the target candidate box of the third current frame image satisfies the keyframe rule of the two-hand separation keyframe, the third current frame image is taken as the two-hand separation keyframe and the two-hand separation keyframe is activated.

[0131] When the hand separation keyframe is activated, the hand separation primitive is activated.

[0132] Preferably, the target candidate box further includes a sticker candidate box, then the determination module 14 is further configured to:

[0133] For any frame image after the three frames, if it is determined that there is a first intersection between the heat sink candidate frame and the sticker candidate frame, and the relative size of the first intersection is greater than a preset relative size threshold, the paper tearing behavior in the video is determined to be unsuccessful.

[0134] Preferably, the step of performing target detection on each frame of the video of the paper-tearing behavior to obtain a target candidate box for each frame includes:

[0135] Target detection is performed on each frame of the video of the paper-tearing behavior;

[0136] When multiple left hands are detected in any frame of an image, the distance between the center point of the candidate box of each left hand and the center point of the candidate box of the pre-set working area of ​​both hands is calculated, and the candidate box of the left hand with the shortest distance is taken as the detected target candidate box.

[0137] When multiple right hands are detected in any frame of an image, the distance between the center point of the candidate box of each right hand and the center point of the candidate box of the working area of ​​both hands is calculated, and the candidate box of the right hand with the shortest distance is taken as the detected target candidate box.

[0138] It is worth noting that the working process of each module in the ontology-based paper tearing behavior recognition device 10 described in the embodiments of the present invention can refer to the working process of the ontology-based paper tearing behavior recognition method described in the above embodiments, and will not be repeated here.

[0139] This invention provides a computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the paper tearing behavior recognition method based on the ontology library as described in the above embodiments.

[0140] See Figure 4 , Figure 4 This is a structural block diagram of an ontology-based paper tearing behavior recognition device 20 provided in an embodiment of the present invention. The ontology-based paper tearing behavior recognition device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described ontology-based paper tearing behavior recognition method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0141] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the ontology-based paper tearing behavior recognition device 20.

[0142] The ontology-based paper-tearing behavior recognition device 20 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The ontology-based paper-tearing behavior recognition device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the ontology-based paper-tearing behavior recognition device 20 and does not constitute a limitation on the ontology-based paper-tearing behavior recognition device 20. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the ontology-based paper-tearing behavior recognition device 20 may also include input / output devices, network access devices, buses, etc.

[0143] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the ontology-based paper tearing behavior recognition device 20, connecting all parts of the ontology-based paper tearing behavior recognition device 20 via various interfaces and lines.

[0144] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the paper tearing behavior recognition device 20 based on the ontology library by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0145] The modules / units integrated into the ontology-based paper-tearing behavior recognition device 20, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0146] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0147] Compared with the prior art, the paper tearing behavior recognition method, apparatus, device and storage medium based on ontology library provided by the embodiments of the present invention have the following beneficial effects:

[0148] (1) By analyzing paper-tearing behavior based on an ontology database, rather than relying excessively on target detection algorithms, the problem of false detection, misdetection, and missed detection caused by small or obscured targets during the paper-tearing process can be effectively solved. This invention enables reasonable and effective supervision and identification of paper-tearing behavior even when stickers cannot be effectively detected in real time.

[0149] (2) Based on cognition, the essence of paper tearing behavior is expressed, enabling the machine to deeply understand industrial production behavior, achieving an organic combination of cognition and perception, with strong adaptability, generalization and transferability; for the industrial production scenarios faced by paper tearing behavior, if a large amount of data collection and retraining are required every time the scenario changes, it will consume huge human, material and financial resources. However, the present invention significantly reduces the migration cost compared with the existing technology, and can effectively face complex and ever-changing scenarios outside the dataset.

[0150] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A paper-tearing behavior recognition method based on an ontology database, characterized in that, include: Construct an ontology library for the paper-tearing behavior; wherein, the paper-tearing behavior is described hierarchically in the ontology library, the paper-tearing behavior is divided into three primitives according to the chronological order, the primitives are represented by keyframes, and keyframe rules are established for each keyframe. A video of the paper-tearing behavior to be identified is acquired, and target detection is performed on each frame of the paper-tearing behavior video to obtain target candidate boxes for each frame; wherein, the target candidate boxes include: left-hand candidate boxes, right-hand candidate boxes, and heat sink candidate boxes; The target candidate box is used to determine whether the image containing the target candidate box satisfies the corresponding key frame rules in chronological order. When it is determined that there are three frames that satisfy the corresponding key frame rules in chronological order, the corresponding key frame and the primitive corresponding to the key frame are activated in sequence. When all the aforementioned primitives are activated, it is determined that the paper-tearing behavior in the video of the paper-tearing behavior has successfully occurred.

2. The paper tearing behavior recognition method based on ontology library as described in claim 1, characterized in that, The ontology library includes: behavior-level, primitive-level, keyframe-level, and entity-level; among which... The behavioral levels include: paper tearing behavior; The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive; The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule; The physical levels include: left hand, right hand, heat sink, and sticker.

3. The paper tearing behavior recognition method based on ontology library as described in claim 2, characterized in that, The keyframe rules for the keyframe of the hand holding the heat sink include one of the following: The candidate boxes for the heat sink and the candidate boxes for the left hand intersect; The candidate boxes for the heat sink and the right-hand candidate boxes have an intersection; The keyframe rules for the two-handed interaction keyframes include one of the following: The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is less than a preset first distance threshold. The left-hand candidate box and the right-hand candidate box have an intersection; The left-hand candidate box and the right-hand candidate box intersect with the heat sink candidate box, and the overlap rate of the left-hand candidate box and the right-hand candidate box on the vertical axis is greater than the preset overlap rate. The keyframe rule for the hand separation keyframe includes one of the following: The distance between the center point of the left-hand candidate box and the center point of the right-hand candidate box is greater than a preset second distance threshold; wherein, the second distance threshold is greater than the first distance threshold; The left-hand candidate box and the right-hand candidate box do not overlap.

4. The paper tearing behavior recognition method based on ontology library as described in claim 2, characterized in that, The step of using the target candidate box to determine whether the image containing the target candidate box satisfies the corresponding keyframe rules in chronological order, and when it is determined that three images satisfy the corresponding keyframe rules in chronological order, sequentially activating the corresponding keyframes and the primitives corresponding to the keyframes, includes: Determine whether the target candidate box of each frame image satisfies the keyframe rule of the hand holding the heat sink keyframe. When it is determined that there is a target candidate box of the first current frame image that satisfies the keyframe rule of the hand holding the heat sink keyframe, the first current frame image is used as the hand holding the heat sink keyframe and the hand holding the heat sink keyframe is activated. When the keyframe of the hand holding the heat sink is activated, the hand holding the heat sink primitive is activated; When the hand holding the heat sink primitive is activated, it continues to determine whether the target candidate box of each frame image after the hand holding the heat sink keyframe satisfies the keyframe rule of the two-hand interaction keyframe. When it is determined that there is a target candidate box of the second current frame image that satisfies the keyframe rule of the two-hand interaction keyframe, the second current frame image is used as the two-hand interaction keyframe, and the two-hand interaction keyframe is activated. When the two-hand interaction keyframe is activated, the two-hand interaction primitive is activated; When the two-hand interaction primitive is activated, it continues to determine whether the target candidate box of each frame image after the two-hand separation keyframe satisfies the keyframe rule of the two-hand separation keyframe. When it is determined that the target candidate box of the third current frame image satisfies the keyframe rule of the two-hand separation keyframe, the third current frame image is taken as the two-hand separation keyframe and the two-hand separation keyframe is activated. When the hand separation keyframe is activated, the hand separation primitive is activated.

5. The paper tearing behavior recognition method based on ontology library as described in claim 2, characterized in that, The target candidate box also includes a sticker candidate box. Therefore, when all the primitives are activated, the ontology-based paper tearing behavior recognition method further includes: For any frame image after the three frames, if it is determined that there is a first intersection between the heat sink candidate frame and the sticker candidate frame, and the relative size of the first intersection is greater than a preset relative size threshold, the paper tearing behavior in the video is determined to be unsuccessful.

6. The paper tearing behavior recognition method based on ontology library as described in claim 1, characterized in that, The step of performing target detection on each frame of the video of the paper-tearing behavior to obtain the target candidate box for each frame includes: Target detection is performed on each frame of the video of the paper-tearing behavior; When multiple left hands are detected in any frame of an image, the distance between the center point of the candidate box of each left hand and the center point of the candidate box of the pre-set working area of ​​both hands is calculated, and the candidate box of the left hand with the shortest distance is taken as the detected target candidate box. When multiple right hands are detected in any frame of an image, the distance between the center point of the candidate box of each right hand and the center point of the candidate box of the working area of ​​both hands is calculated, and the candidate box of the right hand with the shortest distance is taken as the detected target candidate box.

7. A paper-tearing behavior recognition device based on an ontology library, characterized in that, include: An ontology library construction module is used to construct an ontology library for the paper-tearing behavior; wherein, the paper-tearing behavior is described hierarchically in the ontology library, the paper-tearing behavior is divided into three primitives according to the chronological order, the primitives are represented by keyframes, and keyframe rules are established for each keyframe. The target detection module is used to acquire the paper-tearing behavior video to be identified, and to perform target detection on each frame of the paper-tearing behavior video to obtain the target candidate box of each frame; wherein, the target candidate box includes: left-hand candidate box, right-hand candidate box and heat sink candidate box; The activation module is used to determine whether the images where the target candidate box is located satisfy the corresponding key frame rules in chronological order. When it is determined that there are three frames that satisfy the corresponding key frame rules in chronological order, the corresponding key frame and the primitives corresponding to the key frame are activated in sequence. The determination module is used to determine that the paper-tearing behavior in the video of the paper-tearing behavior has successfully occurred when all the primitives are activated.

8. The paper tearing behavior recognition device based on ontology library as described in claim 7, characterized in that, The ontology library includes: behavior-level, primitive-level, keyframe-level, and entity-level; among which... The behavioral levels include: paper tearing behavior; The primitive levels include: hand holding heat sink primitive, two-hand interaction primitive, and two-hand separation primitive; The keyframe levels include: keyframes of hand holding heat sink, keyframes of hands interacting, and keyframes of hands separated; wherein, each of the keyframes has a corresponding keyframe rule; The physical levels include: left hand, right hand, heat sink, and sticker.

9. A paper-tearing behavior recognition device based on an ontology library, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the paper tearing behavior recognition method based on any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the paper tearing behavior recognition method based on any one of claims 1 to 7.

Images