Behavior recognition method and device, electronic equipment, storage medium and product
By combining object detection and posture analysis in video image processing, the problem of low accuracy in the behavior of identifying personnel using designated items in the prior art is solved, and higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202411078123.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is very accurate when identifying the behavior of a person using a designated item, especially when the person is far away from the designated item or his posture is unclear, misjudgment is easily caused.
By obtaining the video images of the target scene, target detection, area overlap calculation and posture detection are performed, and combined with the information on the person holding the designated item and the use posture in his hands, it is determined whether there is any behavior of the person using the designated item.
Improve the accuracy of the behavior of identifying personnel using designated items, reduce misjudgments, and ensure the accuracy of the detection results.
Smart Images

Figure CN119028015A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a behavior recognition method, device, electronic equipment, storage medium and product. Background Art
[0002] In order to ensure work efficiency, it is necessary to avoid unrelated behaviors in the workplace as much as possible. For example, people who use mobile devices in the target scene can be reminded not to use the mobile devices to avoid unrelated behaviors in the target scene. The mobile device can be a smart phone or a mini tablet, etc., and the target scene can be a classroom, a library, an office, etc.
[0003] In order to strengthen the supervision of designated behaviors not related to work in the target scene, it is necessary to detect the behavior of personnel using designated items. It can be seen that how to improve the accuracy of identifying the behavior of personnel using designated items is an urgent problem to be solved. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a behavior recognition method, device, electronic device, storage medium and product to improve the accuracy of identifying the behavior of a person using a specified object. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present invention provides a behavior recognition method, the method comprising:
[0006] Obtain a video image containing a target scene as the current video frame to be detected;
[0007] Performing target detection on the current video frame to be detected, determining a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected;
[0008] Calculate the overlap of the regions according to the obtained positions of the first image region, the second image region, and the third image region, and obtain a first detection result indicating whether a person in each first image region holds a designated object;
[0009] For each first image region in the current video frame to be detected, performing posture detection on the first image region to obtain a second detection result indicating whether there is a posture of using a designated object in the first image region;
[0010] The detection result of the current video frame to be detected is obtained according to the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is a behavior of a person using a specified object in the current video frame to be detected.
[0011] Optionally, the calculating of the degree of overlap of the regions according to the obtained positions of the first image region, the second image region, and the third image region to obtain a first detection result indicating whether a person in each first image region holds a designated object in his hand includes:
[0012] For each first image region in the current video frame to be detected, according to the position of the first image region and the position of each second image region, calculate a first degree of overlap between the first image region and each second image region;
[0013] In the obtained second image region, determining a second image region whose first coincidence degree is greater than a first threshold value as a second image region matching the first image region;
[0014] Calculating a second degree of overlap between the second image region matching the first image region and each third image region according to the position of the second image region matching the first image region and the position of each third image region;
[0015] In the obtained third image area, a third image area with a second degree of overlap greater than a second threshold is determined as the third image area matching the second image area, and the first detection result of the first image area is determined to be that the person in the first image area holds a designated item in his hand.
[0016] Optionally, the calculating of the degree of overlap of the regions according to the obtained positions of the first image region, the second image region, and the third image region to obtain a first detection result indicating whether a person in each first image region holds a designated object in his hand includes:
[0017] For each third image region in the current video frame to be detected, according to the position of the third image region and the position of each second image region, calculate a third degree of overlap between the third image region and each second image region;
[0018] In the obtained second image region, determining a second image region whose third coincidence degree is greater than a second threshold value as a second image region matching the third image region;
[0019] Calculating a fourth degree of overlap between the second image region matching the third image region and each first image region according to the position of the second image region matching the third image region and the position of each first image region;
[0020] For the first image area in which the fourth degree of overlap is greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is holding a designated item in his hands; for the first image area in which the fourth degree of overlap is not greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is not holding the designated item in his hands.
[0021] Optionally, obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area includes:
[0022] When the first detection result of the first image area indicates that a person in the first image area is holding a designated object in his hand, and the second detection result of the first image area indicates that there is a gesture of using the designated object in the first image area, a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected is determined.
[0023] Optionally, before obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area, the method further includes:
[0024] Obtain alternative detection results of the current video frame to be detected; wherein the alternative detection results of the current video frame to be detected include: a third detection result of the current video frame to be detected, and / or a fourth detection result of the current video frame to be detected; the third detection result of the current video frame to be detected is: performing posture detection on the second image area matching the first image area, and obtaining a detection result indicating whether a posture of using a specified object exists in the image of the second image area matching the first image area; the fourth detection result of the current video frame to be detected is: performing designated object detection on the third image area matching the second image area, and obtaining a detection result indicating whether a designated object exists in the third image area matching the second image area;
[0025] The step of obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area includes:
[0026] The detection result of the current video frame to be detected is obtained according to the first detection result and the second detection result of the first image area and the alternative detection result.
[0027] Optionally, obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area, and the alternative detection result, includes:
[0028] When the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that there is a gesture of using the designated object in the first image region, and the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated object in the second image region, determining a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected;
[0029] When the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that there is a gesture of using the designated object in the first image region, and the fourth detection result of the current video frame to be detected indicates that there is a designated object in the third image region, determining a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected;
[0030] When the first detection result of the first image area indicates that a person in the first image area is holding a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated item in the second image area, and the fourth detection result of the current video frame to be detected indicates that there is a designated item in the third image area, a detection result indicating that there is an act of a person using the designated item in the current video frame to be detected is determined.
[0031] Optionally, the method further includes:
[0032] After obtaining the detection result of the current video frame to be detected, determining the next video frame of the current video frame to be detected from the video image containing the target scene as the current video frame to be detected;
[0033] and returning to the step of performing target detection on the current video frame to be detected, and determining that the current video frame to be detected contains a first image area of a person, a second image area of a hand holding a designated object, and a third image area of the designated object;
[0034] If the detection results of a specified number of consecutive video frames to be detected indicate that there is a behavior of a person using a specified object, it is determined that the final detection result of the target scene is that there is a behavior of a person using a specified object.
[0035] Optionally, the method further includes:
[0036] After determining that the final detection result of the target scene is that there is a behavior of a person using a designated object, a video frame is selected from the video frames whose detection results indicate that there is a behavior of a person using the designated object, and is displayed as an alarm image.
[0037] In a second aspect, an embodiment of the present invention provides a behavior recognition device, the device comprising:
[0038] A first acquisition module is used to acquire a video image containing a target scene as a current video frame to be detected;
[0039] A first determination module is used to perform target detection on the current video frame to be detected, and determine a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected;
[0040] A second determination module is used to calculate the overlap of the regions according to the obtained positions of the first image region, the second image region and the third image region, and obtain a first detection result indicating whether a person in each first image region holds a designated object;
[0041] A third determination module is used to perform posture detection on each first image area in the current video frame to be detected, and obtain a second detection result indicating whether there is a posture of using a specified object in the first image area;
[0042] The fourth determination module is used to obtain the detection result of the current video frame to be detected based on the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is any behavior of a person using a specified object in the current video frame to be detected.
[0043] An embodiment of the present invention further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0044] Memory, used to store computer programs;
[0045] The processor is used to implement the behavior recognition method when executing the program stored in the memory.
[0046] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the behavior recognition method is implemented.
[0047] An embodiment of the present invention further provides a computer program product, including a computer program, and the computer program implements the behavior recognition method when executed by a processor.
[0048] Beneficial effects of the embodiments of the present invention:
[0049] A behavior recognition method provided by an embodiment of the present invention can obtain a video image containing a target scene as a current video frame to be detected; perform target detection on the current video frame to be detected, and determine whether the current video frame to be detected contains a first image area of a person, a second image area containing a hand holding a specified object, and a third image area containing a specified object; calculate the overlap of the areas based on the positions of the obtained first image areas, the second image areas, and the third image areas, and obtain a first detection result indicating whether the person in each first image area holds the specified object in his hand; perform posture detection on each first image area in the current video frame to be detected, and obtain a second detection result indicating whether there is a posture of using the specified object in the first image area; and obtain the detection result of the current video frame to be detected based on the first detection result and the second detection result of the first image area.
[0050] Based on the above processing, the overlap of the regions is calculated according to the positions of the first image region, the second image region, and the third image region, and it can be determined whether the second image region has a matching third image region to determine whether the hand in the second image region holds a designated item; according to whether the first image region has a matching second image region, it can be determined whether the person in the first image region holds a designated item in his hand, and an accurate first detection result is obtained. In addition, since the second detection result indicates whether there is a gesture of using the designated item in the first image region, the detection result can be obtained according to the first detection result and the second detection result, and the two aspects of whether the person holds the designated item in his hand and whether the person's gesture is a gesture of using the designated item can be combined for analysis. If the person holds the designated item in his hand and the person's gesture is a gesture of using the designated item, it can be accurately determined that there is a detection result of the person's behavior of using the designated item in the current video frame to be detected, and the detection result of the current video frame to be detected can be accurately obtained to improve the accuracy of identifying the person's behavior of using the designated item.
[0051] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0053] Figure 1 A schematic diagram of a flow chart of a first behavior recognition method provided by an embodiment of the present invention;
[0054] Figure 2 is a schematic diagram of a first image region, a second image region, and a third image region of a current video frame to be detected in an embodiment of the present invention;
[0055] Figure 3 A schematic diagram of a flow chart of a second behavior recognition method provided by an embodiment of the present invention;
[0056] Figure 4 A schematic diagram of a flow chart of a third behavior recognition method provided by an embodiment of the present invention;
[0057] Figure 5 A schematic diagram of a flow chart of a fourth behavior recognition method provided by an embodiment of the present invention;
[0058] Figure 6 A flowchart of a fifth behavior recognition method provided by an embodiment of the present invention;
[0059] Figure 7 A flowchart of a sixth behavior recognition method provided by an embodiment of the present invention;
[0060] Figure 8 A schematic diagram of the structure of a model structure of a target detection model provided by an embodiment of the present invention;
[0061] Fig. 9 A schematic diagram of the structure of a model structure of a first posture detection model provided by an embodiment of the present invention;
[0062] Fig.10 A flowchart of a seventh behavior recognition method provided by an embodiment of the present invention;
[0063] Fig.11 A schematic diagram of the structure of a behavior recognition device provided by an embodiment of the present invention;
[0064] Fig.12 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.
[0066] In order to more clearly understand the embodiments of the present invention, the prior art is first introduced below.
[0067] In the prior art, in order to detect designated behaviors that are not related to work, the designated objects are detected for the video frames of the surveillance video containing the target scene to determine whether the designated objects exist in the video frames; then, the posture features of the people in the video frames are extracted, and the posture features of the people are analyzed to determine whether the posture of the people in the video frames is the posture of using the designated objects. If the designated object exists in the video frame, and the posture of the people in the video frame is the posture of using the designated object, it is determined that there is a behavior of the person using the designated object. For example, if there is a mobile phone in the video frame, and the posture of the person is the posture of looking at the phone at the desk, it will be determined that there is a behavior of the person using the mobile phone.
[0068] However, in the prior art, the existence of a designated item in a video frame and the posture of a person in the video frame as the posture of using the designated item are used as the basis for analysis, without in-depth consideration of the logic of a person using the designated item. If in actual conditions, the person does not use the designated item, the designated item in the video frame is placed far away from the person, and the posture of the person in the video frame is the posture of using the designated item, then a judgment result that is contrary to the actual conditions will be obtained, that is, there is a person using the designated item. For example, in the video frame, a mobile phone is placed on a cabinet, and a person is bending over a desk to read paper documents. In actual conditions, the person does not use the mobile phone, but based on the prior art, it will be determined that there is a person using the mobile phone. Therefore, the accuracy of judging whether there is a person using the designated item based on the prior art is not high.
[0069] It can be seen that how to improve the accuracy of identifying people's behavior of using designated items is a problem that needs to be solved urgently.
[0070] In order to improve the accuracy of identifying a person's behavior of using a designated object, an embodiment of the present invention provides a behavior recognition method, device, electronic device, storage medium and product.
[0071] Among them, a behavior recognition method provided by an embodiment of the present invention can be applied to an electronic device. The electronic device can obtain a video frame containing a target scene collected by an image acquisition device, and can perform image analysis on the obtained video frame. In a specific application, the electronic device can be a server, and the target scene can be a place where a camera is set.
[0072] A behavior recognition method provided by an embodiment of the present invention may include the following steps:
[0073] Obtain a video image containing a target scene as the current video frame to be detected;
[0074] Performing target detection on the current video frame to be detected, determining a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected;
[0075] For each first image area in the current video frame to be detected, the region overlap is calculated according to the position of the first image area, the position of the obtained second image area, and the position of the third image area, to obtain a first detection result indicating whether a person in the first image area holds a designated object in his hand;
[0076] Performing posture detection on the first image area to obtain a second detection result indicating whether there is a posture of using a designated object in the first image area;
[0077] The detection result of the current video frame to be detected is obtained according to the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is a behavior of a person using a specified object in the current video frame to be detected.
[0078] Based on the above processing, the overlap of the regions is calculated according to the positions of the first image region, the second image region, and the third image region, and it can be determined whether the second image region has a matching third image region to determine whether the hand in the second image region holds a designated item; according to whether the first image region has a matching second image region, it can be determined whether the person in the first image region holds a designated item in his hand, and an accurate first detection result is obtained. In addition, since the second detection result indicates whether there is a gesture of using the designated item in the first image region, the detection result can be obtained according to the first detection result and the second detection result, and the two aspects of whether the person holds the designated item in his hand and whether the person's gesture is a gesture of using the designated item can be combined for analysis. If the person holds the designated item in his hand and the person's gesture is a gesture of using the designated item, it can be accurately determined that there is a detection result of the person's behavior of using the designated item in the current video frame to be detected, and the detection result of the current video frame to be detected can be accurately obtained to improve the accuracy of identifying the person's behavior of using the designated item.
[0079] The following describes a behavior recognition method provided by an embodiment of the present invention in conjunction with the accompanying drawings. Figure 1 As shown, the method may include steps S101-S105.
[0080] S101, obtaining a video image containing a target scene as a current video frame to be detected.
[0081] It is understandable that the electronic device can acquire a video containing a target scene. The electronic device can use a video frame in the video image containing the target scene as the current video frame to be detected. Optionally, in one implementation, the electronic device can continuously acquire video frames, and use a video frame acquired in real time as the current video frame to be detected for real-time processing. Optionally, in another implementation, the electronic device can acquire a video image containing multiple video frames, and in accordance with the order of the video frames in the video image, each video frame in the video image is sequentially processed as the current video frame to be detected in steps S102-S105 to obtain the detection result of the video frame; and then the next video frame of the video frame is used as the current video frame to be detected to perform steps S102-S105, until each video frame in the video image is processed to obtain the detection result of each video frame.
[0082] S102, performing target detection on the current video frame to be detected, and determining that the current video frame to be detected includes a first image region of a person, a second image region of a hand holding a designated object, and a third image region of the designated object.
[0083] It is understandable that target detection can be performed on the current video frame to be detected to determine three types of image areas contained in the video frame to be detected, wherein the three types of image areas are: a first image area containing a person, a second image area containing a hand holding a specified object, and a third image area containing the specified object. Moreover, there is at least one image area of each type determined, that is, there is at least one first image area, one second image area, and one third image area in the video frame to be detected.
[0084] Optionally, in one implementation, in the case where the video image is a video stream acquired in real time, the video stream is a long, uninterrupted video. In this case, the behavior of the target scene can be detected in real time. In the process of performing real-time detection on the video stream using the behavior recognition method, if the first image area, the second image area, or the third image area does not exist in the determined image area, subsequent steps S103-S105 will not be executed. Instead, the next video frame of the current video frame to be detected in the video stream is used as the current video frame to be detected, and step S102 is executed, that is, behavior recognition is performed on the next video frame.
[0085] Optionally, in another implementation, in the case where the video image is a segment of video, each video frame in the segment of video contains the behavior of a person using a specified object. In this case, the segment of video can be used to test the behavior recognition method provided by the embodiment of the present invention. In the process of using a segment of specified video to test the behavior recognition method provided by the embodiment of the present invention, if the first image area, the second image area, or the third image area does not exist in the determined image area, the subsequent steps S103-S105 will not be executed, and the preset processing will be executed to ensure that the behavior in the video frame to be detected can be successfully identified. Exemplarily, the preset processing can be to display the video frame to be detected as an alarm image to the administrator to issue an alarm to the administrator so that the administrator can manually check whether there are problems with the video frame to be detected and the behavior recognition method provided by the embodiment of the present invention. For example, it can be checked whether there are problems with the target detection model used for target detection.
[0086] Exemplarily, the current video frame to be detected can be input into a pre-trained target detection model to obtain the position of the first image area of a person, the position of the second image area of a hand holding a designated object, and the position of the third image area of the designated object in the video frame to be detected output by the target detection model; to determine that the current video frame to be detected contains the first image area of a person, the second image area of a hand holding a designated object, and the third image area of the designated object. The positions output by the target detection model can be: the coordinates of the first image area in the video frame to be detected, the coordinates of the second image area in the video frame to be detected, and the coordinates of the third image area in the video frame to be detected.
[0087] The target detection model is obtained by training based on the first sample image and the sample labels of the content contained in the first sample image. If the first sample image contains a person, the sample label indicates the location of the area in the first sample image containing the person; if the first sample image contains a hand holding a designated object, the sample label indicates the location of the area in the first sample image containing the hand holding the designated object; if the first sample image contains a designated object, the sample label indicates the location of the area in the first sample image containing the designated object. If the first sample image contains multiple contents, the sample label is a label indicating the location of the area of the multiple contents in the first sample image; for example, if the first sample image contains a person, a hand holding a designated object, and a designated object at the same time, the sample label includes: a sample label indicating the location of the area in the first sample image containing the person, a sample label indicating the location of the area in the first sample image containing the hand holding the designated object, and a sample label indicating the location of the area in the first sample image containing the designated object.
[0088] In order to more clearly introduce the embodiments of the present invention, the model structure of the target detection model will be described in detail in the subsequent embodiments.
[0089] It can be understood that the behavior of using the specified object refers to: behavior that will affect the working status in the target scene, and the specified object is an object that is unrelated to the work in the target scene.
[0090] Optionally, in one implementation, the designated item is a mobile device.
[0091] It is understandable that the designated object may be a mobile device, such as a mobile phone, a mini tablet computer, etc. Accordingly, the behavior detected by the embodiment of the present invention is: the behavior of a person using a mobile device, such as the behavior of a person playing with a mobile phone.
[0092] For example, Figure 2 is the current video frame to be detected, where the specified object is a mobile phone, such as Figure 2 As shown, frame 1 is the first image area containing a person in the video frame to be detected, and frame 1 can also be called a person frame; frame 2 is the second image area containing a hand holding a mobile device in the video frame to be detected, and frame 2 can also be called a hand holding mobile phone frame; frame 3 is the third image area containing a specified object in the video frame to be detected, and frame 3 can also be called a mobile phone frame.
[0093] Optionally, in another implementation, the designated item is food. It is understandable that eating in the workplace will also affect work efficiency, and the designated item can be food. Accordingly, the behavior detected by the embodiment of the present invention is: the behavior of a person eating.
[0094] S103, calculating the degree of overlap of the regions according to the obtained positions of the first image region, the second image region and the third image region, and obtaining a first detection result indicating whether a person in each first image region holds a designated object in his hand.
[0095] It is understandable that the area overlap calculation can be performed based on the obtained position of the first image area, the position of the second image area, and the position of the third image area to determine whether the first image area matches the second image area, and the second image area matches the third image area, so as to obtain a first detection result indicating whether the person in each first image area holds a designated object in his hand. Exemplarily, for each first image area in the current video frame to be detected, the first image area contains a person, and the overlap of the first image area with each second image area and the overlap of the second image area with the third image area can be calculated based on the position of the first image area, the obtained position of the second image area, and the position of the third image area.
[0096] The position of the first image area can be represented by the coordinates of the first image area in the video frame to be detected. For example, the horizontal pixels of the video frame to be detected are the pixels of the x-axis, and the vertical pixels are the pixels of the y-axis. The position of the first image area can be: the area position represented by the pixels of the coordinates x1 to x2 on the x-axis and the coordinates y1 to y2 on the y-axis. Similarly, similar to the position of the first image area, the positions of the second image area and the third image area can also be represented by the coordinates of the image areas in the video frame to be detected.
[0097] If the overlap between the third image area and any second image area exceeds a threshold, it can be determined that the second image area has a matching third image area. If the overlap between the first image area and any second image area exceeds a threshold, it can be determined that the first image area has a matching second image area, and then the first detection result of the person holding the designated object in the first image area is determined.
[0098] It can be understood that by calculating the degree of overlap between the first image area, any second image area and any third image area, the second image area and the third image area that match the first image area can be determined, thereby matching the person in the first image area with the hand holding the designated object and the designated object, that is, determining that the person in the first image area, the hand holding the designated object and the designated object belong to the same person.
[0099] Optionally, in one implementation, in the process of using the behavior recognition method to perform real-time detection on the video stream, if there is no matching third image area for any second image area, or there is no matching second image area for any first image area, the subsequent steps S104-S105 may not be executed, and the next video frame of the current video frame to be detected in the video stream is used as the current video frame to be detected, and step S102 is executed, that is, behavior recognition is performed on the next video frame. Optionally, in another implementation, in the process of using a specified video to test the behavior recognition method provided by the embodiment of the present invention, if there is no matching third image area for any second image area, or there is no matching second image area for any first image area, the subsequent steps S104-S105 may not be executed, and preset processing may be executed to issue an alarm to the administrator so that the administrator can manually detect whether there are problems with the video frame to be detected and the behavior recognition method provided by the embodiment of the present invention.
[0100] S104: For each first image region in the current video frame to be detected, perform posture detection on the first image region to obtain a second detection result indicating whether there is a posture of using a designated object in the first image region.
[0101] It is understandable that posture detection can be performed on the person in the first image area to determine the overall posture of the person and obtain a second detection result. That is to say, the second detection result indicating whether there is a posture of using a designated object in the first image area is a posture detection for the person as a whole. Exemplarily, the first image area can be input into a pre-trained first posture detection model to obtain a first detection result output by the first posture detection model indicating whether there is a posture of using a designated object in the first image area in the video frame to be detected; wherein the first posture detection model is trained based on the second sample image and a sample label indicating the presence of a posture of using a designated object in the second sample image.
[0102] In order to more clearly understand the embodiments of the present invention, the model structure of the first posture detection model involved in the embodiments of the present invention will be introduced in the subsequent embodiments.
[0103] It is understandable that, with respect to the process of determining the first detection result in step S103 and the process of determining the second detection result in step S104, the embodiment of the present invention does not limit the order of execution of these two steps, that is, step S103 may be executed first, and then step S104; or step S104 may be executed first, and then step S103; or step S103 and step S104 may be executed simultaneously. In this regard, the embodiment of the present invention is only used as an example for illustration, and no specific limitation is made.
[0104] S105: Obtain a detection result of a current video frame to be detected according to the first detection result and the second detection result of the first image area.
[0105] The detection result of the current video frame to be detected indicates whether there is a behavior of a person using a specified object in the current video frame to be detected.
[0106] It can be understood that if the first detection result indicates that the person in the first image area is holding a designated item in his hand, and the second detection result indicates that the person in the first image area is in a posture of using the designated item, then the detection result of the current video frame to be detected can be determined as: there is an action of a person using the designated item in the current video frame to be detected.
[0107] If the first detection result indicates that the person in the first image area does not hold the designated item in his hand, or the second detection result indicates that the person in the first image area does not have the posture of using the designated item, then the detection result of the current video frame to be detected can be determined as: there is no behavior of the person using the designated item in the current video frame to be detected.
[0108] Based on the above processing, the overlap of the regions is calculated according to the positions of the first image region, the second image region, and the third image region, and it can be determined whether the second image region has a matching third image region to determine whether the hand in the second image region holds a designated item; according to whether the first image region has a matching second image region, it can be determined whether the person in the first image region holds a designated item in his hand, and an accurate first detection result is obtained. In addition, since the second detection result indicates whether there is a gesture of using the designated item in the first image region, the detection result can be obtained according to the first detection result and the second detection result, and the two aspects of whether the person holds the designated item in his hand and whether the person's gesture is a gesture of using the designated item can be combined for analysis. If the person holds the designated item in his hand and the person's gesture is a gesture of using the designated item, it can be accurately determined that there is a detection result of the person's behavior of using the designated item in the current video frame to be detected, and the detection result of the current video frame to be detected can be accurately obtained to improve the accuracy of identifying the person's behavior of using the designated item.
[0109] Optionally, in one embodiment, as Figure 3 As shown, in Figure 1 Based on the behavior recognition method shown, step S103 includes steps S1031-S1034.
[0110] S1031 , for each first image region in the current video frame to be detected, calculate a first degree of overlap between the first image region and each second image region according to a position of the first image region and a position of each second image region.
[0111] It can be understood that, for each first image area, the position of the first image area can be represented by the coordinates of the first image area in the video frame to be detected; for each second image area, the position of the second image area can be represented by the coordinates of the second image area in the video frame to be detected.
[0112] Optionally, in one implementation, for each second image area, the intersection ratio of the second image area and the first image area can be calculated based on the coordinates of the second image area and the first image area. That is, the area of the intersection of the second image area and the first image area is divided by the area of the union of the second image area and the first image area to obtain the degree of overlap between the first image area and the second image area as the first degree of overlap.
[0113] Optionally, in another implementation, for each second image area, the area of the intersection of the second image area and the first image area can be divided by the area of the second image area based on the coordinates of the second image area and the first image area to obtain the degree of overlap between the first image area and the second image area as the first degree of overlap.
[0114] S1032: Determine, in the obtained second image regions, a second image region having a first degree of overlap greater than a first threshold value as a second image region matching the first image region.
[0115] It can be understood that the calculated degree of overlap between the first image area and each second image area is compared with the first threshold, and the second image area with a degree of overlap greater than the first threshold is determined as the second image area matching the first image area. Optionally, if there is no second image area with a degree of overlap greater than the first threshold with the first image area, the subsequent steps will not be executed, and the preset processing will be executed. Optionally, if there is a second image area with a degree of overlap greater than the first threshold with the first image area, the second image area can be determined as the second image area matching the first image area. Optionally, if there are multiple second image areas with a degree of overlap greater than the first threshold with the first image area, a second image area with the largest degree of overlap can be selected from the multiple second image areas as the second image area matching the first image area.
[0116] S1033: Calculate a second degree of overlap between the second image region matching the first image region and each third image region according to the position of the second image region matching the first image region and the position of each third image region.
[0117] It can be understood that, for each third image region, the position of the third image region can be represented by the coordinates of the third image region in the video frame to be detected.
[0118] Optionally, in one implementation, the intersection ratio of the third image area and the second image area matching the first image area can be calculated based on the coordinates of the third image area and the coordinates of the second image area matching the first image area. That is, the area of the intersection of the third image area and the second image area matching the first image area is divided by the union of the third image area and the second image area matching the first image area to obtain the degree of overlap between the second image area matching the first image area and the third image area as the second degree of overlap.
[0119] Optionally, in another implementation, based on the coordinates of the third image area and the coordinates of the second image area matching the first image area, the area of the intersection of the third image area and the second image area matching the first image area can be divided by the area of the third image area to obtain the degree of overlap between the second image area matching the first image area and the third image area as the second degree of overlap.
[0120] S1034, in the obtained third image area, determine a third image area with a second overlap greater than a second threshold as the third image area matching the second image area, and determine that the first detection result of the first image area is that the person in the first image area holds a designated item in his hand.
[0121] It can be understood that the calculated overlap between the second image area matching the first image area and each third image area is compared with the second threshold, and the third image area with a overlap greater than the second threshold is determined as the third image area matching the first image area. Optionally, if there is no third image area with a overlap greater than the second threshold with the second image area matching the first image area, it can be determined that there is no behavior of a person using a designated item in the video frame to be detected. Optionally, if there is a third image area with a overlap greater than the second threshold with the second image area matching the first image area, the third image area can be determined as the third image area matching the first image area. Optionally, if there are multiple third image areas with a overlap greater than the second threshold with the second image area matching the first image area, a third image area with the largest overlap can be selected from the multiple third image areas as the third image area matching the first image area.
[0122] After the above processing, the first image area has a matching second image area, and the second image area matching the first image area has a matching third image area, so it can be determined that the person in the first image area is holding a designated object.
[0123] In the embodiment of the present invention, for each first image area, the degree of overlap between the first image area and each second image area is calculated, and the second image area matching the first image area is determined. The degree of overlap between the second image area matching the first image area and each third image area is calculated, and the third image area matching the second image area matching the first image area is determined. When the first image area matches the second image area, and the second image area matches the third image area, the first image area, the second image area, and the third image area are matched. That is, for each first image area, the person in the first image area is matched with the hand-held designated item of the same person and the designated item. In the embodiment of the present invention, by determining whether the hand-held designated item of the same person and the designated item are matched for the first image area, it can be accurately determined that the first detection result of the first image area is that the hand of the person in the first image area holds the designated item.
[0124] It should be noted that not all matching of any two image areas among the first image area, the second image area, and the third image area can lead to a detection result that the person is holding a designated object.
[0125] Exemplarily, for each first image area, the degree of overlap between the first image area and each third image area is calculated to determine the third image area that matches the first image area, and then the degree of overlap between the third image area that matches the first image area and each second image area is calculated to determine the second image area that matches the third image area that matches the first image area. The first image area matches the third image area, and the third image area matches the third image area, but the first image area is not overlapped with the second image area, and the first image area may not match the second image area. In other words, it can only be determined that there is a designated item around the person in the first image area, but it is impossible to determine whether the person in the first image area is holding the designated item in his hand.
[0126] Exemplarily, for each first image area, the degree of overlap between the first image area and each third image area is calculated to determine the third image area that matches the first image area; the degree of overlap between the first image area and each second image area is calculated to determine the second image area that matches the first image area. The first image area matches the second image area, and the first image area matches the third image area, but the degree of overlap between the second image area and the third image area is not calculated, and the second image area may not match the third image area. In other words, it can only be determined that the person in the first image area holds an item in his hand and there is a designated item next to the person. It is impossible to determine whether the item held in the hand in the second image area and the designated item next to the person are the same designated item. In other words, it is impossible to determine whether the person in the first image area holds the designated item in his hand.
[0127] It is understandable that, in order to determine the detection result of whether a person holds a designated object in his hand, for the above step S103, in addition to the implementation method of processing each first image area in the above embodiment, there are other implementation methods.
[0128] Optionally, in one implementation, step S103 may include steps A1-A4.
[0129] A1, for each third image region in the current video frame to be detected, according to the position of the third image region and the position of each second image region, calculate a third overlap degree between the third image region and each second image region.
[0130] A2: In the obtained second image regions, determine a second image region whose third coincidence degree is greater than a second threshold value as a second image region matching the third image region.
[0131] A3, calculating a fourth degree of overlap between the second image region matching the third image region and each first image region according to the position of the second image region matching the third image region and the position of each first image region.
[0132] A4, for the first image area in which the fourth degree of overlap is greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is holding a designated item in his hand; for the first image area in which the fourth degree of overlap is not greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is not holding the designated item in his hand.
[0133] It can be understood that for each third image area in the current video frame to be detected, the third degree of overlap between the third image area and each second image area is calculated according to the position of the third image area and each second image area; if there is a second image area in the obtained second image area that matches the third image area and has a third degree of overlap greater than the second threshold, then the fourth degree of overlap between the second image area matching the third image area and each first image area is calculated according to the position of the second image area matching the third image area and the position of each first image area; if there is a first image area in the obtained first image area that has a fourth degree of overlap greater than the first threshold, then it can be determined that the first detection result of the first image area is that the person in the first image area holds a designated item in his hand. Among them, the specific implementation method of calculating the degree of match has been described in the above embodiment, and will not be repeated here. It should be noted that, in the present implementation, the first threshold for comparing the fourth degree of overlap between the first image area and the second image area may be the same as the first threshold in the method performed for each first image area, and both are criteria for judging the degree of overlap between the first image area and the second image area; similarly, the second threshold in the present implementation and the second threshold in the method performed for each first image area may be the same, and both are criteria for judging the degree of overlap between the second image area and the third image area.
[0134] It can be understood that in this implementation, for each third image area, the hand holding the designated object and the person to whom the matched hand belongs are matched for the designated object in the third image area, so as to determine the first image area, the second image area and the third image area belonging to the same person. For each first image area in the video frame to be detected, if the first image area matches the second image area matching the third image area, it can be determined that the first detection result of the first image area is that the person's hand holds the designated object; if the first image area does not match the second image area matching the third image area, it can be determined that the first detection result of the first image area is that the person does not hold the designated object in his hand. In the embodiment of the present invention, by determining whether the third image area matches the hand holding the designated object of the same person and the person to whom the hand belongs, it can be accurately determined that the first detection result of the matched first image area is that the person in the first image area holds the designated object in his hand, and the first detection result of the unmatched first image area is that the person in the first image area does not hold the designated object in his hand, and the first detection result of each first image area is obtained.
[0135] Optionally, in another embodiment, Figure 4 As shown, in Figure 1 Based on the behavior recognition method shown, step S105 includes step S1051.
[0136] S1051, when the first detection result in the first image area indicates that a person in the first image area is holding a designated object in his hand, and the second detection result in the first image area indicates that there is a gesture of using the designated object in the first image area, determine the detection result indicating that there is an action of a person using the designated object in the current video frame to be detected.
[0137] It is understandable that, generally speaking, if a person holds a designated object in his hand and his posture is that of using the designated object, then the person is likely to be using the designated object. For example, if a person holds a mobile phone in his hand and his posture is that of looking at a table, then the person is likely to be using the mobile phone.
[0138] For each first image area, if the first detection result of the first image area indicates that the person in the first image area is holding a designated object in his hand, and the second detection result of the first image area indicates that there is a gesture of using the designated object in the first image area, then it can be determined that there is an action of a person using the designated object in the current video frame to be detected.
[0139] If the first detection result indicates that the person in the first image area does not hold the designated item, it can be considered that the person has no possibility of using the designated item, and even if the second detection result indicates that there is a gesture of using the designated item in the first image area, it can be determined that there is no behavior of the person using the designated item in the current video frame to be detected. If the second detection result indicates that there is a gesture of using the designated item in the first image area, it can be considered that the person is not using the designated item, and even if the first detection result indicates that the person in the first image area holds the designated item, it can be determined that there is no behavior of the person using the designated item in the current video frame to be detected.
[0140] In an embodiment of the present invention, if a person's hand holds a designated object and the person's posture is that of using the designated object, it can be determined that the person is using the designated object. Taking into account the behavioral logic of the person using the object, it can be determined more accurately whether there is a person using the designated object in the video frame, and the omission of identifying the person's use of the designated object can be reduced.
[0141] Optionally, in another embodiment, Figure 5 As shown, in Figure 1 On the basis of the behavior recognition method shown, the behavior recognition method further includes step S501, and step S105 includes step S1052.
[0142] S501, obtaining candidate detection results of the current video frame to be detected.
[0143] Among them, the alternative detection results of the current video frame to be detected include: the third detection result of the current video frame to be detected, and / or the fourth detection result of the current video frame to be detected. The third detection result of the current video frame to be detected is: performing posture detection on the second image area matching the first image area, and the obtained detection result indicating whether the posture of using the specified object exists in the image of the second image area matching the first image area; the fourth detection result of the current video frame to be detected is: performing designated object detection on the third image area matching the second image area, and the obtained detection result indicating whether the designated object exists in the third image area matching the second image area.
[0144] It can be understood that in step S102, the first image area, the second image area and the third image area in the current video frame to be detected are determined. After step S102 and before step S105, in addition to determining the first detection result in step S103 and determining the second detection result in step S104, alternative detection results of the current video frame to be detected can also be obtained, providing more references for determining the detection result of the current video frame to be detected.
[0145] The embodiment of the present invention does not limit the execution order of step S301, step S103 and step S104, and the three steps can be executed simultaneously; step S103 and step S104 can also be executed first, and then step S301; step S301 can also be executed first, and then step S103 and step S104. In this regard, the embodiment of the present invention is only used as an example for illustration and is not specifically limited.
[0146] The candidate detection results of the current video frame to be detected include: the third detection result of the current video frame to be detected, and / or the fourth detection result of the current video frame to be detected. That is, the embodiment of the present invention can obtain the third detection result, or can obtain the fourth detection result, or can simultaneously obtain the third detection result and the fourth detection result.
[0147] For the third detection result of the current video frame to be detected, the posture of the hand holding the designated object in the second image area matching the first image area can be detected to obtain a detection result of whether there is a posture of using the designated object in the second image area matching the first image area. In other words, the third detection result indicating whether there is a posture of using the designated object in the second image area matching the first image area is a local posture detection result for the hand of the person.
[0148] Exemplarily, a second image area matching the first image area is input into a pre-trained second posture detection model, and a second posture detection result output by the second posture detection model is obtained, indicating whether there is an act of using a specified object in the second image area matching the first image area; wherein the second posture detection model is trained based on a third sample image and a sample label indicating the presence of an act of using a specified object in the third sample image.
[0149] With respect to the fourth detection result of the current video frame to be detected, the third image region matching the second image region may be detected for a designated object to obtain a detection result indicating whether the designated object exists in the third image region matching the second image region.
[0150] It is understandable that in the aforementioned embodiment, the third image area in the video frame to be detected is determined by the target detection model. Generally speaking, the size of the picture that the target detection model can process is limited. In the process of inputting the video frame to be detected into the target detection model, in order to adapt to the target detection model, the video frame to be detected will be resized to reduce the resolution of the video frame to be detected, resulting in a relatively blurred video frame to be detected. If the specified object is small, there is a possibility of misjudgment of the third image area determined by the target detection model. In an embodiment of the present invention, after using the target detection model to determine the third image area containing the specified object, the third image area matching the second image area can be detected again for the specified object, so as to further determine whether there is a specified object in the third image area matching the second image area on the basis of the target detection model determining that the third image area contains the specified object, to ensure that the determined third image area matching the second image area contains the specified object, thereby improving the accuracy of detection. Exemplarily, the third image area that matches the second image area in the video frame to be detected can be identified by a recognition model that does not require resizing of the third image area that matches the second image area in the video frame to be detected. That is, the recognition model can identify the third image area that matches the second image area with a higher resolution. Since the resolution of the image area that matches the second image area is higher, the image contains richer information. Therefore, it is possible to accurately identify whether a specified object exists in the third image area that matches the second image area.
[0151] Specifically, the third image area matching the second image area is input into a pre-trained recognition model, and the recognition model outputs a recognition result indicating whether the designated object exists in the third image area matching the second image area; wherein the recognition model is trained based on the fourth sample image and the sample label indicating whether the designated object exists in the fourth sample image. It can be understood that the recognition model can output the result of whether the designated object exists in the input image, and when training the recognition model, the fourth sample image with the designated object and the sample label indicating that the designated object exists in the fourth sample image can be used for training; the fourth sample image with the designated object can also be used as a positive sample, and the fourth sample image without the designated object can be used as a negative sample, and the sample label of the negative sample indicates that the designated object does not exist in the fourth sample image, and the positive sample and the negative sample can be used for training.
[0152] S1052: Obtain a detection result of a current video frame to be detected according to the first detection result and the second detection result of the first image area, and the alternative detection result.
[0153] It can be understood that in the process of determining the detection result of the current video frame to be detected, the first detection result and the second detection result of the first image area, as well as the alternative detection result, can be used to determine whether there is a person using a specified item in the current video frame to be detected based on multiple detection results.
[0154] In an embodiment of the present invention, the detection result of the current video frame to be detected can be accurately determined by analyzing multiple results using the first detection result and the second detection result of the first image area, the third detection result of the current video frame to be detected, and the fourth detection result of the current video frame to be detected.
[0155] Optionally, in one embodiment, as Figure 6 As shown, in Figure 5 Based on the behavior recognition method shown, step S1052 includes steps S10521-S10523.
[0156] S10521, when the first detection result of the first image area indicates that a person in the first image area is holding a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, and the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated item in the second image area, determine the detection result indicating that there is an action of a person using the designated item in the current video frame to be detected.
[0157] It can be understood that when the detection result of the current video frame to be detected is determined based on the first detection result, the second detection result and the third detection result, the first detection result indicates that the person in the first image area is holding a designated item in his hand, the second detection result indicates that the person in the first image area is in a posture of using the designated item, and the third detection result indicates that the gesture in the second image area that matches the second image area is a posture of using the designated item. In this case, it can be determined that there is an action of a person using the designated item in the current video frame to be detected.
[0158] S10522, when the first detection result of the first image area indicates that a person in the first image area is holding a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, and the fourth detection result of the current video frame to be detected indicates that there is a designated item in the third image area, determine the detection result indicating that there is an action of a person using the designated item in the current video frame to be detected.
[0159] It can be understood that when the detection result of the current video frame to be detected is determined based on the first detection result, the second detection result and the fourth detection result, the first detection result indicates that the person in the first image area is holding a designated item in his hand, the second detection result indicates that the person in the first image area is in a posture of using the designated item, and the fourth detection result indicates that the designated item does exist in the third image area matching the second image area. Then, it can be determined that there is behavior of a person using the designated item in the current video frame to be detected.
[0160] S10523, when the first detection result of the first image area indicates that a person in the first image area holds a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated item in the second image area, and the fourth detection result of the current video frame to be detected indicates that there is a designated item in the third image area, determine the detection result indicating that there is an act of a person using the designated item in the current video frame to be detected.
[0161] It can be understood that when the detection result of the current video frame to be detected is determined based on the first detection result, the second detection result, the third detection result and the fourth detection result, the first detection result indicates that the person in the first image area is holding a designated item in his hand, the second detection result indicates that the person in the first image area is in a posture of using the designated item, the third detection result indicates that the gesture in the second image area matching the first image area is a posture of using the designated item, and the fourth detection result indicates that the designated item does exist in the third image area matching the second image area. Then, it can be determined that there is behavior of a person using the designated item in the current video frame to be detected.
[0162] If the first detection result of the first image area indicates that the person in the first image area does not hold the designated item in his hands, the second detection result of the first image area indicates that there is no gesture of using the designated item in the first image area, the third detection result of the current video frame to be detected indicates that there is no gesture of using the designated item in the second image area, or the fourth detection result of the current video frame to be detected indicates that there is no designated item in the third image area, then it can be determined that there is no behavior of a person using the designated item in the current video frame to be detected.
[0163] In the embodiment of the present invention, it is determined whether there is a person using a designated object in the current video frame to be detected based on a variety of detection results, which can improve the accuracy of identifying the presence of a person using a designated object.
[0164] Optionally, in one embodiment, as Figure 7 As shown, in Figure 1 Based on the behavior recognition method shown, the behavior recognition method further includes steps S701-S703.
[0165] S701, after obtaining the detection result of the current video frame to be detected, determining the next video frame of the current video frame to be detected from the video image containing the target scene as the current video frame to be detected.
[0166] It is understandable that the video image containing the target scene may have multiple frames. After obtaining the detection result for the current video frame to be detected, another video frame may be obtained according to the shooting order of the video image as the current video frame to be detected. Then, the process returns to step S102. At this time, the new video frame to be detected is also the current video frame to be detected, and its detection result can also be obtained.
[0167] S702: If the detection results of a specified number of consecutive video frames to be detected indicate that a person uses a specified object, determine that a final detection result of the target scene is that a person uses a specified object.
[0168] It can be understood that if there are detection results for a specified number of consecutive video frames to be detected, it means that there is a behavior of a person using a specified item. That is to say, within a continuous specified time, there is always a behavior of a person using a specified item in the target scene. Then it can be determined that there is a behavior of a person using a specified item in the target scene.
[0169] S703, after determining that the final detection result of the target scene is that a person uses the specified object, select a video frame from the video frames whose detection results indicate that a person uses the specified object, and display it as an alarm image.
[0170] It is understandable that there are behaviors of people using designated items in the target scene. In order to reduce the behaviors of using designated items in the target scene, a video frame can be selected from the video frames whose detection results indicate that there are behaviors of people using designated items, and displayed as an alarm image. Exemplarily, the person using the designated item in the video frame can be circled and then displayed. Optionally, the electronic device can be a surveillance video playback terminal, and the electronic device can display the alarm image to the administrator who is viewing the surveillance video. After seeing the alarm image, the administrator will remind the person using the designated item in the target scene through a sound player.
[0171] In the embodiment of the present invention, an alarm can be issued when there is a continuous occurrence of people using designated items in the target scene, thereby reducing the occurrence of people using designated items in the target scene and improving the work efficiency of people in the target scene.
[0172] In order to more clearly understand the embodiments of the present invention, the model structure of the target detection model involved in the aforementioned embodiments will be introduced below.
[0173] The target detection model includes: the first input layer, the backbone network, the feature fusion network and the prediction network.
[0174] like Figure 8 As shown, the first input layer is used to adjust the input image to the image input size of the target detection model. The first input layer can also be called the input layer.
[0175] The backbone network is used to slice the image obtained by the first input layer to obtain the feature map after slicing; and to extract features from the feature map after slicing. The backbone network includes: image slicing (Focus) module, two-dimensional convolution (Conv2D) layer, batch normalization (Batch Normalization, BN) layer, SiLU activation function layer, crossstage layer (CSPLayer) module, concatenation (Concate) layer and spatial pyramid pooling bottleneck structure (SpatialPyramid Pooling Bottleneck, SPPBottleneck) module. The image slicing module is used to downsample the initial information of the image and then upsample it, so that the model pays more attention to the features of small targets; the two-dimensional convolution layer is used to extract features from the feature map; the BN layer is used to normalize the values of the feature map; the SiLU activation function layer is used to increase the nonlinearity of the network; the two-dimensional convolution layer, the BN normalization layer and the SiLU activation function layer belong to the Conv2D_BN_SiLU convolution normalization activation module. The skip layer module is a composite residual structure for feature extraction; the spatial pyramid pooling bottleneck structure module can extract features through maximum pooling with different pooling kernel sizes to increase the receptive field of the network. Among them, the skip layer module includes 5 convolution normalization activation modules and a splicing layer for splicing on the number of channels; the spatial pyramid pooling bottleneck structure module, two convolution normalization activation modules, and two MaxPool2d k5 maximum pooling kernels of size 5 and one MaxPool2d k13 maximum pooling kernel of size 13, among which the maximum pooling layer can perform downsampling processing.
[0176] The feature fusion network is used to splice the feature maps extracted by the backbone network; the spliced feature maps are sampled to obtain feature maps of multiple levels, and feature fusion is performed on feature maps of different levels to obtain feature maps of multiple dimensions. The feature fusion network includes: convolution normalization activation module, Upsampling layer, Downsample layer, two-dimensional convolution layer, skip layer module and splicing layer. The Upsampling layer is used to upsample the feature layer; the Downsample layer is used to downsample the feature layer.
[0177] The prediction network is used to predict based on the feature map of each dimension obtained by the feature fusion layer, obtain the detection result under the dimension, and obtain the detection content representing the position of the target object in the input image based on the detection results under each dimension. The prediction network includes three head subnetworks, wherein the subnetwork includes: 5 convolutional normalization activation modules, each subnetwork can output a category (Class, Cls), a regression box (Regression, Reg) and a confidence (Object, Obj).
[0178] In order to more clearly understand the embodiments of the present invention, the model structure of the first posture detection model involved in the aforementioned embodiments will be introduced below.
[0179] The first posture monitoring model may include: a second input layer, a feature extraction network, a pooling network, and an output layer.
[0180] like Fig. 9 As shown, the second input layer is used to receive the input image, wherein the second input layer can also be called the input layer.
[0181] The feature extraction network is used to extract the features of the image received by the second input layer to obtain a feature map. The feature extraction network includes: a convolutional normalization activation module, a residual (Residual Block, ResBlock) module, a splicing layer and an average pooling (AvgPool) layer. Among them, the average pooling layer is used for downsampling processing. The ResBlock module is used for jump connection, directly adding the input of the previous module to the output of a layer in the model. Among them, the residual module includes three convolutional normalization activation modules and a splicing layer.
[0182] The output layer is used to activate the feature map after dimensionality reduction obtained by the average pooling layer to obtain the category of the input image. The output layer can be a fully connected (FC) layer.
[0183] It should be noted that the second posture detection model and the recognition model involved in the above embodiment have the same model structure as the first posture detection model.
[0184] Optionally, in one embodiment, the present invention also provides a behavior recognition method, such as Fig.10 As shown, the behavior recognition method may include steps S1001-S1007.
[0185] S1001, obtaining a video image containing a target scene as a current video frame to be detected.
[0186] S1002, performing target detection on the current video frame to be detected, and determining that the current video frame to be detected includes a first image area of a person, a second image area of a hand holding a designated object, and a third image area of the designated object.
[0187] S1003, for each first image area in the current video frame to be detected, the area overlap is calculated according to the position of the first image area, the position of the obtained second image area and the position of the third image area, to obtain a first detection result indicating whether the person in the first image area is holding a specified object in his hand.
[0188] S1004: Perform posture detection on the first image region to obtain a second detection result indicating whether there is a posture of using a designated object in the first image region.
[0189] S1005 , performing posture detection on the second image area to be used, and obtaining a third detection result indicating whether there is a posture of using the designated object in the image of the second image area to be used.
[0190] S1006: Detecting the designated object in the third image region to be used, and obtaining a fourth detection result indicating whether the designated object exists in the third image region to be used.
[0191] S1007, obtaining a detection result of a current video frame to be detected according to the first detection result and the second detection result of the first image area, and the third detection result and the fourth detection result of the current video frame to be detected.
[0192] It can be understood that the specific implementation of the above steps has been described in the above embodiments and will not be elaborated here.
[0193] Based on the above processing, the overlap of the regions is calculated according to the positions of the first image region, the second image region, and the third image region, and it can be determined whether the second image region has a matching third image region to determine whether the hand in the second image region holds a designated item; according to whether the first image region has a matching second image region, it can be determined whether the person in the first image region holds a designated item in his hand, and an accurate first detection result is obtained. In addition, since the second detection result indicates whether there is a gesture of using the designated item in the first image region, the detection result can be obtained according to the first detection result and the second detection result, and the two aspects of whether the person holds the designated item in his hand and whether the person's gesture is a gesture of using the designated item can be combined for analysis. If the person holds the designated item in his hand and the person's gesture is a gesture of using the designated item, it can be accurately determined that there is a detection result of the person's behavior of using the designated item in the current video frame to be detected, and the detection result of the current video frame to be detected can be accurately obtained to improve the accuracy of identifying the person's behavior of using the designated item.
[0194] Optionally, in one embodiment, the present invention further provides a behavior recognition device, such as Fig.11 As shown, the behavior recognition device includes:
[0195] A first acquisition module 1110 is used to acquire a video image containing a target scene as a current video frame to be detected;
[0196] The first determination module 1120 is used to perform target detection on the current video frame to be detected, and determine a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected;
[0197] The second determination module 1130 is used to calculate the overlap of the regions according to the obtained positions of the first image region, the second image region and the third image region, and obtain a first detection result indicating whether a person in each first image region holds a designated object;
[0198] The third determination module 1140 is used to perform posture detection on each first image region in the current video frame to be detected, and obtain a second detection result indicating whether there is a posture of using a specified object in the first image region;
[0199] The fourth determination module 1150 is used to obtain the detection result of the current video frame to be detected based on the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is any behavior of a person using a specified object in the current video frame to be detected.
[0200] Optionally, the second determining module includes:
[0201] A first calculation unit is used to calculate, for each first image region in the current video frame to be detected, a first degree of overlap between the first image region and each second image region according to a position of the first image region and a position of each second image region;
[0202] A first determining unit is used to determine, in the obtained second image region, a second image region having a first degree of coincidence greater than a first threshold value as a second image region matching the first image region;
[0203] a second calculation unit, configured to calculate a second degree of overlap between the second image region matching the first image region and each third image region according to a position of the second image region matching the first image region and a position of each third image region;
[0204] The second determination unit is used to determine, in the obtained third image area, a third image area with a second degree of overlap greater than a second threshold as a third image area matching the second image area, and determine that the first detection result of the first image area is that the person in the first image area holds a designated item in his hand.
[0205] Optionally, the second determining module includes:
[0206] A third calculation unit is used to calculate, for each third image region in the current video frame to be detected, a third degree of overlap between the third image region and each second image region according to a position of the third image region and a position of each second image region;
[0207] A third determining unit is used to determine, in the obtained second image region, a second image region having a third degree of coincidence greater than a second threshold value as a second image region matching the third image region;
[0208] a fourth calculation unit, configured to calculate a fourth degree of overlap between the second image region matching the third image region and each of the first image regions according to a position of the second image region matching the third image region and a position of each of the first image regions;
[0209] The fourth determination unit is used to determine, for a first image area in the obtained first image area where the fourth degree of overlap is greater than the first threshold, a first detection result of the first image area as: the person in the first image area is holding a designated item in his hands; and for a first image area in the obtained first image area where the fourth degree of overlap is not greater than the first threshold, determine the first detection result of the first image area as: the person in the first image area is not holding the designated item in his hands.
[0210] Optionally, a fourth determination module is specifically used to determine a detection result indicating that there is an action of a person using a specified object in a current video frame to be detected, when a first detection result in the first image area indicates that a person in the first image area is holding a specified object in his hand and a second detection result in the first image area indicates that there is a gesture of using the specified object in the first image area.
[0211] Optionally, the behavior recognition device further includes:
[0212] A second acquisition module is used to obtain alternative detection results of the current video frame to be detected; wherein the alternative detection results of the current video frame to be detected include: a third detection result of the current video frame to be detected, and / or a fourth detection result of the current video frame to be detected; the third detection result of the current video frame to be detected is: a posture detection is performed on the second image area matching the first image area, and the obtained detection result indicates whether there is a posture of using a specified object in the image of the second image area matching the first image area; the fourth detection result of the current video frame to be detected is: a detection of a specified object is performed on the third image area matching the second image area, and the obtained detection result indicates whether there is a specified object in the third image area matching the second image area;
[0213] The fourth determination module includes: a second determination unit, which is used to obtain the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area and the alternative detection result.
[0214] The second determination unit is specifically configured to determine a detection result indicating that a person in the current video frame to be detected has an action of using a designated object when the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that a person in the first image region has a gesture of using the designated object, and the third detection result of the current video frame to be detected indicates that a person in the second image region has a gesture of using the designated object, and the detection result of the current video frame to be detected indicates that a person in the current video frame to be detected has an action of using the designated object;
[0215] When the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that there is a gesture of using the designated object in the first image region, and the fourth detection result of the current video frame to be detected indicates that there is a designated object in the third image region, determining a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected;
[0216] When the first detection result of the first image area indicates that a person in the first image area is holding a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated item in the second image area, and the fourth detection result of the current video frame to be detected indicates that there is a designated item in the third image area, a detection result indicating that there is an act of a person using the designated item in the current video frame to be detected is determined.
[0217] Optionally, the behavior recognition device further includes:
[0218] A fifth determination module, configured to determine, after obtaining the detection result of the current video frame to be detected, the next video frame of the current video frame to be detected from the video image containing the target scene as the current video frame to be detected;
[0219] an execution module, configured to return to the step of performing target detection on the current video frame to be detected, and determining that the first image region of the current video frame to be detected contains a person, the second image region contains a hand holding a designated object, and the third image region contains the designated object;
[0220] The sixth determination module is used to determine that the final detection result of the target scene is the existence of a person using the specified object if the detection results of a specified number of consecutive video frames to be detected indicate the existence of a person using the specified object.
[0221] Optionally, the behavior recognition device further includes:
[0222] The alarm module is used to select a video frame from the video frames whose detection results indicate that a person uses a specified object after determining that the final detection result of the target scene is that a person uses the specified object, and display it as an alarm image.
[0223] Based on the above processing, the overlap of the regions is calculated according to the positions of the first image region, the second image region, and the third image region, and it can be determined whether the second image region has a matching third image region to determine whether the hand in the second image region holds a designated item; according to whether the first image region has a matching second image region, it can be determined whether the person in the first image region holds a designated item in his hand, and an accurate first detection result is obtained. In addition, since the second detection result indicates whether there is a gesture of using the designated item in the first image region, the detection result can be obtained according to the first detection result and the second detection result, and the two aspects of whether the person holds the designated item in his hand and whether the person's gesture is a gesture of using the designated item can be combined for analysis. If the person holds the designated item in his hand and the person's gesture is a gesture of using the designated item, it can be accurately determined that there is a detection result of the person's behavior of using the designated item in the current video frame to be detected, and the detection result of the current video frame to be detected can be accurately obtained to improve the accuracy of identifying the person's behavior of using the designated item.
[0224] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 1201 , a communication interface 1202 , a memory 1203 and a communication bus 1204 , wherein the processor 1201 , the communication interface 1202 , and the memory 1203 communicate with each other via the communication bus 1204 .
[0225] Memory 1203, used for storing computer programs;
[0226] The processor 1201 is used to implement the above-mentioned behavior recognition method when executing the program stored in the memory 1203.
[0227] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0228] The communication interface is used for communication between the above electronic device and other devices.
[0229] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0230] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0231] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned behavior recognition methods are implemented.
[0232] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any behavior recognition method in the above embodiments.
[0233] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0234] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0235] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0236] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A behavior recognition method, It is characterized in that The method comprises: Obtain a video image containing the target scene as the current video frame to be detected; Performing target detection on the current video frame to be detected, determining a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected; Calculate the overlap of the regions according to the obtained positions of the first image region, the second image region, and the third image region, and obtain a first detection result indicating whether a person in each first image region holds a designated object; For each first image region in the current video frame to be detected, performing posture detection on the first image region to obtain a second detection result indicating whether there is a posture of using a designated object in the first image region; The detection result of the current video frame to be detected is obtained according to the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is a behavior of a person using a specified object in the current video frame to be detected.
2. The method according to claim 1, It is characterized in that The calculation of the overlap of the regions is performed based on the obtained positions of the first image region, the second image region, and the third image region to obtain a first detection result indicating whether a person in each first image region holds a designated object, including: For each first image region in the current video frame to be detected, according to the position of the first image region and the position of each second image region, calculate a first degree of overlap between the first image region and each second image region; In the obtained second image region, determining a second image region whose first coincidence degree is greater than a first threshold value as a second image region matching the first image region; Calculating a second degree of overlap between the second image region matching the first image region and each third image region according to the position of the second image region matching the first image region and the position of each third image region; In the obtained third image area, a third image area with a second degree of overlap greater than a second threshold is determined as the third image area matching the second image area, and the first detection result of the first image area is determined to be that the person in the first image area holds a designated item in his hand.
3. The method according to claim 1, It is characterized in that The calculation of the degree of overlap of the regions is performed based on the obtained positions of the first image region, the second image region, and the third image region to obtain a first detection result indicating whether a person in each first image region holds a designated object, including: For each third image region in the current video frame to be detected, according to the position of the third image region and the position of each second image region, calculate a third degree of overlap between the third image region and each second image region; In the obtained second image region, determining a second image region whose third coincidence degree is greater than a second threshold value as a second image region matching the third image region; Calculating a fourth degree of overlap between the second image region matching the third image region and each first image region according to the position of the second image region matching the third image region and the position of each first image region; For the first image area in which the fourth degree of overlap is greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is holding a designated item in his hands; for the first image area in which the fourth degree of overlap is not greater than the first threshold value, the first detection result of the first image area is determined as: the person in the first image area is not holding the designated item in his hands.
4. The method according to claim 1, It is characterized in that The step of obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area includes: When the first detection result of the first image area indicates that a person in the first image area is holding a designated object in his hand, and the second detection result of the first image area indicates that there is a gesture of using the designated object in the first image area, a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected is determined.
5. The method according to claim 2, It is characterized in that Before obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area, the method further includes: Obtain alternative detection results of the current video frame to be detected; wherein the alternative detection results of the current video frame to be detected include: a third detection result of the current video frame to be detected, and / or a fourth detection result of the current video frame to be detected; the third detection result of the current video frame to be detected is: performing posture detection on the second image area matching the first image area, and obtaining a detection result indicating whether a posture of using a specified object exists in the image of the second image area matching the first image area; the fourth detection result of the current video frame to be detected is: performing designated object detection on the third image area matching the second image area, and obtaining a detection result indicating whether a designated object exists in the third image area matching the second image area; The step of obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area includes: The detection result of the current video frame to be detected is obtained according to the first detection result and the second detection result of the first image area and the alternative detection result.
6. The method according to claim 5, It is characterized in that The step of obtaining the detection result of the current video frame to be detected according to the first detection result and the second detection result of the first image area and the alternative detection result includes: When the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that there is a gesture of using the designated object in the first image region, and the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated object in the second image region, determining a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected; When the first detection result of the first image region indicates that a person in the first image region holds a designated object in his hand, the second detection result of the first image region indicates that there is a gesture of using the designated object in the first image region, and the fourth detection result of the current video frame to be detected indicates that there is a designated object in the third image region, determining a detection result indicating that there is an action of a person using the designated object in the current video frame to be detected; When the first detection result of the first image area indicates that a person in the first image area is holding a designated item in his hand, the second detection result of the first image area indicates that there is a gesture of using the designated item in the first image area, the third detection result of the current video frame to be detected indicates that there is a gesture of using the designated item in the second image area, and the fourth detection result of the current video frame to be detected indicates that there is a designated item in the third image area, a detection result indicating that there is an act of a person using the designated item in the current video frame to be detected is determined.
7. The method according to claim 1, It is characterized in that The method further comprises: After obtaining the detection result of the current video frame to be detected, determining the next video frame of the current video frame to be detected from the video image containing the target scene as the current video frame to be detected; and returning to the step of performing target detection on the current video frame to be detected, and determining that the current video frame to be detected contains a first image area of a person, a second image area of a hand holding a designated object, and a third image area of the designated object; If the detection results of a specified number of consecutive video frames to be detected indicate that there is a behavior of a person using a specified object, it is determined that the final detection result of the target scene is that there is a behavior of a person using a specified object.
8. The method according to claim 7, It is characterized in that The method further comprises: After determining that the final detection result of the target scene is that there is a behavior of a person using a designated object, a video frame is selected from the video frames whose detection results indicate that there is a behavior of a person using the designated object, and is displayed as an alarm image.
9. A behavior recognition device, It is characterized in that The device comprises: A first acquisition module is used to acquire a video image containing a target scene as a current video frame to be detected; A first determination module is used to perform target detection on the current video frame to be detected, and determine a first image region containing a person, a second image region containing a hand holding a designated object, and a third image region containing the designated object in the current video frame to be detected; A second determination module is used to calculate the overlap of the regions according to the obtained positions of the first image region, the second image region and the third image region, and obtain a first detection result indicating whether a person in each first image region holds a designated object; A third determination module is used to perform posture detection on each first image area in the current video frame to be detected, and obtain a second detection result indicating whether there is a posture of using a specified object in the first image area; The fourth determination module is used to obtain the detection result of the current video frame to be detected based on the first detection result and the second detection result of the first image area; wherein the detection result of the current video frame to be detected indicates whether there is any behavior of a person using a specified object in the current video frame to be detected.
10. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, It is characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.
