A method, device and computing equipment for identifying mobile phone playing behavior in office area
By only detecting key points on the upper body of the human body in the mobile phone behavior recognition in the office area and combining with the telephone behavior screening, the problems of low recognition accuracy and high false alarm rate in the existing technology are solved, and efficient mobile phone behavior recognition and computing resources are achieved optimization.
Patent Information
- Application Number
- CN202110336542.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-03-29
AI Technical Summary
The prior art is difficult to adapt to specific scenarios in the recognition of mobile phone behavior in office areas, especially in the top view angle and sitting posture, which leads to low pose estimation accuracy, and is difficult to distinguish between making a phone behavior and playing a mobile phone behavior, resulting in high false alarm rate. Too large resources.
Only the key points of the head, shoulder, elbow and hand of the human upper body are detected to generate candidate areas for mobile phone behavior recognition, and use mobile phone classification recognition model for identification, combined with telephone behavior screening, reduce false alarm rate and reduce calculation cost.
It improves the accuracy and robustness of mobile phone behavior recognition, reduces the false positive rate, reduces the complexity of the model and the use of computing resources, and adapts to specific scenarios.
Smart Images

Figure CN112966655B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and is a method, a device and a computer device for identifying mobile phone playing behavior in an office area. Background Art
[0002] In certain scenarios, office workers need to stay focused at work at all times. In order to ensure and improve office efficiency and quality, it is necessary to monitor the behavior of office workers and identify abnormal behaviors, including monitoring abnormal behaviors such as playing with mobile phones during office hours. Monitoring abnormal behaviors of playing with mobile phones often requires long-term monitoring videos from multiple surveillance cameras in the office area, which is time-consuming, labor-intensive and inefficient to observe manually.
[0003] At present, the methods for identifying mobile phone playing behaviors based on video images mainly use deep learning methods to achieve human posture estimation and mobile phone detection. Some methods mainly use video image data to achieve mobile phone playing behaviors through deep learning-based human detection, mobile phone detection and human posture estimation technology, which can often achieve high-precision recognition results. However, due to different shooting angles in different scenes, different performance requirements of users, and different technical difficulties and challenges, it is difficult to simply migrate the above methods to this scenario.
[0004] In certain scenes, the shooting angle is a bird's-eye view, and most of the time, office workers are sitting or standing, which often results in the lower body being largely blocked, that is, only the upper body is exposed when working, or some key points are lost due to other blocking reasons, which makes it difficult to use human body key points for posture estimation. As a result, the existing posture estimation method of mobile phone public patents is difficult to achieve high accuracy.
[0005] Since both phone calls and playing with the phone are caused by the phone being near the hand, the two behaviors are quite similar. How to avoid the false positive interference of phone calls on playing with the phone is an important condition for reducing the false positive rate of phone playing identification. Existing methods only judge whether it is playing with the phone based on whether the phone is near the hand, which makes it difficult for the system to distinguish between phone calls and playing with the phone, resulting in a high risk of false positives.
[0006] In actual applications, it is often necessary to monitor multiple rooms at the same time, which requires additional consideration of recognition efficiency and hardware resource usage. Existing methods usually use object detection algorithms in the area near the hand to detect mobile phones, but the detection algorithm itself has high resource usage and large computational cost, which undoubtedly puts a greater pressure on limited computing resources. Summary of the invention
[0007] In order to overcome the problems existing in the related art, the present invention provides a method, device and computer equipment for identifying mobile phone playing behavior in an office area, which can only detect the key points of the head, shoulders, elbows and hands of the upper body of the human body. It can not only better adapt to the mobile phone playing recognition task requirements in specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model.
[0008] According to a first aspect of an embodiment of the present invention, a method for identifying mobile phone playing behavior in an office area is provided, comprising inputting a surveillance video to be identified and extracting key frames;
[0009] Detecting key points of the upper body of a human body sub-image corresponding to the key frame;
[0010] Using the key point information of the upper body, candidate regions for identifying mobile phone playing behaviors are generated;
[0011] Using the mobile phone playing classification and recognition model to identify the mobile phone playing behavior recognition candidate area, and obtain the classification confidence of the mobile phone playing behavior recognition candidate area;
[0012] It is determined whether there is any mobile phone playing behavior in the surveillance video corresponding to the mobile phone playing behavior recognition candidate area according to the classification confidence.
[0013] According to a second aspect of an embodiment of the present invention, there is provided a device for identifying mobile phone playing behavior in an office area, comprising a key frame acquisition module, a key point detection module, a candidate area generation module, a behavior recognition module and a behavior determination module;
[0014] The key frame acquisition module is used to input the surveillance video to be identified and extract the key frame;
[0015] A key point detection module, used to detect key points of the upper body of a human body sub-image corresponding to the key frame;
[0016] A candidate region generation module is used to generate candidate regions for mobile phone playing behavior recognition using key point information of the upper body;
[0017] A behavior recognition module, used to use a mobile phone playing classification recognition model to recognize the mobile phone playing behavior recognition candidate area, and obtain a classification confidence of the mobile phone playing behavior recognition candidate area;
[0018] The behavior determination module is used to determine whether there is any mobile phone playing behavior in the surveillance video corresponding to the mobile phone playing behavior identification candidate area according to the classification confidence.
[0019] The technical solution provided by the embodiments of the present invention may include the following beneficial effects: only the key points of the head, shoulders, elbows and hands of the upper body of the human body can be detected, which can not only better adapt to the requirements of mobile phone playing recognition tasks in specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model.
[0020] The embodiments of the present invention can also use the key points of the upper body to add a new phone call behavior screening to reduce the false positives of the phone playing behavior due to the phone call behavior, effectively reducing the false positive rate of the system. At the same time, it still has strong robustness when some key points of the human body are missing.
[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.
[0023] Figure 1 is a flow chart of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0024] Figure 2 is another flow chart of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0025] Figure 3 is a flowchart of identifying candidate areas for mobile phone playing behavior in a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0026] FIG4 is an example diagram of candidate area generation of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0027] Figure 5 It is a flow chart of generating a classification and recognition model for playing with mobile phones in a method for identifying mobile phone playing behaviors in an office area according to an exemplary embodiment of the present invention;
[0028] Figure 6 It is a schematic diagram of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0029] Figure 7 is a schematic block diagram of a candidate area generation module of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0030] Figure 8It is a schematic block diagram of a mobile phone playing classification recognition model generation module of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention;
[0031] Fig. 9 The diagram is a schematic diagram showing the structure of a computing device according to an exemplary embodiment. DETAILED DESCRIPTION
[0032] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.
[0033] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0034] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0035] The present invention provides a method, device and computer equipment for identifying mobile phone playing behavior in an office area, which can only detect the key points of the head, shoulders, elbows and hands of the upper body of the human body. It can not only better adapt to the mobile phone playing recognition task requirements of specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model.
[0036] Figure 1 It is a flow chart of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0037] Reference Figure 1 , including the following steps:
[0038] In step 101, a surveillance video to be identified is input and key frames are extracted.
[0039] After completing step 101, in step 102, key points of the upper body of the human body sub-image corresponding to the key frame are detected.
[0040] Afterwards, in step 103, the key point information of the upper body is used to generate a candidate region for identifying the mobile phone playing behavior.
[0041] In step 104, the mobile phone playing behavior classification and recognition model is used to identify the mobile phone playing behavior recognition candidate area to obtain the classification confidence of the mobile phone playing behavior recognition candidate area.
[0042] In step 105, it is determined whether there is any mobile phone playing behavior in the surveillance video corresponding to the mobile phone playing behavior identification candidate area according to the classification confidence.
[0043] As can be seen from this embodiment, the present invention inputs the surveillance video to be identified, extracts the key frames; detects the key points of the upper body of the human body sub-image corresponding to the key frames; generates a candidate area for mobile phone playing behavior identification using the upper body key point information; uses the mobile phone playing classification and recognition model to identify the candidate area for mobile phone playing behavior identification, and obtains the classification confidence of the candidate area for mobile phone playing behavior identification; and determines whether there is a mobile phone playing behavior in the surveillance video corresponding to the candidate area for mobile phone playing behavior identification based on the classification confidence. This embodiment can only detect the key points of the head, shoulders, elbows and hands of the upper body of the human body, which can not only better adapt to the mobile phone playing identification task requirements of specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model. At the same time, it can also avoid dependence on the key points of the lower body of the human body.
[0044] Figure 2 It is another flow chart of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0045] Reference Figure 2 , including the following steps:
[0046] In step 201, a surveillance video to be identified is input and key frames are extracted.
[0047] Then, in step 202, a human body detection algorithm is used to perform human body detection on the key frame;
[0048] Furthermore, the process proceeds to step 203, where a sub-image containing a human body is cut out from the key frame using the human body detection result.
[0049] Then, the process proceeds to step 204 to detect key points of the upper body of the human body sub-image.
[0050] Then, step 205 is performed to determine whether the action is to make a phone call based on the key point information of the upper body. If yes, the process proceeds to step 211 to output the current frame as a non-mobile phone playing action frame.
[0051] In a preferred embodiment, the step of determining whether the action is to make a phone call can be implemented by the following method:
[0052] First, the key points of the hand and elbow in the key point information of the upper body determine whether the hand is higher than the elbow. If the hand is higher than the elbow, then determine whether the angle between the key points of the hand, elbow and shoulder forms a smaller angle. In a preferred embodiment, the angle can be determined by the behavioral statistics of the scene. The smaller angle here can be 30°, for example. When the angle between the key points of the hand, elbow and shoulder forms a smaller angle, determine whether the distance between the hand and the head is smaller than the distance between the head and the elbow. When the angle is smaller and the distance between the hand and the head is smaller than the distance between the head and the elbow, it is considered that the human body in this frame is making a phone call.
[0053] In this embodiment, the key points of the upper body are used to add a phone call behavior filter to reduce the false positives of phone playing behaviors and effectively reduce the false positive rate of the system.
[0054] If the result of step 205 is negative, the process proceeds to step 206 to generate a candidate region for identifying the mobile phone playing behavior using the key point information of the upper body.
[0055] Figure 3 It is a flowchart of identifying candidate areas for mobile phone playing behavior in a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0056] Reference Figure 3 , including the following steps:
[0057] Step 301, obtaining the human body detection result and the upper body key point detection result.
[0058] Step 302: identifying key points of both elbows and both hands according to the human body detection result and the upper body key point detection result.
[0059] Step 303, using the key point information of the elbows and hands of the upper body, a candidate region for identifying the mobile phone playing behavior is generated.
[0060] Figure 4a-4c This is an example diagram of candidate area generation of a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0061] In a preferred embodiment, the candidate region for mobile phone playing behavior recognition is generated in the following manner: When only one hand and the key points of the corresponding elbow are recognized, a first candidate region or a second candidate region is generated in a first expansion manner as the candidate region for mobile phone playing behavior recognition corresponding to the key frame.
[0062] In a preferred embodiment, the first expansion method generates a first candidate area or a second candidate area as a candidate area for mobile phone playing behavior recognition corresponding to the key frame, referring to Figure 4a , taking the left hand as an example, the left forearm, that is, the line connecting the left elbow key point 5 and the left hand key point 3 extending along the left hand direction 1 / 2 of the left forearm length is taken as the center, and a rectangular frame is made with a certain pixel value obtained by statistics as the side length, and the first candidate area is generated. The certain pixel here can be 200 pixels. Here, the first candidate area and the second candidate area are just to distinguish the candidate areas generated by the left and right hands.
[0063] When only one hand key point is identified and its corresponding elbow key point is not detected, the second expansion method is used to generate the third candidate area or the fourth candidate area as the candidate area for mobile phone playing behavior identification corresponding to the key frame. In a preferred embodiment, the second expansion method implements the candidate area generation method, referring to Figure 4b , taking the right hand key point 6 as an example, with this point as the center and a larger pixel value as the side length, a rectangle is made to generate the third candidate area, which is used as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame. The larger pixel value here can be set to 300 pixels. It is worth noting that the third candidate area or the fourth candidate area here can only be one candidate area, and two candidate areas cannot be generated at the same time. For example: if only the hand key point of the left hand is identified, and the elbow key point corresponding to the left hand is not detected, the fourth candidate area is generated by the second expansion method. Conversely, if only the hand key point of the right hand is identified, and the elbow key point corresponding to the right hand is not detected, the third candidate area is generated by the second expansion method. Here, the third candidate area or the fourth candidate area is only used to distinguish the candidate areas generated by different hands.
[0064] refer to Figure 4a , the key points of both hands and both elbows are detected, and the first expansion method is used to generate the fifth candidate area and the sixth candidate area. In an example, when the key points of both elbows and both hands of a person are detected at the same time, that is, the left elbow key point 3, the right elbow key point 4, the left hand key point 5, and the right hand key point 6, the first expansion method is first used to generate the fifth candidate area and the sixth candidate area. Here is an implementation method of using the first expansion method to generate the fifth candidate area and the sixth candidate area.
[0065] In a preferred embodiment, the IOU of the two frames of the fifth candidate area and the sixth candidate area is calculated, and it is determined whether the two IOUs are greater than a certain threshold to determine whether the two frames intersect. IOU is Intersection over Union. The IOU threshold here can be 0.1. That is, when it is determined that the two IOUs are greater than 0.1, it is considered that the fifth candidate area and the sixth candidate area have an intersection, and the seventh candidate area is generated in the third expansion method as the candidate area for mobile phone playing behavior recognition corresponding to the key frame.
[0066] In a preferred embodiment, the seventh candidate area is generated in the third expansion mode as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame, which can be referred to as Figure 4c , taking the center of the intersection of the fifth candidate area and the sixth candidate area as the center point, and taking the certain pixel value obtained by statistics as the side length to make a rectangle, then the seventh candidate area is generated. In an example, the certain pixel value here is 150 pixel value.
[0067] When it is determined that the two IOUs of the fifth candidate area and the sixth candidate area are not greater than a certain threshold, it is considered that the fifth candidate area and the sixth candidate area have no intersection, and the fifth candidate area and the sixth candidate area are used as candidate areas for mobile phone playing behavior recognition corresponding to the key frame.
[0068] This embodiment uses the upper body key points to generate the candidate area for playing with the phone, which is a fast and efficient method for searching potential areas for mobile phones. It greatly reduces the search range of the mobile phone, so that the mobile phone playing classification model can be effectively combined in the subsequent identification to realize the regional positioning of the mobile phone and the identification of the mobile phone playing behavior. This can replace the mobile phone detection algorithm, not only maintaining the high-precision mobile phone playing recognition performance, but also greatly reducing the system's computational cost and improving the recognition efficiency. In addition, the candidate area generation method of this embodiment is still highly robust when some key points of the human body are missing.
[0069] After completing step 206, in step 207, based on the candidate area for identifying the mobile phone playing behavior, the original image corresponding to the candidate area for identifying the mobile phone playing behavior is cropped to obtain a candidate area sub-image.
[0070] Afterwards, in step 208, the cropped candidate region sub-image is subjected to mobile phone playing classification and recognition using the mobile phone playing classification and recognition model to obtain the classification confidence of the cropped candidate region sub-image.
[0071] Finally, step 209 is performed to determine whether a candidate sub-image corresponding to the key frame has a mobile phone playing classification confidence greater than a confidence threshold. If so, step 210 is performed to output the current frame as a mobile phone playing behavior frame. Otherwise, step 211 is performed to output the current frame as a non-mobile phone playing behavior frame. The confidence threshold here can be 0.9.
[0072] In a preferred embodiment, Figure 5 This is a flowchart of generating a classification and recognition model for mobile phone playing in a method for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention. Figure 5 , which may include:
[0073] Step 501, randomly cropping the image of the area near the human hand in the image.
[0074] Step 502, the cropped image with the mobile phone in the image is classified as a positive sample, and the cropped image without the mobile phone in the image is classified as a negative sample, so that the ratio of positive and negative samples is within a preset interval. In one example, after randomly cropping the image of the area near the human hand in the image, the image is randomly traversed to crop a sub-image of a fixed size.
[0075] The cropped images with mobile phones in the images are classified as positive samples, and the cropped images without mobile phones in the images are classified as negative samples, so that the ratio of positive and negative samples is between 1:5 and 1:10.
[0076] Step 503: divide the positive and negative samples into a training set and a test set according to a preset ratio. The preset ratio here can be 7:3, that is, the ratio of the training set to the test set is 7:3.
[0077] Step 504, using the training atlas to train the mobile phone classification model, wherein one training saves a model at a preset iteration interval. In a preferred embodiment, ResNet-18 is used to train the mobile phone classification model. The iteration interval here refers to the training cycle.
[0078] Step 505: The trained output model is tested on the test set, and the model with the highest accuracy is selected as the mobile phone playing classification and recognition model.
[0079] As can be seen from this embodiment, the present invention inputs the surveillance video to be identified, extracts the key frames; detects the key points of the upper body of the human body sub-image corresponding to the key frames; generates a candidate area for mobile phone playing behavior identification using the upper body key point information; uses the mobile phone playing classification and recognition model to identify the candidate area for mobile phone playing behavior identification, and obtains the classification confidence of the candidate area for mobile phone playing behavior identification; and determines whether there is a mobile phone playing behavior in the surveillance video corresponding to the candidate area for mobile phone playing behavior identification based on the classification confidence. This embodiment can only perform key point detection of the head, shoulders, elbows and hands of the upper body of the human body, which can not only better adapt to the mobile phone playing identification task requirements of specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model.
[0080] Corresponding to the aforementioned application function realization method embodiment, the present invention also provides a device for identifying mobile phone playing behavior in an office area and corresponding embodiments.
[0081] Figure 6 It is a schematic diagram of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0082] Reference Figure 6 , in an office area mobile phone playing behavior recognition device, it may include: a key frame acquisition module 601, a key point detection module 602, a candidate area generation module 603, a behavior recognition module 604 and a behavior determination module 605;
[0083] The key frame acquisition module 601 is used to input the surveillance video to be identified and extract the key frame;
[0084] A key point detection module 602 is used to detect key points of the upper body of the human body sub-image corresponding to the key frame;
[0085] The candidate region generation module 603 is used to generate a candidate region for identifying mobile phone playing behavior using the key point information of the upper body.
[0086] The behavior recognition module 604 is used to use the mobile phone playing classification recognition model to identify the mobile phone playing behavior recognition candidate area and obtain the classification confidence of the mobile phone playing behavior recognition candidate area;
[0087] The behavior determination module 605 is used to determine whether there is a behavior of playing with a mobile phone in the surveillance video corresponding to the candidate area for identifying the behavior of playing with a mobile phone according to the classification confidence.
[0088] In a preferred embodiment, the device further comprises: a human body detection module 606, a cutting module 607;
[0089] A human body detection module 606, configured to perform human body detection on the key frame using a human body detection algorithm;
[0090] The cropping module 607 is used to crop a sub-image containing a human body from the key frame using the human body detection result.
[0091] In a preferred embodiment, the device further comprises: a mobile phone playing classification recognition model generation module 608, which is used to generate a mobile phone playing classification recognition model.
[0092] In a preferred embodiment, Figure 7 It is a schematic block diagram of a candidate area generation module of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0093] Reference Figure 7 The candidate region generation module includes: a result acquisition unit 701, an elbow key point recognition unit 702 and a candidate region determination unit 703.
[0094] The result acquisition unit 701 is used to acquire the human body detection result and the upper body key point detection result.
[0095] The elbow key point recognition unit 702 is used to recognize the key points of both elbows and both hands according to the human body detection result and the upper body human body key point detection result.
[0096] The candidate region determination unit 703 is used to generate a candidate region for identifying mobile phone playing behavior using key point information of both elbows and both hands of the upper body.
[0097] In a preferred embodiment, when the candidate area determination unit 703 only recognizes that one hand and the key point of the corresponding elbow are recognized, a first candidate area or a second candidate area is generated in a first outward expansion manner as a candidate area for identifying mobile phone playing behavior corresponding to the key frame.
[0098] When only one hand key point is identified but its corresponding elbow key point is not detected, the second expansion method is used to generate a third candidate area or a fourth candidate area as a candidate area for identifying the mobile phone playing behavior corresponding to the key frame.
[0099] After detecting the key points of both hands and both elbows, the first expansion method is used to generate the fifth candidate area and the sixth candidate area. If it is determined that the fifth candidate area and the sixth candidate area have no intersection, the fifth candidate area and the sixth candidate area are used as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame. If it is determined that the fifth candidate area and the sixth candidate area have an intersection, the third expansion method is used to generate the seventh candidate area as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame.
[0100] In another preferred embodiment, Figure 8It is a schematic block diagram of a mobile phone playing classification recognition model generation module of a device for identifying mobile phone playing behavior in an office area according to an exemplary embodiment of the present invention.
[0101] Reference Figure 8 The mobile phone playing classification recognition model generation module includes: a clipping unit 801, a sample classification unit 802, an atlas division unit 803, an atlas training unit 804 and an atlas testing unit 805.
[0102] The cropping unit 801 is used to randomly crop the image of the area near the human hand in the image.
[0103] The sample classification unit 802 is used to classify the cropped images with mobile phones in the images as positive samples, and classify the cropped images without mobile phones in the images as negative samples, so that the ratio of positive and negative samples is within a preset interval.
[0104] The atlas division unit 803 is used to divide the positive and negative samples into a training atlas and a test atlas according to a preset ratio.
[0105] The atlas training unit 804 is used to train the mobile phone classification model using the training atlas; one training saves a model at a preset iteration interval.
[0106] The atlas testing unit 805 is used to test the model output by the training on the test set, and select the model with the highest accuracy as the mobile phone playing classification and recognition model.
[0107] As can be seen from this embodiment, the device of the present invention inputs the surveillance video to be identified, extracts the key frames; detects the key points of the upper body of the human body sub-image corresponding to the key frames; generates a candidate area for mobile phone playing behavior identification using the upper body key point information; uses the mobile phone playing classification and recognition model to identify the candidate area for mobile phone playing behavior identification, and obtains the classification confidence of the candidate area for mobile phone playing behavior identification; and determines whether there is a mobile phone playing behavior in the surveillance video corresponding to the candidate area for mobile phone playing behavior identification based on the classification confidence. This embodiment can only perform key point detection of the head, shoulders, elbows and hands of the upper body of the human body, which can not only better adapt to the mobile phone playing identification task requirements of specific scenarios, but also effectively reduce the complexity and computational cost of the mobile phone playing behavior model.
[0108] Fig. 9 It is a structural schematic diagram of a computing device according to an exemplary embodiment, and the computing device can be used to implement the above-mentioned method for identifying mobile phone playing behavior in an office area.
[0109] See also Fig. 9 , the computing device 900 includes a memory 910 and a processor 920 .
[0110] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 920 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).
[0111] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.
[0112] The memory 910 stores executable codes, and when the executable codes are processed by the processor 920 , the processor 920 can execute the above-mentioned method for identifying the behavior of using mobile phones in an office area.
[0113] The above method according to the present invention has been described in detail above with reference to the accompanying drawings.
[0114] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0115] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.
[0116] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.
[0117] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.
[0118] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0119] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for identifying mobile phone use behavior in office areas. include: Input the surveillance video to be identified and extract the key frames; Detecting key points of the upper body of a human body sub-image corresponding to the key frame; Using the key point information of the upper body, candidate regions for identifying mobile phone playing behaviors are generated; Using the mobile phone playing classification and recognition model to identify the mobile phone playing behavior recognition candidate area, and obtain the classification confidence of the mobile phone playing behavior recognition candidate area; Determining whether there is a mobile phone playing behavior in the surveillance video corresponding to the mobile phone playing behavior recognition candidate area according to the classification confidence; Based on the key point information of the upper body, determine whether the person is making a phone call; The method of using the upper body key point information to generate the candidate area for mobile phone playing behavior recognition includes: Obtaining human body detection results and upper body key point detection results; According to the human body detection result and the upper body key point detection result, identifying the key points of both elbows and both hands; The key point information of the elbows and hands of the upper body is used to generate candidate regions for mobile phone playing behavior recognition; The method of using the key point information of the elbows and hands of the upper body to generate the candidate area for mobile phone playing behavior recognition includes: When only one hand and the key points of the corresponding elbow are identified, a first candidate region or a second candidate region is generated in a first outward expansion manner as a candidate region for identifying the mobile phone playing behavior corresponding to the key frame; When only one hand key point is identified and its corresponding elbow key point is not detected, the second expansion method is used to generate a third candidate area or a fourth candidate area as a candidate area for identifying the mobile phone playing behavior corresponding to the key frame; After detecting the key points of both hands and both elbows, the first expansion method is used to generate the fifth candidate area and the sixth candidate area. If it is determined that the fifth candidate area and the sixth candidate area have no intersection, the fifth candidate area and the sixth candidate area are used as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame. If it is determined that the fifth candidate area and the sixth candidate area have an intersection, the third expansion method is used to generate the seventh candidate area as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame.
2. The method according to claim 1, further comprising: include: Performing human body detection on the key frames using a human body detection algorithm; Using the human body detection result, a sub-image containing the human body is cut out from the key frame.
3. The method according to claim 1, further comprising: According to the candidate region for identifying the mobile phone playing behavior, the original image corresponding to the candidate region for identifying the mobile phone playing behavior is cropped to obtain a sub-image of the candidate region; in, The mobile phone playing behavior identification candidate area is identified using the mobile phone playing classification recognition model to obtain the classification confidence of the mobile phone playing behavior identification candidate area, including: using the mobile phone playing classification recognition model to perform mobile phone playing classification recognition on the cropped candidate area sub-image to obtain the classification confidence of the cropped candidate area sub-image.
4. A device for identifying mobile phone use behavior in an office area. include: Key frame acquisition module, key point detection module, candidate region generation module, behavior recognition module and behavior determination module; The key frame acquisition module is used to input the surveillance video to be identified and extract the key frame; A key point detection module, used to detect key points of the upper body of a human body sub-image corresponding to the key frame; A candidate region generation module is used to generate candidate regions for mobile phone playing behavior recognition using key point information of the upper body; A behavior recognition module, used to use a mobile phone playing classification recognition model to recognize the mobile phone playing behavior recognition candidate area, and obtain a classification confidence of the mobile phone playing behavior recognition candidate area; A behavior determination module, used for judging whether there is a behavior of playing with a mobile phone in the surveillance video corresponding to the candidate area for identifying the behavior of playing with a mobile phone according to the classification confidence; The candidate region generation module includes: a result acquisition unit, an elbow key point recognition unit and a candidate region determination unit; A result acquisition unit, used to acquire a human body detection result and a detection result of key points of the upper body; An elbow key point recognition unit, used for recognizing key points of both elbows and both hands according to the human body detection result and the upper body human body key point detection result; A candidate region determination unit, configured to generate a first candidate region or a second candidate region as a candidate region for identifying mobile phone playing behavior corresponding to the key frame in a first outward expansion manner when only a key point of a hand and a corresponding elbow is identified; When only one hand key point is identified and its corresponding elbow key point is not detected, the third candidate area or the fourth candidate area is generated by using the second expansion method as the mobile phone playing behavior identification candidate area corresponding to the key frame; After detecting the key points of both hands and both elbows, the first expansion method is used to generate the fifth candidate area and the sixth candidate area. If it is determined that the fifth candidate area and the sixth candidate area have no intersection, the fifth candidate area and the sixth candidate area are used as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame. If it is determined that the fifth candidate area and the sixth candidate area have an intersection, the third expansion method is used to generate the seventh candidate area as the candidate area for identifying the mobile phone playing behavior corresponding to the key frame.
5. The device according to claim 4, further comprising: include: Human body detection module and cropping module; A human body detection module, used for performing human body detection on the key frame using a human body detection algorithm; The cropping module is used to crop a sub-image containing a human body in the key frame by using the human body detection result.
6. A computing device, include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 3.
