Shooting behavior detection method, device, equipment and storage medium

By using an object detection network to perform landmark recognition and perspective transformation on single-frame video images, the problems of misjudgment and high resource consumption in existing technologies for human behavior recognition are solved, and efficient and accurate shooting behavior detection is achieved.

CN114743264BActive Publication Date: 2024-11-15PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210371875.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-11
Publication Date
2024-11-15
Estimated Expiration
2042-04-11

AI Technical Summary

Technical Problem

Existing human behavior recognition technologies are prone to misjudgments during the recognition process and consume a lot of resources, especially since they ignore information about objects in the environment other than the human body, causing device lag.

Method used

The target detection network identifies landmarks in single-frame video images, uses the intersection-union function to detect the positional correlation of image acquisition devices, hands, side faces, and ears, and combines perspective transformation to determine the shooting behavior and generate warning information.

Benefits of technology

It improves the accuracy of human behavior recognition, reduces resource consumption, and effectively identifies shooting behavior and generates early warnings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114743264B_ABST
    Figure CN114743264B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a shooting behavior detection method, device, equipment and storage medium for improving the accuracy of human behavior recognition. The shooting behavior detection method includes: obtaining a single-frame video image of a target person through a preset monitoring device; performing landmark recognition on the single-frame video image through a target detection network; if an image acquisition device exists in the single-frame video image, performing position association detection on the image acquisition device, the hand, side face and ear of the target person according to the intersection and union ratio function; if the image acquisition device is in a candidate state, performing perspective transformation on the camera of the image acquisition device to generate a corresponding camera perspective transformation plane; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated, and the early warning message is sent to an early warning terminal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a shooting behavior detection method, device, equipment and storage medium. Background Art

[0002] With the rapid development of computer vision technology, biometric technology has expanded from face recognition to human behavior recognition. Large amounts of video and image data collected by cameras are used to judge human behavior, monitor some illegal behaviors in public places, or be used in families to prevent dangerous behaviors of the elderly or children.

[0003] At present, the mainstream behavior recognition methods are mostly based on the human body. First, human body detection is used to locate the recognition area, and then the key points of the human body are located. Based on the key points, it is determined whether the human body posture matches the predefined behavior template to complete the behavior recognition. Different from the template matching method, there is another recognition method, which is to combine the temporal features in the video sequence, perform spatiotemporal encoding on the located human body area, and directly perform behavior classification.

[0004] Although key point matching can be used to roughly determine human movements, the recognition granularity is relatively coarse. It focuses on human body information while ignoring other objects in the environment, resulting in many misjudgments during the application process. Video analysis will consume more resources, cause equipment to freeze, and affect normal use. Summary of the invention

[0005] The present invention provides a shooting behavior detection method, device, equipment and storage medium for improving the accuracy of human behavior recognition.

[0006] The first aspect of the present invention provides a shooting behavior detection method, comprising: when a target person is in a preset detection area, obtaining a single-frame video image of the target person through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; performing landmark recognition on the single-frame video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device; if the landmark recognition result is that an image acquisition device exists in the single-frame video image, then performing position association detection on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-combination function to obtain a position association Detection result; if the position association detection result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to a preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to an early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0007] In a feasible implementation manner, the method of performing landmark recognition on the single-frame video image through a target detection network to obtain a landmark recognition result includes: performing feature extraction on the single-frame video image through a backbone feature extraction network of the target detection network to generate three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer; performing feature enhancement on the three effective feature layers through an enhanced feature extraction network of the target detection network to generate corresponding three enhanced feature layers; performing landmark prediction on the three enhanced feature layers through a prediction network of the target detection network to obtain a landmark prediction probability; if the landmark prediction probability is greater than or equal to a preset image acquisition device prediction probability, determining that the landmark recognition result is that an image acquisition device exists in the single-frame video image; if the landmark prediction probability is less than the preset image acquisition device prediction probability, determining that the landmark recognition result is that no image acquisition device exists in the single-frame video image.

[0008] In a feasible implementation, the backbone feature extraction network of the target detection network performs feature extraction on the single-frame video image to generate three effective feature layers, including: focusing processing is performed on the single-frame video image through the backbone feature extraction network of the target detection network to generate a twelve-channel single-frame video image; the twelve-channel single-frame video image is subjected to feature extraction through three different convolution kernels to generate corresponding three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer.

[0009] In a feasible implementation manner, the three effective feature layers are feature enhanced by the enhanced feature extraction network of the target detection network to generate corresponding three enhanced feature layers, including: convolving and upsampling the third effective feature layer of the three effective feature layers by the enhanced feature extraction network of the target detection network, and combining and extracting features with the second effective feature layer to obtain an upsampled feature map; convolving and upsampling the upsampled feature map, and combining and extracting features with the first effective feature layer to obtain a first enhanced feature layer; convolving, downsampling and extracting features from the first enhanced feature layer to obtain a second enhanced feature layer; convolving, downsampling and extracting features from the second enhanced feature layer to obtain a third enhanced feature layer.

[0010] In a feasible implementation, the prediction network of the target detection network performs marker prediction on the three enhanced feature layers to obtain marker prediction probabilities, including: performing marker prediction on each of the three enhanced feature layers through the prediction network of the target detection network to obtain three marker prediction results corresponding to each enhanced feature layer; stacking the corresponding three marker prediction results in each enhanced feature layer to generate marker candidate results for each enhanced feature layer; and generating marker prediction probabilities based on the marker candidate results for each enhanced feature layer.

[0011] In a feasible implementation manner, if the result of the marker recognition is that an image acquisition device exists in the single-frame video image, then position association detection is performed on the image acquisition device, the hands, side face and ears of the target person according to an intersection-and-union function to obtain a position association detection result, including: if the result of the marker recognition is that an image acquisition device exists, then position frame detection is performed on the image acquisition device, the hands, side face and ears of the target person through a target detection network to generate corresponding image acquisition device position frame, hand position frame, side face position frame and ear position frame; according to the intersection-and-union function, the image acquisition device position frame and the hand position frame, a first intersection-and-union value is obtained, wherein the first intersection-and-union value is used to indicate the degree of position association between the image acquisition device and the hand of the target person; according to the intersection-and-union function, the image acquisition device position frame, the hand position frame and the side face position frame, a second intersection-and-union value is obtained, wherein the second intersection-and-union value is used to indicate the degree of position association between the image acquisition device and the hands and side face of the target person; according to the intersection-and-union function, the image acquisition device position frame, the hand position frame and the side face position frame, The position frame, the hand position frame, the side face position frame and the ear position frame are used to obtain a third intersection-and-union ratio, wherein the third intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand, side face and ear of the target person; if the first intersection-and-union ratio is greater than or equal to a threshold value, and the second intersection-and-union ratio is less than a threshold value and / or the third intersection-and-union ratio is less than a threshold value, it is determined that the image acquisition device has a positional association with the hand, and has no positional association with the side face and / or the ear, and the positional association detection result is determined to be the image acquisition device. The acquisition device is in a candidate state; if the first intersection-and-union ratio is greater than or equal to a threshold, and the second intersection-and-union ratio is greater than or equal to a threshold and / or the third intersection-and-union ratio is greater than or equal to a threshold, then it is determined that there is a position association between the image acquisition device and the hand, the side face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a voice listening state; if the first intersection-and-union ratio is less than a threshold, then it is determined that there is no position association between the image acquisition device and the hand, and the position association detection result is determined to be that the image acquisition device is in a single state.

[0012] In a feasible implementation, if the position association detection result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, including: if the position association detection result is that the image acquisition device is in a candidate state, camera detection is performed on the single-frame video image; if the camera corresponding to the image acquisition device exists in the single-frame video image, the single-frame video image is segmented to obtain a camera image of the image acquisition device; edge detection is performed on the camera image to generate a corresponding camera edge image; and perspective transformation is performed on the camera edge image to obtain a camera perspective transformation plane.

[0013] The second aspect of the present invention provides a shooting behavior detection device, including: an acquisition module, which is used to acquire a single-frame video image of the target person through a preset monitoring device when the target person is in a preset detection area, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; an identification module, which is used to perform landmark recognition on the single-frame video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device; a detection module, which is used to perform position association detection on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function if the landmark recognition result is that the image acquisition device exists in the single-frame video image, to obtain Position association detection result; a transformation module, used to perform perspective transformation on the camera of the image acquisition device if the position association detection result is that the image acquisition device is in a candidate state, and generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear; a determination module, used to determine that the image acquisition device is in a shooting state if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to a preset angle, and generate an early warning message, and send the early warning message to an early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0014] In a feasible embodiment, the recognition module includes: a feature extraction unit, which is used to extract features from the single-frame video image through the backbone feature extraction network of the target detection network to generate three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer; a feature enhancement unit, which is used to enhance features of the three effective feature layers through the enhanced feature extraction network of the target detection network to generate corresponding three enhanced feature layers; a marker prediction unit, which is used to perform marker prediction on the three enhanced feature layers through the prediction network of the target detection network to obtain a marker prediction probability; a first determination unit, which is used to determine that the marker recognition result is that there is an image acquisition device in the single-frame video image if the marker prediction probability is greater than or equal to a preset image acquisition device prediction probability; and a second determination unit, which is used to determine that the marker recognition result is that there is no image acquisition device in the single-frame video image if the marker prediction probability is less than the preset image acquisition device prediction probability.

[0015] In a feasible implementation, the feature extraction unit is specifically used to: focus the single-frame video image through the backbone feature extraction network of the target detection network to generate a twelve-channel single-frame video image; perform feature extraction on the twelve-channel single-frame video image through three different convolution kernels to generate corresponding three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer.

[0016] In a feasible implementation manner, the feature enhancement unit is specifically used to: perform convolution and up-sampling on the third effective feature layer of the three effective feature layers through the enhanced feature extraction network of the target detection network, and combine and extract features with the second effective feature layer to obtain an up-sampled feature map; perform convolution and up-sampling on the up-sampled feature map, and combine and extract features with the first effective feature layer to obtain a first enhanced feature layer; perform convolution, down-sampling and feature extraction on the first enhanced feature layer to obtain a second enhanced feature layer; perform convolution, down-sampling and feature extraction on the second enhanced feature layer to obtain a third enhanced feature layer.

[0017] In a feasible implementation, the marker prediction unit is specifically used to: perform marker prediction on each of the three enhanced feature layers through the prediction network of the target detection network to obtain three marker prediction results corresponding to each enhanced feature layer; stack the corresponding three marker prediction results in each enhanced feature layer to generate marker candidate results for each enhanced feature layer; and generate marker prediction probabilities based on the marker candidate results for each enhanced feature layer.

[0018] In a feasible implementation manner, the detection module is specifically used for: if the landmark recognition result is that there is an image acquisition device, then the position frame detection is performed on the image acquisition device, the hand, the side face and the ear of the target person through the target detection network to generate corresponding image acquisition device position frame, hand position frame, side face position frame and ear position frame; according to the intersection and union ratio function, the image acquisition device position frame and the hand position frame, a first intersection and union ratio is obtained, wherein the first intersection and union ratio is used to indicate the degree of positional association between the image acquisition device and the hand of the target person; according to the intersection and union ratio function, the image acquisition device position frame, the hand position frame and the side face position frame, a second intersection and union ratio is obtained, wherein the second intersection and union ratio is used to indicate the degree of positional association between the image acquisition device and the hand and side face of the target person; according to the intersection and union ratio function, the image acquisition device position frame, the hand position frame, the side face position frame and the ear position frame, a third intersection and union ratio is obtained, wherein , the third intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand, the side face and the ear of the target person; if the first intersection-and-union ratio is greater than or equal to a threshold value, and the second intersection-and-union ratio is less than a threshold value and / or the third intersection-and-union ratio is less than a threshold value, it is determined that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or the ear, and the positional association detection result is determined to be that the image acquisition device is in a candidate state; if the first intersection-and-union ratio is greater than or equal to a threshold value, and the second intersection-and-union ratio is greater than or equal to a threshold value and / or the third intersection-and-union ratio is greater than or equal to a threshold value, it is determined that the image acquisition device is positionally associated with the hand, the side face and / or the ear, and the positional association detection result is determined to be that the image acquisition device is in a voice listening state; if the first intersection-and-union ratio is less than a threshold value, it is determined that the image acquisition device is not positionally associated with the hand, and the positional association detection result is determined to be that the image acquisition device is in a single state.

[0019] In a feasible implementation, the transformation module is specifically used for: if the position association detection result is that the image acquisition device is in a candidate state, performing camera detection on the single-frame video image; if a camera corresponding to the image acquisition device exists in the single-frame video image, segmenting the single-frame video image to obtain a camera image of the image acquisition device; performing edge detection on the camera image to generate a corresponding camera edge image; and performing perspective transformation on the camera edge image to obtain a camera perspective transformation plane.

[0020] The third aspect of the present invention provides a shooting behavior detection device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the shooting behavior detection device executes the above-mentioned shooting behavior detection method.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned shooting behavior detection method.

[0022] In the technical solution provided by the present invention, when the target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; a marker is identified on the single-frame video image through a target detection network to obtain a marker identification result, wherein the marker is used to indicate an image acquisition device; if the marker identification result is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-combination ratio function to obtain a position association detection result. result; if the position association detection result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspectively transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior. In the embodiment of the present invention, by obtaining a single-frame video image of the target person, the landmark recognition and location information are associated in the network structure of feature extraction and multi-target detection, the landmark is used to indicate the image acquisition device, and the more similar landmark states are excluded according to the different association degrees of the landmark, and the candidate state of the landmark is obtained, and then according to the edge detection and perspective transformation principles, the plane angle between the landmark and the single-frame video image is calculated. If the plane angle is less than or equal to the preset angle, it is determined that the landmark is in a shooting state, that is, the target person has a shooting behavior, thereby improving the accuracy of human behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an embodiment of a shooting behavior detection method in an embodiment of the present invention;

[0024] Figure 2 It is a schematic diagram of another embodiment of the shooting behavior detection method in an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of an embodiment of a shooting behavior detection device in an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of another embodiment of the shooting behavior detection device in an embodiment of the present invention;

[0027] Figure 5 It is a schematic diagram of an embodiment of a shooting behavior detection device in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a shooting behavior detection method, device, equipment and storage medium for improving the accuracy of human behavior recognition.

[0029] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0031] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0032] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the shooting behavior detection method in the embodiment of the present invention includes:

[0033] 101. When the target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device;

[0034] It is understandable that the execution subject of the present invention may be a shooting behavior detection device or a terminal, which is not limited here. The embodiment of the present invention is described by taking the shooting behavior detection device as the execution subject as an example.

[0035] It should be noted that the collection of video images is carried out after the person being collected confirms the authorization. When the person being collected enters the detection area, the shooting behavior detection device performs shooting behavior detection on the person being collected.

[0036] 102. Performing landmark recognition on a single-frame video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device;

[0037] The target detection network includes a variety of target detection networks, for example, Yolo series Yolov1, Yolov2, Yolov3, Yolov4, Yolov5, Yolo X, R-CNN and SSD target detection networks. In this embodiment, Yolo X is selected as the target detection network. The target detection network Yolo X includes a backbone feature extraction network CSPDarknet, an enhanced feature extraction network FPN and a prediction network Yolo Head, wherein the backbone feature extraction network CSPDarknet is used to shoot the behavior detection device to extract features from a single-frame video image to obtain three effective feature layers; the enhanced feature extraction network FPN is used to perform feature fusion on the three effective feature layers, thereby strengthening the features; the prediction network Yolo Head is used to predict the markers in a single-frame video image.

[0038] Image acquisition devices include various image acquisition devices, such as mobile phones, tablets, cameras, camcorders and other devices that can perform image acquisition.

[0039] 103. If the result of the landmark recognition is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, the side face and the ear of the target person according to the intersection-and-union function to obtain a position association detection result;

[0040] The shooting behavior detection device generates the image acquisition device position frame, the target person's hand position frame, the side face position frame and the ear position frame through the target detection network Yolo X, and then calculates the ratio of the intersection and union between these position frames through the intersection-over-Union (IoU) function to obtain the intersection-over-union ratio, so as to judge the position correlation degree between the image acquisition device, the target person's hand, side face and ear according to the intersection-over-union ratio, and then determine the state of the image acquisition device.

[0041] For example, the image acquisition device position frame corresponding to the image acquisition device is A, and the hand position frame corresponding to the target person's hand is B. If there is an intersection between the image acquisition device position frame and the hand position frame, then the intersection-and-union ratio is obtained according to the intersection-and-union ratio function formula IOU=(A∩B) / (A∪B). If the intersection-and-union ratio is greater than or equal to the threshold, it is determined that there is a position association between the image acquisition device and the target person's hand.

[0042] 104. If the position association detection result is that the image acquisition device is in a candidate state, a perspective transformation is performed on the camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or ear;

[0043] The shooting behavior detection device can perform a perspective transform on the camera of the image acquisition device to generate a corresponding camera perspective transform plane, or perform an affine transform on the camera of the image acquisition device to generate a corresponding camera affine transform plane. Both the camera perspective transform plane and the camera affine transform plane can be used to calculate the angle between the plane of the single-frame video image. Among them, perspective transformation is the process of projecting an image from one viewing plane to another viewing plane, so perspective transformation is also called projection mapping. Perspective transformation is a spatial three-dimensional transformation. According to the coordinate variance, the three-dimensional coordinates are projected to another viewing plane. For example, a three-dimensional cube is perspective transformed to obtain a two-dimensional cube viewing plane.

[0044] 105. If the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0045] If the image acquisition device is in a shooting state, it is determined that the target person is in shooting behavior, and the shooting behavior detection device generates warning information. The warning information includes multiple warning information, such as "Please note! Someone is taking pictures secretly." Voice warning information, "Warning light flashing" light warning information, voice and light combined sound and light warning information, and other warning information.

[0046] In an embodiment of the present invention, when a target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; a marker recognition is performed on the single-frame video image through a target detection network to obtain a marker recognition result, wherein the marker is used to indicate an image acquisition device; if the marker recognition result is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; if the position association detection result is that the image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; If the measurement result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior, thereby improving the accuracy of human behavior recognition.

[0047] See also Figure 2 Another embodiment of the shooting behavior detection method in the embodiment of the present invention includes:

[0048] 201. When the target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device;

[0049] It should be noted that the collection of video images is carried out after the person being collected confirms the authorization. When the person being collected enters the detection area, the shooting behavior detection device performs shooting behavior detection on the person being collected.

[0050] The monitoring device includes various monitoring devices, such as monitoring cameras, ordinary cameras, smart cameras, etc. This embodiment selects a single frame of video image in the video stream shot by the monitoring device and detects it through the shooting behavior detection device, which can reduce the resource consumption caused by video stream analysis.

[0051] 202. Extract features from a single-frame video image through a backbone feature extraction network of a target detection network to generate three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer, and a third effective feature layer;

[0052] The target detection network includes a variety of target detection networks, for example, Yolo series Yolov1, Yolov2, Yolov3, Yolov4, Yolov5, Yolo X, R-CNN and SSD target detection networks. In this embodiment, Yolo X is selected as the target detection network. The target detection network Yolo X includes a backbone feature extraction network CSPDarknet, an enhanced feature extraction network FPN and a prediction network Yolo Head, wherein the backbone feature extraction network CSPDarknet is used to shoot the behavior detection device to extract features from a single-frame video image to obtain three effective feature layers; the enhanced feature extraction network FPN is used to perform feature fusion on the three effective feature layers, thereby strengthening the features; the prediction network Yolo Head is used to predict the markers in a single-frame video image.

[0053] Specifically, (1) the shooting behavior detection device focuses on the single-frame video image through the backbone feature extraction network of the target detection network to generate a single-frame video image of twelve channels; (2) the single-frame video image of twelve channels is subjected to feature extraction through three different convolution kernels to generate three corresponding effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer.

[0054] For example, a single-frame video image is (640, 640, 3), wherein the image size of the single-frame video image is 640*640, and the number of channels is 3. The shooting behavior detection device performs focusing processing on the single-frame video image through the backbone feature extraction network CSPDarknet of the target detection network Yolo X, that is, assigning a value to every other pixel in the single-frame video image, including 1, 2, 3, and 4, generating four independent feature layers and stacking them, converting three channels into twelve channels, and generating a twelve-channel single-frame video image (320, 320, 12); extracting features from the twelve-channel single-frame video image (320, 320, 12) through three different convolution kernels, generating corresponding three effective feature layers, and the three effective feature layers include a first effective feature layer feat1=(80, 80, 256), a second effective feature layer feat2=(40, 40, 512), and a third effective feature layer feat3=(20, 20, 1024).

[0055] 203. Perform feature enhancement on the three effective feature layers through the enhanced feature extraction network of the target detection network to generate three corresponding enhanced feature layers;

[0056] (1) The shooting behavior detection device convolves and upsamples the third effective feature layer of the three effective feature layers through the enhanced feature extraction network of the target detection network, and combines and extracts features with the second effective feature layer to obtain an upsampled feature map; (2) The upsampled feature map is convolved and upsampled, and combined and extracted features with the first effective feature layer to obtain a first enhanced feature layer; (3) The first enhanced feature layer is convolved, downsampled and feature extracted to obtain a second enhanced feature layer; (4) The second enhanced feature layer is convolved, downsampled and feature extracted to obtain a third enhanced feature layer.

[0057] For example, the shooting behavior detection device performs a 1*1 convolution adjustment channel on the third effective feature layer feat3=(20, 20, 1024) of the three effective feature layers through the enhanced feature extraction network FPN of the target detection network Yolo X to obtain P5, and P5 is upsampled UmSampling2d and combined with the second effective feature layer feat2=(40, 40, 512), and then CSPLayer is used to perform feature extraction to obtain the feature layer P5_upsample=(40, 40, 512); the feature layer P5_upsample=(40, 40, 512) is subjected to a 1*1 convolution adjustment channel to obtain P4, and P4 is upsampled UmSampling2d and combined with the first effective feature layer feat1=(80, 80, 256) are combined, and then CSPLayer is used for feature extraction to obtain the first enhanced feature layer P3_out = (80, 80, 256); the first enhanced feature layer P3_out = (80, 80, 256) is downsampled by 3*3 convolution once, stacked with P4 after downsampling, and then CSPLayer is used for feature extraction to obtain the second enhanced feature layer P4_out = (40, 40, 512); the second enhanced feature layer P4_out = (40, 40, 512) is downsampled by 3*3 convolution once, stacked with P5 after downsampling, and then CSPLayer is used for feature extraction to obtain the third enhanced feature layer P5_out = (20, 20, 1024).

[0058] 204. Performing marker prediction on the three enhanced feature layers through the prediction network of the target detection network to obtain marker prediction probabilities;

[0059] (1) The shooting behavior detection device performs marker prediction on each of the three enhanced feature layers through the prediction network of the target detection network to obtain three marker prediction results corresponding to each enhanced feature layer; (2) The corresponding three marker prediction results are stacked in each enhanced feature layer to generate marker candidate results for each enhanced feature layer; (3) Based on the marker candidate results of each enhanced feature layer, the marker prediction probability is generated.

[0060] For example, the shooting behavior detection device performs marker prediction on the third enhanced feature layer (20, 20, 1024) of the three enhanced feature layers through the prediction network Yolo Head of the target detection network Yolo X, and obtains the corresponding three marker prediction results, and the three marker prediction results include Reg(20, 20, 4), Obj(20, 20, 1) and Cls(20, 20, num_classes), wherein the 4 in Reg(20, 20, 4) can be divided into two 2s, the first 2 is the offset of the center point of the prediction box compared to the feature point, and the second 2 is the parameter of the width and height of the prediction box compared to the logarithmic exponent, and Obj(20, 20, 1) is used to indicate the prediction box of each feature point. The probability of containing objects inside, Cls(20, 20, num_classes) is used to indicate the probability that each feature point corresponds to a certain type of object. The predicted value in the last dimension num_classes represents the probability of belonging to each class. The three marker prediction results are stacked to generate the marker candidate results of the third enhanced feature layer. The steps of marker prediction for the first enhanced feature layer and the second enhanced feature layer are the same as the above steps, that is, generating the marker candidate results of the first enhanced feature layer and the marker candidate results of the second enhanced feature layer; based on the marker candidate results of each enhanced feature layer, the marker prediction probability is generated.

[0061] 205. If the marker prediction probability is greater than or equal to the preset image acquisition device prediction probability, then determining that the marker recognition result is that the image acquisition device exists in the single-frame video image;

[0062] For example, if the preset image acquisition device prediction probability is 0.7 and the marker prediction probability is 0.8, the marker prediction probability is greater than the preset image acquisition device prediction probability, that is, the marker recognition result is determined to be the presence of an image acquisition device in a single frame of video image; or if the marker prediction probability is 0.7, the marker prediction probability is equal to the preset image acquisition device prediction probability, then the marker recognition result is determined to be the presence of an image acquisition device in a single frame of video image.

[0063] 206. If the marker prediction probability is less than the preset image acquisition device prediction probability, determining that the marker recognition result is that the image acquisition device does not exist in the single-frame video image;

[0064] For example, if the preset image acquisition device prediction probability is 0.7, if the marker prediction probability is 0.5, the marker prediction probability is less than the preset image acquisition device prediction probability, and the marker recognition result is determined to be that there is no image acquisition device in the single frame video image; or if the marker prediction probability is 0.3, the marker prediction probability is less than the preset image acquisition device prediction probability, and the marker recognition result is determined to be that there is no image acquisition device in the single frame video image.

[0065] 207. If the result of the landmark recognition is that an image acquisition device exists in the single-frame video image, position association detection is performed on the image acquisition device, the hand, the side face and the ear of the target person according to the intersection-and-union function to obtain a position association detection result;

[0066] The shooting behavior detection device generates the image acquisition device position frame, the target person's hand position frame, the side face position frame and the ear position frame through the target detection network Yolo X, and then calculates the ratio of the intersection and union between these position frames through the intersection-over-Union (IoU) function to obtain the intersection-over-union ratio, so as to judge the position correlation degree between the image acquisition device, the target person's hand, side face and ear according to the intersection-over-union ratio, and then determine the state of the image acquisition device.

[0067] For example, the image acquisition device position frame corresponding to the image acquisition device is A, and the hand position frame corresponding to the target person's hand is B. If there is an intersection between the image acquisition device position frame and the hand position frame, then the intersection-and-union ratio is obtained according to the intersection-and-union ratio function formula IOU=(A∩B) / (A∪B). If the intersection-and-union ratio is greater than or equal to the threshold, it is determined that there is a position association between the image acquisition device and the target person's hand.

[0068] Specifically, (1) if the result of the landmark recognition is that there is an image acquisition device, the shooting behavior detection device performs position frame detection on the image acquisition device, the hands, side face and ears of the target person through the target detection network, and generates corresponding image acquisition device position frame, hand position frame, side face position frame and ear position frame;

[0069] (2) obtaining a first intersection-and-union ratio value according to the intersection-and-union ratio function, the image acquisition device position frame, and the hand position frame, wherein the first intersection-and-union ratio value is used to indicate the degree of position correlation between the image acquisition device and the hand of the target person;

[0070] (3) obtaining a second intersection-and-union ratio value according to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame, and the side face position frame, wherein the second intersection-and-union ratio value is used to indicate the degree of positional correlation between the image acquisition device and the hand and side face of the target person;

[0071] (4) obtaining a third intersection-and-union ratio value according to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame, the side face position frame, and the ear position frame, wherein the third intersection-and-union ratio value is used to indicate the degree of positional association between the image acquisition device and the hand, side face, and ear of the target person;

[0072] (5) If the first intersection-over-union ratio is greater than or equal to the threshold, and the second intersection-over-union ratio is less than the threshold and / or the third intersection-over-union ratio is less than the threshold, it is determined that the image acquisition device is positionally associated with the hand and is not positionally associated with the profile face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a candidate state;

[0073] (6) If the first intersection-over-union ratio is greater than or equal to the threshold, and the second intersection-over-union ratio is greater than or equal to the threshold and / or the third intersection-over-union ratio is greater than or equal to the threshold, it is determined that the image acquisition device is positionally associated with the hand, the side face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a state of listening to speech;

[0074] (7) If the first intersection-to-union ratio is less than the threshold, it is determined that there is no position association between the image acquisition device and the hand, and the position association detection result is determined to be that the image acquisition device is in a single state.

[0075] For example, if the result of the landmark recognition is that there is an image acquisition device, the shooting behavior detection device performs position frame detection on the image acquisition device, the hands, side face and ears of the target person through the target detection network Yolo X, and generates the corresponding image acquisition device position frame A (c A , x A ,y A , w A ,h A )、Hand position frame B(c B , x B ,y B , w B ,h B )、Side face position frame C(c C , x C ,y C , w C ,h C ) and ear position frame D(c D , x D ,y D , w D ,h D ), where "c" is used to indicate the marker type, "x" is the horizontal coordinate of the center point of the position box, "y" is the vertical coordinate of the center point of the position box, "w" is the width of the position box, and "h" is the height of the position box.

[0076] According to the intersection-over-union function IoU, the image acquisition device position frame A(c A, x A ,y A , w A ,h A ) and hand position frame B(c B , x B ,y B , w B ,h B ), obtaining a first intersection-and-union ratio, wherein the first intersection-and-union ratio is used to indicate the degree of correlation between the positions of the image acquisition device and the hand of the target person;

[0077] According to the intersection-over-union function IoU, the image acquisition device position frame A(c A , x A ,y A , w A ,h A )、Hand position frame B(c B , x B ,y B , w B ,h B ) and the side face position frame C(c C , x C ,y C , w C ,h C ), obtaining a second intersection-and-union ratio, wherein the second intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand and side face of the target person;

[0078] According to the intersection-over-union function IoU, the image acquisition device position frame A(c A , x A ,y A , w A ,h A )、Hand position frame B(c B , x B ,y B , w B ,h B )、Side face position frame C(c C , x C ,y C , w C ,h C ) and ear position frame D(c D , x D ,y D , w D ,h D ), obtaining a third intersection-and-union ratio, wherein the third intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand, side face, and ear of the target person;

[0079] If the first intersection-in-combination ratio is 0.8, the second intersection-in-combination ratio is 0.6, the third intersection-in-combination ratio is 0.5, and the threshold is 0.7, then the first intersection-in-combination ratio 0.8 is greater than the threshold 0.7, and the second intersection-in-combination ratio 0.6 is less than the threshold 0.7 and / or the third intersection-in-combination ratio 0.5 is less than the threshold 0.7, that is, it is determined that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a candidate state;

[0080] If the first intersection-and-union ratio is 0.8, the second intersection-and-union ratio is 0.7, the third intersection-and-union ratio is 0.8, and the threshold is 0.7, then the first intersection-and-union ratio of 0.8 is greater than the threshold of 0.7, and the second intersection-and-union ratio of 0.7 is equal to the threshold of 0.7 and / or the third intersection-and-union ratio of 0.8 is greater than the threshold of 0.7, that is, it is determined that the image acquisition device is positionally associated with the hand, the side face, and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a state of listening to speech;

[0081] If the first intersection-over-union ratio is 0.5 and the threshold is 0.7, the first intersection-over-union ratio 0.5 is less than the threshold 0.7, which means that it is determined that there is no position association between the image acquisition device and the hand, and the position association detection result is that the image acquisition device is in a single state.

[0082] 208. If the position association detection result is that the image acquisition device is in a candidate state, a perspective transformation is performed on the camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or ear;

[0083] The shooting behavior detection device can perform a perspective transform on the camera of the image acquisition device to generate a corresponding camera perspective transform plane, or perform an affine transform on the camera of the image acquisition device to generate a corresponding camera affine transform plane. Both the camera perspective transform plane and the camera affine transform plane can be used to calculate the angle between the plane of the single-frame video image. Among them, perspective transformation is the process of projecting an image from one viewing plane to another viewing plane, so perspective transformation is also called projection mapping. Perspective transformation is a spatial three-dimensional transformation. According to the coordinate variance, the three-dimensional coordinates are projected to another viewing plane. For example, a three-dimensional cube is perspective transformed to obtain a two-dimensional cube viewing plane.

[0084] Specifically, (1) if the position association detection result shows that the image acquisition device is in a candidate state, the shooting behavior detection device performs camera detection on the single-frame video image. If there is a camera corresponding to the image acquisition device in the single-frame video image, the single-frame video image is segmented to obtain a camera image of the image acquisition device; (2) edge detection is performed on the camera image to generate a corresponding camera edge image; (3) perspective transformation is performed on the camera edge image to obtain a camera perspective transformation plane.

[0085] For example, if the position association detection result is that the image acquisition device is in the candidate state, the shooting behavior detection device performs camera detection on the single-frame video image. If there is a camera corresponding to the image acquisition device in the single-frame video image, the single-frame video image is segmented to obtain the camera image of the image acquisition device; edge detection is performed on the camera image to generate the corresponding camera edge image; according to the perspective transformation formula and the camera edge image to obtain the camera perspective transformation plane, where the horizontal coordinate of the camera perspective transformation plane is Vertical axis Used to indicate a linear transformation, [a 31 a 32 ] is used to indicate translation, [a 13 a 23 ] T Used to create perspective.

[0086] 209. If the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0087] If the image acquisition device is in a shooting state, it is determined that the target person is in shooting behavior, and the shooting behavior detection device generates warning information. The warning information includes multiple warning information, such as "Please note! Someone is taking pictures secretly." Voice warning information, "Warning light flashing" light warning information, voice and light combined sound and light warning information, and other warning information.

[0088] The shooting behavior detection device may issue an early warning through an early warning terminal in a variety of ways, for example, issuing an early warning by playing a voice message of "Please note! Someone is secretly taking photos." through the early warning terminal, issuing an early warning by controlling the warning light of the early warning terminal to flash, issuing an early warning by playing a voice message of "Please note! Someone is secretly taking photos." through the early warning terminal and controlling the warning light of the early warning terminal to flash, and the like.

[0089] In an embodiment of the present invention, when a target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; a marker recognition is performed on the single-frame video image through a target detection network to obtain a marker recognition result, wherein the marker is used to indicate an image acquisition device; if the marker recognition result is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; if the position association detection result is that the image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; If the measurement result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior, thereby improving the accuracy of human behavior recognition.

[0090] The above describes the shooting behavior detection method in the embodiment of the present invention. The following describes the shooting behavior detection device in the embodiment of the present invention. Figure 3 , an embodiment of the shooting behavior detection device in the embodiment of the present invention includes:

[0091] The acquisition module 301 is used to acquire a single-frame video image of the target person through a preset monitoring device when the target person is in a preset detection area, wherein the monitoring device is used to detect the detection area, and the detection area is the area captured by the monitoring device;

[0092] The recognition module 302 is used to perform landmark recognition on a single frame of video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device;

[0093] The detection module 303 is used to perform position association detection on the image acquisition device, the hand, side face and ear of the target person according to the intersection-and-union function to obtain a position association detection result if the landmark recognition result is that the image acquisition device exists in the single-frame video image;

[0094] A transformation module 304 is configured to perform perspective transformation on a camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device if the position association detection result indicates that the image acquisition device is in a candidate state, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear;

[0095] The determination module 305 is used to determine that the image acquisition device is in a shooting state if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to a preset angle, and generate an early warning message, and send the early warning message to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0096] In an embodiment of the present invention, when a target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; a marker recognition is performed on the single-frame video image through a target detection network to obtain a marker recognition result, wherein the marker is used to indicate an image acquisition device; if the marker recognition result is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; if the position association detection result is that the image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; If the measurement result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior, thereby improving the accuracy of human behavior recognition.

[0097] See also Figure 4 Another embodiment of the shooting behavior detection device in the embodiment of the present invention includes:

[0098] The acquisition module 301 is used to acquire a single-frame video image of the target person through a preset monitoring device when the target person is in a preset detection area, wherein the monitoring device is used to detect the detection area, and the detection area is the area captured by the monitoring device;

[0099] The recognition module 302 is used to perform landmark recognition on a single frame of video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device;

[0100] The detection module 303 is used to perform position association detection on the image acquisition device, the hand, side face and ear of the target person according to the intersection-and-union function to obtain a position association detection result if the landmark recognition result is that the image acquisition device exists in the single-frame video image;

[0101] A transformation module 304 is configured to perform perspective transformation on a camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device if the position association detection result indicates that the image acquisition device is in a candidate state, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear;

[0102] The determination module 305 is used to determine that the image acquisition device is in a shooting state if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to a preset angle, and generate an early warning message, and send the early warning message to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

[0103] Optionally, the identification module 302 includes:

[0104] The feature extraction unit 3021 is used to extract features from a single-frame video image through a backbone feature extraction network of the target detection network to generate three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer;

[0105] A feature enhancement unit 3022 is used to enhance the features of the three effective feature layers through the enhanced feature extraction network of the target detection network to generate corresponding three enhanced feature layers;

[0106] A marker prediction unit 3023 is used to perform marker prediction on the three enhanced feature layers through a prediction network of the target detection network to obtain marker prediction probabilities;

[0107] A first determining unit 3024 is configured to determine that the landmark recognition result is that an image acquisition device exists in the single-frame video image if the landmark prediction probability is greater than or equal to a preset image acquisition device prediction probability;

[0108] The second determination unit 3025 is configured to determine that the result of the landmark recognition is that the image acquisition device does not exist in the single-frame video image if the landmark prediction probability is less than a preset image acquisition device prediction probability.

[0109] Optionally, the feature extraction unit 3021 may also be specifically used for:

[0110] The single-frame video image is focused through the backbone feature extraction network of the target detection network to generate a twelve-channel single-frame video image;

[0111] The twelve-channel single-frame video image is subjected to feature extraction through three different convolution kernels to generate three corresponding effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer.

[0112] Optionally, the feature enhancement unit 3022 may also be specifically used for:

[0113] The third effective feature layer of the three effective feature layers is convolved and upsampled through the enhanced feature extraction network of the target detection network, and combined and feature extracted with the second effective feature layer to obtain an upsampled feature map;

[0114] The upsampled feature map is convolved and upsampled, and combined with the first effective feature layer and feature extracted to obtain a first enhanced feature layer;

[0115] Convolution, downsampling and feature extraction are performed on the first enhanced feature layer to obtain a second enhanced feature layer;

[0116] The second enhanced feature layer is convolved, downsampled and feature extracted to obtain a third enhanced feature layer.

[0117] Optionally, the marker prediction unit 3023 may also be specifically used for:

[0118] Performing marker prediction on each of the three enhanced feature layers through the prediction network of the target detection network, and obtaining three marker prediction results corresponding to each enhanced feature layer;

[0119] In each enhanced feature layer, the corresponding three marker prediction results are stacked to generate marker candidate results for each enhanced feature layer;

[0120] Based on the marker candidate results of each enhanced feature layer, the marker prediction probability is generated.

[0121] Optionally, the detection module 303 may also be specifically used for:

[0122] If the result of the landmark recognition is that there is an image acquisition device, the target detection network performs position frame detection on the image acquisition device, the hands, side face and ears of the target person, and generates corresponding image acquisition device position frame, hand position frame, side face position frame and ear position frame;

[0123] Obtaining a first intersection-and-union ratio value according to the intersection-and-union ratio function, the image acquisition device position frame, and the hand position frame, wherein the first intersection-and-union ratio value is used to indicate a degree of correlation between the positions of the image acquisition device and the hand of the target person;

[0124] According to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame and the side face position frame, a second intersection-and-union ratio is obtained, wherein the second intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand and side face of the target person;

[0125] According to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame, the side face position frame and the ear position frame, a third intersection-and-union ratio is obtained, wherein the third intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand, side face and ear of the target person;

[0126] If the first intersection-over-union ratio is greater than or equal to the threshold, and the second intersection-over-union ratio is less than the threshold and / or the third intersection-over-union ratio is less than the threshold, it is determined that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a candidate state;

[0127] If the first intersection-and-combination ratio is greater than or equal to the threshold, and the second intersection-and-combination ratio is greater than or equal to the threshold and / or the third intersection-and-combination ratio is greater than or equal to the threshold, it is determined that there is a position association between the image acquisition device and the hand, the side face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a state of listening to speech;

[0128] If the first intersection-to-union ratio is less than the threshold, it is determined that there is no position association between the image acquisition device and the hand, and the position association detection result is determined to be that the image acquisition device is in a single state.

[0129] Optionally, the transformation module 304 may also be specifically configured to:

[0130] If the position association detection result is that the image acquisition device is in a candidate state, a camera detection is performed on the single-frame video image. If a camera corresponding to the image acquisition device exists in the single-frame video image, the single-frame video image is segmented to obtain a camera image of the image acquisition device.

[0131] Perform edge detection on the camera image to generate a corresponding camera edge image;

[0132] Perform perspective transformation on the camera edge image to obtain the camera perspective transformation plane.

[0133] In an embodiment of the present invention, when a target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; a marker recognition is performed on the single-frame video image through a target detection network to obtain a marker recognition result, wherein the marker is used to indicate an image acquisition device; if the marker recognition result is that an image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; if the position association detection result is that the image acquisition device exists in the single-frame video image, a position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-and-union function to obtain a position association detection result; If the measurement result is that the image acquisition device is in a candidate state, the camera of the image acquisition device is perspective transformed to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or ear; if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior, thereby improving the accuracy of human behavior recognition.

[0134] above Figure 3 and Figure 4 The shooting behavior detection device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The shooting behavior detection device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0135] Figure 5 is a structural diagram of a shooting behavior detection device provided by an embodiment of the present invention. The shooting behavior detection device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the shooting behavior detection device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the shooting behavior detection device 500.

[0136] The shooting behavior detection device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the shooting behavior detection device shown does not constitute a limitation on the shooting behavior detection device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0137] The present invention also provides a shooting behavior detection device, wherein the computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the shooting behavior detection method in the above-mentioned embodiments.

[0138] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the shooting behavior detection method.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0141] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A shooting behavior detection method, It is characterized in that The shooting behavior detection method comprises: When the target person is in a preset detection area, a single-frame video image of the target person is obtained through a preset monitoring device, wherein the monitoring device is used to detect the detection area, and the detection area is the area photographed by the monitoring device; Performing landmark recognition on the single-frame video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device; If the landmark recognition result is that an image acquisition device exists in the single-frame video image, position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to an intersection-over-union function to obtain a position association detection result; If the position association detection result is that the image acquisition device is in a candidate state, a perspective transformation is performed on the camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand, and is not positionally associated with the side face and / or the ear; If the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to the preset angle, it is determined that the image acquisition device is in a shooting state, and an early warning message is generated and sent to the early warning terminal, wherein the shooting state is used to indicate that the target person is in shooting behavior.

2. The shooting behavior detection method according to claim 1, It is characterized in that The performing of landmark recognition on the single-frame video image through the target detection network to obtain a landmark recognition result includes: Perform feature extraction on the single-frame video image through a backbone feature extraction network of a target detection network to generate three effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer, and a third effective feature layer; Performing feature enhancement on the three effective feature layers through the enhanced feature extraction network of the target detection network to generate corresponding three enhanced feature layers; Performing marker prediction on the three enhanced feature layers through the prediction network of the target detection network to obtain marker prediction probabilities; If the marker prediction probability is greater than or equal to the preset image acquisition device prediction probability, then determining the marker recognition result as the presence of an image acquisition device in the single-frame video image; If the marker prediction probability is less than the preset image acquisition device prediction probability, the marker recognition result is determined to be that the image acquisition device does not exist in the single-frame video image.

3. The shooting behavior detection method according to claim 2, It is characterized in that The backbone feature extraction network of the target detection network extracts features from the single-frame video image to generate three effective feature layers, including: Focusing the single-frame video image through a backbone feature extraction network of a target detection network to generate a single-frame video image of twelve channels; The twelve-channel single-frame video image is subjected to feature extraction through three different convolution kernels to generate three corresponding effective feature layers, wherein the three effective feature layers include a first effective feature layer, a second effective feature layer and a third effective feature layer.

4. The shooting behavior detection method according to claim 2, It is characterized in that The step of performing feature enhancement on the three effective feature layers through the enhanced feature extraction network of the target detection network to generate three corresponding enhanced feature layers includes: Convolving and upsampling the third effective feature layer among the three effective feature layers through the enhanced feature extraction network of the target detection network, and combining and extracting features with the second effective feature layer to obtain an upsampled feature map; Convolving and upsampling the upsampled feature map, combining and extracting features from the first effective feature layer, to obtain a first enhanced feature layer; Convolution, downsampling and feature extraction are performed on the first enhanced feature layer to obtain a second enhanced feature layer; The second enhanced feature layer is convolved, downsampled and feature extracted to obtain a third enhanced feature layer.

5. The shooting behavior detection method according to claim 2, It is characterized in that The performing marker prediction on the three enhanced feature layers through the prediction network of the target detection network to obtain marker prediction probabilities includes: Performing marker prediction on each of the three enhanced feature layers through the prediction network of the target detection network to obtain three marker prediction results corresponding to each enhanced feature layer; In each enhanced feature layer, the corresponding three marker prediction results are stacked to generate marker candidate results for each enhanced feature layer; A marker prediction probability is generated based on the marker candidate results of each enhanced feature layer.

6. The shooting behavior detection method according to claim 1, It is characterized in that If the landmark recognition result is that there is an image acquisition device in the single-frame video image, position association detection is performed on the image acquisition device, the hand, side face and ear of the target person according to the intersection-and-combination function to obtain a position association detection result, including: If the landmark recognition result is that there is an image acquisition device, position frame detection is performed on the image acquisition device, the hands, side face and ears of the target person through the target detection network to generate corresponding image acquisition device position frame, hand position frame, side face position frame and ear position frame; Obtaining a first intersection-and-union ratio value according to the intersection-and-union ratio function, the image acquisition device position frame, and the hand position frame, wherein the first intersection-and-union ratio value is used to indicate a degree of position correlation between the image acquisition device and the hand of the target person; According to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame and the side face position frame, a second intersection-and-union ratio is obtained, wherein the second intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand and side face of the target person; According to the intersection-and-union ratio function, the image acquisition device position frame, the hand position frame, the side face position frame and the ear position frame, a third intersection-and-union ratio is obtained, wherein the third intersection-and-union ratio is used to indicate the degree of positional association between the image acquisition device and the hand, side face and ear of the target person; If the first intersection-over-union ratio is greater than or equal to a threshold value, and the second intersection-over-union ratio is less than a threshold value and / or the third intersection-over-union ratio is less than a threshold value, it is determined that the image acquisition device is positionally associated with the hand, and is not positionally associated with the profile face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a candidate state; If the first intersection-and-combination ratio is greater than or equal to a threshold, and the second intersection-and-combination ratio is greater than or equal to a threshold and / or the third intersection-and-combination ratio is greater than or equal to a threshold, it is determined that there is a position association between the image acquisition device and the hand, the profile face and / or the ear, and the position association detection result is determined to be that the image acquisition device is in a state of listening to speech; If the first intersection-over-union ratio is less than a threshold value, it is determined that there is no position association between the image acquisition device and the hand, and the position association detection result is determined to be that the image acquisition device is in a single state.

7. The shooting behavior detection method according to any one of claims 1 to 6, It is characterized in that If the position association detection result indicates that the image acquisition device is in a candidate state, performing perspective transformation on the camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device includes: If the position association detection result is that the image acquisition device is in a candidate state, performing camera detection on the single-frame video image, and if a camera corresponding to the image acquisition device exists in the single-frame video image, segmenting the single-frame video image to obtain a camera image of the image acquisition device; Performing edge detection on the camera image to generate a corresponding camera edge image; Performing perspective transformation on the camera edge image to obtain a camera perspective transformation plane.

8. A shooting behavior detection device, It is characterized in that The shooting behavior detection device comprises: An acquisition module, used for acquiring a single-frame video image of the target person through a preset monitoring device when the target person is in a preset detection area, wherein the monitoring device is used to detect the detection area, and the detection area is an area photographed by the monitoring device; A recognition module, used for performing landmark recognition on the single-frame video image through a target detection network to obtain a landmark recognition result, wherein the landmark is used to indicate an image acquisition device; A detection module, for performing position association detection on the image acquisition device, the hand, the side face and the ear of the target person according to an intersection-over-union function to obtain a position association detection result if the landmark recognition result is that an image acquisition device exists in the single-frame video image; a transformation module, configured to, if the position association detection result indicates that the image acquisition device is in a candidate state, perform perspective transformation on the camera of the image acquisition device to generate a camera perspective transformation plane corresponding to the image acquisition device, wherein the candidate state is used to indicate that the image acquisition device is positionally associated with the hand and is not positionally associated with the side face and / or the ear; A determination module is used to determine that the image acquisition device is in a shooting state if the angle between the camera perspective transformation plane and the plane of the single-frame video image is less than or equal to a preset angle, and generate an early warning message, and send the early warning message to an early warning terminal, wherein the shooting state is used to indicate that the target person has a shooting behavior.

9. A shooting behavior detection device, It is characterized in that The shooting behavior detection device comprises: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instruction in the memory so that the shooting behavior detection device executes the shooting behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, It is characterized in that When the instruction is executed by the processor, the shooting behavior detection method as described in any one of claims 1-7 is implemented.