Behavior detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a behavior detection method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a current to-be-detected video frame; detecting human heads and human bodies in the current to-be-detected video frame to obtain a human head image area occupied by each human head and a human body image area occupied by each human body; calculating an overlapping coefficient between each head image area and each detected human body image area; if there is an overlapping coefficient representing that the head in the head image area and the human body in the human body image area corresponding to the overlapping coefficient belong to the same person, obtaining an initial detection result of the head image area; and according to the initial detection result of other head image areas including the head in the head image area and the initial detection result of the head image area, determining a preset behavior currently existing in the head in the head image area. Therefore, the accuracy of behavior detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a behavior detection method, device, electronic equipment and storage medium. Background Art

[0002] In complex and ever-changing work scenarios, accurate monitoring of individual behavior is crucial. For example, in industrial production and construction environments, it is necessary to detect whether workers in the environment are wearing safety helmets; in road traffic environments, it is necessary to detect whether motorcycle drivers in the environment are wearing helmets.

[0003] In the related technology, computer vision technology can be used to identify whether the person in the image performs specific behaviors. However, due to factors such as dense crowds, changing environments, and complex and diverse individual behaviors, the accuracy of behavior detection is low, and false detections and missed detections occur frequently.

[0004] Therefore, how to improve the accuracy of behavior detection and avoid missed detection and false detection of individual behaviors has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a behavior detection method, device, electronic device and storage medium to improve the accuracy of behavior detection and avoid missed detection and false detection of individual behaviors. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present application, a behavior detection method is first provided, the method comprising:

[0007] Obtain the current video frame to be detected from the video to be detected;

[0008] Based on the pre-trained target detection model, the human head and human body in the current video frame to be detected are detected to obtain the head image area occupied by each human head and the human body image area occupied by each human body in the current video frame to be detected;

[0009] For each detected head image region, the overlap coefficient between the head image region and each detected human body image region is calculated; wherein the overlap coefficient between a head image region and a human body image region represents: the area ratio of the image region where the head image region intersects with the human body image region to the head image region or the human body image region;

[0010] If there is an overlap coefficient indicating that the head in the head image area and the human body in the human body image area corresponding to the overlap coefficient belong to the same person, the head image area is input into a pre-trained classification model to obtain an initial detection result of the head image area; wherein the initial detection result of the head image area indicates: a preset behavior of the head in the head image area;

[0011] Based on the initial detection results of other head image areas that contain a head in the head image area in at least one video frame before the current head image area in the video to be detected, and the initial detection result of the head image area, a final detection result indicating a preset behavior currently existing in the head in the head image area is determined.

[0012] In some embodiments, the training step of the target detection model includes:

[0013] Obtain multiple first sample images required for this round of training, and labels of each first sample image; wherein, the first sample image includes a first positive sample image and a first negative sample image, the first positive sample image includes a human head and / or a human body; the first negative sample image does not include a human head and a human body; the label of a first sample image represents the head image area occupied by the human head in the first sample image, and / or the human body image area occupied by the human body; based on the current target detection model, detect the human head and human body in each first sample image required for this round of training, and obtain the prediction result of the first sample image; wherein the prediction result of a first sample image represents: the head area occupied by each human head in the first sample image image area, and the human image area occupied by each human body; based on the difference between the label of the first sample image and the prediction result of the first sample image, the loss value of this round is calculated; based on the loss value of this round, the model parameters of the current target detection model are adjusted, and the step of obtaining multiple first sample images required for this round of training and the label of each first sample image is returned to execute until convergence is reached; wherein, the first sample images required for each round of training include the designated first negative sample images used in the previous round of training; the designated first negative sample images used in a round of training are: during the process of this round of training, the first negative sample images in which the existence of human head image areas and / or human body image areas are detected.

[0014] In some embodiments, based on the current target detection model, the human head and human body in each first sample image to be used in this round of training are detected to obtain the prediction result of the first sample image, including: preprocessing each first sample image to obtain a preprocessed first sample image; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.; inputting each preprocessed first sample image into the current target detection model to obtain the first parameter, second parameter and position information of each first candidate detection box in the first sample image; wherein the first parameter of a first candidate detection box represents the probability that the image area corresponding to the first candidate detection box is not the background, and the second parameter of a first candidate detection box represents the maximum value of the probability that the image area corresponding to the first candidate detection box contains a human head and the probability that the human body is contained; for For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame; use the first candidate detection frame with a confidence greater than a first preset threshold as the second candidate detection frame; based on the non-maximum suppression algorithm, select at least one from the second candidate detection frames as a predicted detection frame; for each predicted detection frame, if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human head, then determine that the image area corresponding to the predicted detection frame in the first sample image is a human head image area occupied by a human head; if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, then determine that the image area corresponding to the predicted detection frame in the first sample image is a human body image area occupied by a human body.

[0015] In some embodiments, the method detects heads and bodies in the current video frame to be detected based on a pre-trained target detection model to obtain the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected, including: preprocessing the current video frame to be detected to obtain the preprocessed current video frame to be detected; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.; inputting the preprocessed current video frame to be detected into the pre-trained target detection model to obtain the first parameter, second parameter and position information of each first candidate detection frame in the current video frame to be detected; wherein the first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame contains a head and the maximum value of the probability of containing a human body; for each first candidate detection frame, calculating the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame; taking the first candidate detection frame whose confidence is greater than the second preset threshold as the second candidate detection frame; based on the non-maximum suppression algorithm, selecting at least one from the second candidate detection frames as the predicted detection frame; for each predicted detection frame, if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human head, then determining that the image area corresponding to the predicted detection frame in the current video frame to be detected is a human head image area occupied by a human head; if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, then determining that the image area corresponding to the predicted detection frame in the current video frame to be detected is a human body image area occupied by a human body.

[0016] In some embodiments, the step of inputting the head image region into a pre-trained classification model to obtain an initial detection result of the head image region includes: inputting the head image region into the pre-trained classification model, performing feature extraction on the head image region based on a specified network layer in the pre-trained classification model, and obtaining image features corresponding to the head image region; wherein the specified network layer includes: a convolution layer and a pooling layer; inputting the image features corresponding to the head image region into a normalization layer of the trained classification model to obtain normalized image features corresponding to the head image region; inputting the normalized image features corresponding to the head image region into a cosine classifier in the pre-trained classification model to obtain a comparison between the normalized image features corresponding to the head image region and each preset row. is the cosine similarity between the standard image features of a preset behavior; wherein the standard image feature of a preset behavior is: determined based on a second sample image representing the preset behavior; if there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the human head image area and the standard image features of each preset behavior, then the products of the cosine similarities corresponding to the human head image area and the temperature coefficient are calculated respectively; wherein a target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity; the calculated products are processed by the activation function in the pre-trained classification model to obtain a classification result representing the probability that the human head in the human head image area has each preset behavior, as the initial detection result of the human head image area.

[0017] In some embodiments, the training step of the classification model includes: obtaining second sample images and labels of each second sample image; wherein the label of a second sample image represents a preset behavior of a human head in the second sample image; inputting each second sample image into the classification model of the initial structure, performing feature extraction on the second sample image based on a specified network layer in the classification model of the initial structure, and obtaining image features corresponding to the second sample image; inputting the image features corresponding to the second sample image into the normalization layer of the classification model of the initial structure, and obtaining the normalized image features corresponding to the human head image area; inputting the normalized image features corresponding to the second sample image into the cosine classifier in the classification model of the initial structure, and obtaining the normalized image features corresponding to the human head image area and the standard images of the current preset behaviors. cosine similarity between features; if there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the second sample image and the standard image features of each preset behavior, then the products of each cosine similarity corresponding to the second sample image and the temperature coefficient are calculated respectively; wherein a target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity; the calculated products are processed by the activation function in the classification model of the initial structure to obtain a classification result representing the probability that the human head in the second sample image has each preset behavior; based on the difference between the classification result of the second sample image and the label of the second sample image, a loss value is calculated; based on the obtained loss value, the model parameters of the classification model of the initial structure are adjusted until convergence is reached.

[0018] In some embodiments, the final detection result indicating the preset behavior currently existing in the human head in the human head image area is determined based on the initial detection results of other human head image areas that include the human head in the human head image area in at least one video frame before the current video to be detected in the video to be detected, and the initial detection result of the human head image area, including: in the initial detection results of other human head image areas that include the human head in the human head image area in at least one video frame before the current video to be detected in the video to be detected, and the initial detection results of the human head image area, if the proportion of the initial detection results indicating that the preset behavior existing in the human head in the human head image area is a specified preset behavior is greater than a third preset threshold, it is determined that the human head in the human head image area currently has the specified preset behavior.

[0019] In some embodiments, obtaining the current video frame to be detected from the video to be detected includes: based on a preset sampling interval, sampling a video frame from the video to be detected in the order of video frames as the current video frame to be detected.

[0020] In some embodiments, the preset behaviors of the presence of a human head in any human head image area are: wearing a helmet, wearing other items except the helmet, or not wearing any items.

[0021] According to a second aspect of an embodiment of the present application, a behavior detection device is provided, the device comprising:

[0022] A video frame acquisition module is used to acquire the current video frame to be detected from the video to be detected;

[0023] The target detection module is used to detect the head and body in the current video frame to be detected based on the pre-trained target detection model, and obtain the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected;

[0024] The overlap coefficient calculation module is used to calculate the overlap coefficient between each detected head image region and each detected human body image region; wherein the overlap coefficient between a head image region and a human body image region represents: the area ratio of the image region where the head image region intersects with the human body image region to the head image region or the human body image region;

[0025] The classification module is used for inputting the human head image area into a pre-trained classification model to obtain an initial detection result of the human head image area if there is an overlap coefficient indicating that the human head in the human head image area and the human body in the human body image area corresponding to the overlap coefficient belong to the same person; wherein the initial detection result of the human head image area indicates: a preset behavior of the presence of the human head in the human head image area;

[0026] The behavior detection module is used to determine a final detection result indicating a preset behavior currently existing in the human head in the head image area based on the initial detection results of other head image areas containing a head in the head image area in at least one video frame before the current video to be detected in the video to be detected, and the initial detection result of the head image area.

[0027] According to a third aspect of the embodiments of the present application, an electronic device is provided, including:

[0028] Memory, used to store computer programs;

[0029] The processor is used to implement any of the above-mentioned behavior detection methods when executing the program stored in the memory.

[0030] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned behavior detection methods is implemented.

[0031] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described behavior detection methods.

[0032] Beneficial effects of the embodiments of the present application:

[0033] The embodiment of the present application provides a behavior detection method, which can determine the head image area occupied by each head and the human body image area occupied by each human body in the current video frame to be detected through the target detection model. Furthermore, for each head image area, according to the overlap coefficient between the head image area and each detected human body image area, the head image area that needs to be classified can be determined through each overlap coefficient of the head image area, thereby avoiding false detection of the head image area. Furthermore, if there is an overlapping coefficient characterizing that the head in the head image area and the human body in the human body image area corresponding to the overlapping coefficient belong to the same person, the head image area is input into a pre-trained classification model to obtain an initial detection result of the head image area. The initial detection result of a head image area indicates: a preset behavior of the head in the head image area. It can be understood that since the behavior of the head of the same person does not change in a short period of time, the behavior of the head of the same person will not change. Therefore, the initial detection results of other head image areas that contain the head in the head image area in at least one video frame before the current video to be detected in the video to be detected, as well as the initial detection result of the head image area, can be combined to determine the preset behavior of the head in the head image area. In this way, it is possible to avoid incorrect detection results due to errors in the classification results of a single head image area, thereby improving the accuracy of behavior detection and avoiding missed detections and false detections.

[0034] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0036] Figure 1 A first flow chart of the behavior detection method provided in the embodiment of the present application;

[0037] Figure 2 A schematic diagram of a process for detecting an image using the target detection model provided in an embodiment of the present application;

[0038] Figure 3 A second flow chart of the behavior detection method provided in the embodiment of the present application;

[0039] Figure 4 A schematic diagram of a process for performing target detection on an image provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of an overlapping area provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a process for screening a human head image area provided in an embodiment of the present application;

[0042] Figure 7 A third flow chart of the behavior detection method provided in the embodiment of the present application;

[0043] Figure 8 A schematic diagram of a process for obtaining a classification result of a head image region provided in an embodiment of the present application;

[0044] Fig. 9 A structural diagram of a model training device provided in an embodiment of the present application;

[0045] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0047] In complex and ever-changing work scenarios, accurate monitoring of individual behavior is crucial. For example, in industrial production and construction environments, it is necessary to detect whether workers in the environment are wearing safety helmets; in road traffic environments, it is necessary to detect whether motorcycle drivers in the environment are wearing helmets.

[0048] In the related technology, computer vision technology can be used to identify whether the person in the image performs specific behaviors. However, due to factors such as dense crowds, changing environments, and complex and diverse individual behaviors, the accuracy of behavior detection is low, and false detections and missed detections occur frequently.

[0049] Therefore, how to improve the accuracy of behavior detection and avoid missed detection and false detection of individual behaviors has become an urgent problem to be solved.

[0050] The present application embodiment provides a behavior detection method, which can be applied to an electronic device. For example, the electronic device can be a smart device for providing a behavior detection function. Figure 1 , Figure 1 The first flow chart of the behavior detection method provided in the embodiment of the present application includes the following steps:

[0051] S101: Acquire a current video frame to be detected from the video to be detected.

[0052] S102: Based on the pre-trained target detection model, detect the head and body in the current video frame to be detected, and obtain the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected.

[0053] S103: For each detected head image region, calculating an overlap coefficient between the head image region and each detected human body image region.

[0054] The overlap coefficient between a head image region and a human body image region represents: the ratio of the area of ​​the image region where the head image region and the human body image region intersect to the area of ​​the head image region or the human body image region.

[0055] S104: If there is an overlap coefficient indicating that the head in the head image region and the human body in the human body image region corresponding to the overlap coefficient belong to the same person, the head image region is input into a pre-trained classification model to obtain an initial detection result of the head image region.

[0056] Among them, the initial detection result of a head image area indicates: the preset behavior of the presence of a head in the head image area.

[0057] S105: Determine a final detection result indicating a preset behavior currently existing in the human head in the head image area based on initial detection results of other human head image areas in at least one video frame before the current video to be detected in the video to be detected and the initial detection result of the human head image area.

[0058] The embodiment of the present application provides a behavior detection method, which can determine the head image area occupied by each head and the human body image area occupied by each human body in the current video frame to be detected through the target detection model. Furthermore, for each head image area, according to the overlap coefficient between the head image area and each detected human body image area, the head image area that needs to be classified can be determined through each overlap coefficient of the head image area, thereby avoiding false detection of the head image area. Furthermore, if there is an overlapping coefficient characterizing that the head in the head image area and the human body in the human body image area corresponding to the overlapping coefficient belong to the same person, the head image area is input into a pre-trained classification model to obtain an initial detection result of the head image area. The initial detection result of a head image area indicates: a preset behavior of the head in the head image area. It can be understood that since the behavior of the head of the same person does not change in a short period of time, the behavior of the head of the same person will not change. Therefore, the initial detection results of other head image areas that contain the head in the head image area in at least one video frame before the current video to be detected in the video to be detected, as well as the initial detection result of the head image area, can be combined to determine the preset behavior of the head in the head image area. In this way, it is possible to avoid incorrect detection results due to errors in the classification results of a single head image area, thereby improving the accuracy of behavior detection and avoiding missed detections and false detections.

[0059] In this application, in the aforementioned industrial production and construction environments, in order to urge workers entering the construction site or production workshop to wear safety helmets, access control gates and cameras for collecting videos to be detected can be set at the entrances and exits of the construction site or production workshop. In this scenario, the behavior detection method provided in this application can be used to detect whether workers are wearing safety helmets when entering and exiting the access control gate. Subsequently, the gate door can be opened when the worker is wearing a safety helmet, or the worker can be reminded to wear a safety helmet when the worker is not wearing a safety helmet.

[0060] Alternatively, in a road traffic environment, it is necessary to detect whether a motorcycle or electric vehicle driver in the environment is wearing a helmet. A camera for collecting videos to be detected can be set up on a designated road section. In this scenario, the behavior detection method provided in this application can be used to detect whether a motorcycle or electric vehicle driver passing through this road section is wearing a helmet. Subsequently, if multiple drivers who are not wearing helmets are detected in the collected video, a traffic slogan can be set up on this road section to remind drivers passing through this road section to wear helmets.

[0061] With respect to step S101 , the video to be detected is: a video in which the behavior of a person needs to be detected.

[0062] In one implementation, the video to be detected may be a video stream collected in real time, and the electronic device may select a latest video frame from the acquired video stream collected in real time as the current video frame to be detected.

[0063] For example, in the aforementioned industrial production and construction environments, cameras for collecting videos to be detected are set at the entrances and exits of construction sites or production workshops. Accordingly, the camera can collect videos when someone passes through the access control gate, and the electronic device can obtain the latest video frame in the real-time collected video stream as the current video frame to be detected.

[0064] In another implementation, the video to be detected may be a pre-captured video, and the electronic device may determine the current video frame to be detected from the video frames included in the video to be detected in order from first to last.

[0065] For example, in a road traffic environment, a camera for collecting videos to be detected is set up on a designated road section. Accordingly, a video clip collected by the camera during a morning rush hour on a certain day can be obtained as the video to be detected. Furthermore, the electronic device can determine the current video frame to be detected from the video frames included in the video to be detected in order from the first to the last.

[0066] In some embodiments, step S101 includes:

[0067] Based on the preset sampling interval, a video frame is sampled from the video to be detected in the order of the video frames as the current video frame to be detected.

[0068] It is understandable that the frame rate (FPS, Frame Per Second) of the video captured by the camera is usually between 10-100, that is, the time interval between two adjacent video frames in the captured video is short, and the change of the behavior of the same person can often be ignored.

[0069] In this embodiment, a uniform sampling method can be adopted to select the video frame that needs to be detected from the video to be detected. That is, the electronic device can sample a video frame from the video to be detected in the order of the video frames based on the preset sampling interval as the current video frame to be detected. The preset sampling interval is set by the technician according to actual needs. For example, the preset sampling interval can be 5.

[0070] In one implementation, when the video to be detected is a video stream collected in real time, the latest video frame collected in real time can be obtained according to a preset sampling interval as the current video frame to be detected.

[0071] In another implementation, when the video to be detected is a pre-collected video, sampling can be performed based on a preset sampling interval according to the sequence of each video frame in the video to be detected, and the sampled video frame is determined as the current video frame to be detected.

[0072] Based on the above processing, since the time interval between two adjacent video frames in the video is short, the image pictures of the two video frames are often close. Therefore, a part of the video frames can be selected from the video by uniform sampling, and behavior detection can be performed one by one, thus avoiding the waste of computing resources caused by behavior detection on a large number of repeated video frames. In addition, the frame skipping detection method can provide sufficient computing resources and time resources for each video frame to be detected.

[0073] With respect to step S102 , the pre-trained object detection model is used to detect the heads and bodies in the image to obtain the head image area occupied by each head and the body image area occupied by each body in the image.

[0074] Among them, when an image contains a complete human body of a person, the head image area occupied by a head in the image is: the image area corresponding to the circumscribed rectangle of the head of the person in the image. The human body image area occupied by a human body in the image is: the image area corresponding to the circumscribed rectangle of the human body (including the head) of the person in the image.

[0075] It is understandable that, since the behavior detection method provided by the present application needs to detect the behavior of the human head in the human head image area in the image, the requirement for the integrity of the human body is not high when detecting the human body image area. For example, when only the upper body of a person appears in the image, the image area corresponding to the circumscribed rectangle of the upper body of the person in the image can be determined as a human body image area in the image.

[0076] For example, the target detection model can be trained based on the YOLOX (You Only Look Once X) model.

[0077] In some embodiments, the training step of the target detection model includes:

[0078] Step 1: Obtain multiple first sample images required for this round of training, as well as labels for each first sample image.

[0079] Among them, the first sample image includes a first positive sample image and a first negative sample image, the first positive sample image includes a human head and / or a human body; the first negative sample image does not include a human head and a human body; a label of a first sample image represents a head image area occupied by a human head and / or a human body image area occupied by a human body in the first sample image.

[0080] Step 2: Based on the current target detection model, detect the head and body in each first sample image required for this round of training to obtain the prediction result of the first sample image.

[0081] The prediction result of a first sample image represents: the head image area occupied by each head and the human body image area occupied by each human body in the first sample image.

[0082] Step 3: Based on the difference between the label of the first sample image and the prediction result of the first sample image, calculate the loss value of this round.

[0083] Step 4: Based on the loss value of this round, adjust the model parameters of the current target detection model and return to step 1 until convergence is reached.

[0084] Among them, the first sample image required for each round of training includes the specified first negative sample image used in the previous round of training; the specified first negative sample image used in a round of training is: the first negative sample image in which the presence of a head image area and / or a human body image area is detected during the training process.

[0085] In this embodiment, the target detection model is obtained based on multiple rounds of training. During a round of training of the target detection model, a sample image (i.e., a first sample image) required for this round of training can be obtained. The first sample image includes a positive sample image (i.e., a first positive sample image) and a negative sample image (i.e., a first negative sample image). The positive sample image can be: an image containing only a human head, or an image containing only a human body, or an image containing a human head and a human body. Accordingly, the label of a positive sample image represents: the head image area occupied by each head in the sample image, and / or the human body image area occupied by each human body. The negative sample image does not contain a human head or a human body. Accordingly, the label of a negative sample image is zero.

[0086] The electronic device can detect the human head and human body in each first sample image required for this round of training based on the current target detection model to obtain a prediction result of the first sample image.

[0087] In one implementation, for each first sample image, the first sample image can be directly input into the current target detection model to obtain the prediction result output by the current target detection model. The specific process of obtaining the prediction result output by the current target detection model will be described in subsequent embodiments.

[0088] In some embodiments, the above step 2 includes:

[0089] Step a: preprocess each first sample image to obtain a preprocessed first sample image.

[0090] The preprocessing includes at least one of the following processes: denoising, scaling, etc.

[0091] Step b: Input each preprocessed first sample image into the current target detection model to obtain the first parameter, second parameter and position information of each first candidate detection box in the first sample image.

[0092] Among them, the first parameter of a first alternative detection frame represents the probability that the image area corresponding to the first alternative detection frame is not the background, and the second parameter of a first alternative detection frame represents the maximum value of the probability that the image area corresponding to the first alternative detection frame contains a human head and the probability that the image area contains a human body.

[0093] Step c: For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame.

[0094] Step d: The first candidate detection box whose confidence is greater than the first preset threshold is used as the second candidate detection box.

[0095] Step e: Based on the non-maximum suppression algorithm, select at least one from the second candidate detection box as the predicted detection box.

[0096] Step f: For each predicted detection frame, if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human head, then the image area corresponding to the predicted detection frame in the first sample image is determined to be a head image area occupied by a human head; if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, then the image area corresponding to the predicted detection frame in the first sample image is determined to be a human body image area occupied by a human body.

[0097] In this embodiment, for each first sample image, the first sample image may be preprocessed, wherein the preprocessing includes at least one of the following processing: denoising, scaling, and the like.

[0098] For example, when the target detection model can detect images of a size of 224×224, if the size of a first sample image is 448×448, the size of the first sample image can be reduced to a size that can be detected by the target detection model.

[0099] For another example, the first sample image may be denoised, for example, based on a denoising algorithm such as a Gaussian low-pass filter algorithm or a non-local mean algorithm, to reduce interference of noise on the target detection result.

[0100] Furthermore, each preprocessed first sample image can be input into the current target detection model, and the target detection model can extract features from the preprocessed first sample image, process the obtained image features, and output the first parameters (which can be expressed as S obj ), the second parameter (which can be expressed as S cls ) and position information. The first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the maximum value of the probability that the image area corresponding to the first candidate detection frame contains a human head and the probability that the image area contains a human body.

[0101] See also Figure 2 , Figure 2 A schematic diagram of the process of detecting an image using the target detection model provided in the embodiment of the present application. The target detection model is obtained by training the YOLOX network model. The electronic device can input the image that needs to be detected into the target detection model. The target detection model can output the first parameter (Obj), the second parameter (Cls) and the position information (Reg) of each detection box (i.e., the first candidate detection box) based on the two branches of the detection head (decoupled detection head) structure. Wherein, Conv represents the convolution layer of the target detection model, BN represents the normalization layer of the target detection model, and Relu represents the activation function of the target detection model.

[0102] Correspondingly, the confidence of a first candidate detection box (which can be expressed as S c ) can be calculated by the following formula:

[0103] S c =S cls ×S obj

[0104] Among them, S obj is the first parameter of the first candidate detection box, S cls is the second parameter of the first candidate detection box.

[0105] It can be understood that, for a first candidate detection frame, the smaller the value of the first parameter, the greater the possibility that the image area corresponding to the first candidate detection frame is the background. The smaller the value of the second parameter, the smaller the possibility that a head or a body exists in the image area corresponding to the first candidate detection frame. And because the confidence of a first candidate detection frame is proportional to both the first parameter and the second parameter, the confidence of a first candidate detection frame can reflect the possibility that the image area corresponding to the detection frame is not the background and that a head or a body exists.

[0106] That is, the greater the confidence of a first candidate detection frame, the higher the possibility that the image area corresponding to the detection frame is not the background and that there is a head or a human body. Therefore, the first candidate detection frame with a confidence greater than the first preset threshold can be used as the second candidate detection frame.

[0107] For each target (human head or human body) in the first sample image, the target detection model may output multiple detection frames containing the target. Therefore, the second candidate detection frame can be processed based on the non-maximum suppression algorithm, that is, for one target, only one detection frame containing the target is retained as the predicted detection frame. That is, each predicted detection frame corresponds to the image area occupied by different targets in the first sample image.

[0108] Furthermore, for each predicted detection frame, if the second parameter of the predicted detection frame represents the probability that the image area corresponding to the predicted detection frame contains a human head, the image area corresponding to the predicted detection frame in the first sample image can be determined based on the position information of the predicted detection frame: a head image area occupied by a human head.

[0109] If the second parameter of the predicted detection frame represents the probability that the image area corresponding to the predicted detection frame contains a human body, it can be determined based on the position information of the predicted detection frame that the image area corresponding to the predicted detection frame in the first sample image is a human image area occupied by a human body.

[0110] In this way, each first sample image can be preprocessed, and the preprocessed first sample image can be input into the current target detection model to obtain all the first candidate detection frames in the first sample image. Then, the first candidate detection frames can be screened based on the confidence level to obtain the second candidate detection frames. In this way, the probability of false detection by the target detection model can be reduced, and based on the non-maximum suppression algorithm, for a target (human head or human body), only one second candidate detection frame containing the target is retained as the predicted detection frame. In this way, each human head image area and each human body image area in the first sample image can be obtained according to each predicted detection frame.

[0111] After obtaining the prediction result of the first sample image output by the current object detection model, the loss value of this round can be calculated based on the difference between the label of the first sample image and the prediction result of the first sample image.

[0112] When the first sample image is a positive sample image, the first detection sub-loss value (which may be called L1 Loss) representing position loss, the second detection sub-loss value (which may be called ConfLoss) representing confidence difference, the third detection sub-loss value (which may be called cls Loss) representing category loss, and the fourth detection sub-loss value (which may be called IOU Loss) representing intersection-over-union loss may be calculated based on each head image region and / or body image region in the first sample image represented by the label of the first sample image and each head image region and / or body image region in the first sample image represented by the prediction result of the first sample image. Furthermore, the sum of the first detection sub-loss value, the second detection sub-loss value, the third detection sub-loss value, and the fourth detection sub-loss value may be calculated as the loss value of this round.

[0113] When the first sample image is a negative sample image, the first detection sub-loss value (which may be called L1 Loss) representing the position loss, the second detection sub-loss value (which may be called ConfLoss) representing the confidence difference, the third detection sub-loss value (which may be called cls Loss) representing the category loss, the fourth detection sub-loss value (which may be called IOU Loss) representing the intersection-over-union loss, and the fifth detection sub-loss value (which may be called negLoss) representing the negative sample loss may be calculated based on each head image region and / or human body image region in the first sample image represented by the prediction result of the first sample image. Furthermore, the sum of the first detection sub-loss value, the second detection sub-loss value, the third detection sub-loss value, the fourth detection sub-loss value, and the fifth detection sub-loss value may be calculated as the loss value of this round.

[0114] Furthermore, the model parameters of the current target detection model can be adjusted based on the loss value of this round.

[0115] In the present application, for a first sample image, if the above steps af are followed to obtain the head image area occupied by each head and the body image area occupied by each body in the first sample image, then it can be determined that the sample image is a sample image in which the presence of a head image area and / or a body image area is detected.

[0116] During this round of training, if a first sample image is a negative sample image, and the current target detection model detects the presence of a head image area and / or a human body image area in the first sample image, it indicates that there is a false detection in the prediction result of the first sample image. Therefore, the first sample image can be used as the sample image for the next round of training.

[0117] Based on the above processing, the target detection model can be trained for multiple rounds based on dynamic negative sample training. That is, the first negative sample image in which the head image area and / or the human body image area is detected can be used as the sample image for the next round of training. In this way, the target detection model can accurately learn the characteristics of the negative sample image, enhance the model's learning ability for difficult negative sample images, reduce the probability of false detection of the target detection model, and thus improve the accuracy of target detection.

[0118] In one implementation, the electronic device can directly input the current video frame to be detected into a pre-trained target detection model, and accordingly, the target detection model can output the head image area occupied by each head in the current video frame to be detected, and the human body image area occupied by each human body. The specific process of obtaining the head image area occupied by each head in the current video frame to be detected, and the human body image area occupied by each human body will be described in subsequent embodiments.

[0119] In some embodiments, see Figure 3 , Figure 3 A second flow chart of the behavior detection method provided in the embodiment of the present application, in Figure 1 On the basis of, step S102 comprises:

[0120] S1021: Preprocess the current video frame to be detected to obtain the preprocessed current video frame to be detected.

[0121] The preprocessing includes at least one of the following processes: denoising, scaling, etc.

[0122] S1022: Input the preprocessed current video frame to be detected into a pre-trained target detection model to obtain the first parameter, the second parameter and the position information of each first candidate detection box in the current video frame to be detected.

[0123] Among them, the first parameter of a first alternative detection frame represents the probability that the image area corresponding to the first alternative detection frame is not the background, and the second parameter of a first alternative detection frame represents the maximum value of the probability that the image area corresponding to the first alternative detection frame contains a human head and the probability that the image area contains a human body.

[0124] S1023: For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame.

[0125] S1024: Use the first candidate detection box whose confidence is greater than the second preset threshold as the second candidate detection box.

[0126] S1025: Based on the non-maximum suppression algorithm, select at least one from the second candidate detection box as the predicted detection box.

[0127] S1026: For each predicted detection frame, if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human head, then the image area corresponding to the predicted detection frame in the current video frame to be detected is determined to be a head image area occupied by a human head; if the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, then the image area corresponding to the predicted detection frame in the current video frame to be detected is determined to be a human image area occupied by a human body.

[0128] In this embodiment, for the current video frame to be detected, the current video frame to be detected can be preprocessed. The specific preprocessing process is the same as that of step a above, that is, the process of preprocessing the first sample image, and will not be described in detail here. It can be understood that the way the first sample image is preprocessed during the training of the target detection model is consistent with the way the current video frame to be detected is preprocessed during the prediction based on the trained target detection model.

[0129] Furthermore, the preprocessed current video frame to be detected can be input into a pre-trained target detection model. For steps S1022-S1026, the preprocessed current video frame to be detected is input into the trained target detection model to obtain the head image area occupied by each head in the current video frame to be detected, and the human body image area occupied by each human body. The process can refer to the relevant description of the above steps b-f, that is, the preprocessed first sample image is input into the current target detection model to obtain the head image area occupied by each head in the first sample image, and the human body image area occupied by each human body. The details will not be repeated here.

[0130] See also Figure 4 , Figure 4 A schematic diagram of the process of performing target detection on an image provided in an embodiment of the present application. Figure 4 In the process, the image to be detected can be preprocessed, and then the preprocessed image can be input into the target detection model for feature extraction. The target detection model can perform model inference to obtain the first parameter, second parameter and position information of each first candidate detection box. Then, each first candidate detection box can be post-processed (such as screening by confidence, non-maximum suppression processing, etc.) to obtain each predicted detection box.

[0131] In this way, the electronic device can detect the heads and bodies in the current video frame to be detected based on the pre-trained target detection model, and obtain the head image area occupied by each head in the current video frame to be detected, and the body image area occupied by each body.

[0132] It can be understood that, based on the above step S102, if the head image area and the human body image area are not detected in the current video frame to be detected, it indicates that there is no person in the current video frame to be detected. Therefore, there is no need to perform subsequent processing on the current video frame to be detected according to steps S103-S105.

[0133] If only the head image area is detected for the current video frame to be detected, it means that only a very small part of the person has entered the image screen of the current video frame to be detected (for example, only the head has entered the image screen of the current video frame to be detected), or the target detection model has mistakenly detected a non-human head target as a human head. Therefore, there is no need to perform subsequent processing on the current video frame to be detected according to steps S103-S105.

[0134] If only the human image area is detected for the current video frame to be detected, since the behavior detection method provided in the present application needs to detect the behavior of the human head, behavior detection cannot be performed when only the human image area is detected, and there is no need to perform subsequent processing on the current video frame to be detected according to steps S103-S105.

[0135] In the present application, when a head image region and a human body image region are detected in the current video frame to be detected, subsequent processing can be performed on the current video frame to be detected according to steps S103 to S105.

[0136] For steps S103-S104, for a person in the current video frame to be detected, there must be an intersecting image area (which can be called an overlapping area) between the image area occupied by the person's head and the image area occupied by the person's body, and the area of ​​the intersecting image area is close to the area of ​​the image area occupied by the person's head.

[0137] See also Figure 5 , Figure 5 A schematic diagram of an overlapping area provided in an embodiment of the present application. Figure 5 In the figure, the solid-line frame A represents the image area occupied by the head of the figure (i.e., a head image area), and the dotted-line frame B represents the image area occupied by the body of the figure (i.e., a body image area). The overlapping area (A∩B) of the rectangular frame A and the rectangular frame B is a rectangular area filled with oblique lines.

[0138] Correspondingly, for each detected human head image region, an overlap coefficient between the human head image region and each detected human body image region may be calculated.

[0139] In one implementation, for a head image region and a human body image region, the area of ​​the image region where the head image region and the human body image region intersect can be calculated, and the ratio of the area of ​​the smaller area between the head image region and the human body image region can be calculated as the overlap coefficient of the head image region and the human body image region.

[0140] In this embodiment, the image area occupied by the head of a person in the image is often smaller than the image area occupied by the body of the person, and because the area of ​​the intersecting image area is close to the area of ​​the image area occupied by the head of the person, the larger the calculated overlap coefficient is, the greater the probability that the head image area and the body image area belong to the same person. That is, if the calculated overlap coefficient is greater than the first preset coefficient, it can be considered that the overlap coefficient indicates that the head in the head image area and the body in the body image area corresponding to the overlap coefficient belong to the same person.

[0141] In another implementation, for a head image region and a human body image region, the area of ​​the image region where the head image region and the human body image region intersect can be calculated, and the ratio of the area of ​​the larger area between the head image region and the human body image region can be calculated as the overlap coefficient of the head image region and the human body image region.

[0142] In this embodiment, the image area occupied by the head of a person in the image is often larger than the image area occupied by the body of the person. Based on the physiological relationship between the head and the body of a person, it can be known that the size relationship between the area of ​​the image area occupied by the head of a person in the image and the area of ​​the image area occupied by the body of the person is within a fixed range. For example, when only the upper body of a person appears in the image, the ratio of the image area of ​​the head of the person to the image area of ​​the body is usually between 0.2-0.5. When the whole body of a person appears in the image, the ratio of the image area of ​​the head of the person to the image area of ​​the body is usually between 0.1-0.3.

[0143] Since the area of ​​the intersecting image area is close to the area of ​​the image area occupied by the head of the person, the calculated overlap coefficient is within the preset coefficient interval, indicating that the probability that the head image area and the body image area belong to the same person is greater. That is, if the calculated overlap coefficient is within the preset coefficient interval, it can be considered that the overlap coefficient indicates that the head in the head image area and the body in the body image area corresponding to the overlap coefficient belong to the same person. The preset coefficient interval can be 0.1-0.6.

[0144] It is understandable that there must be overlapping image areas between the image area occupied by a person's head and the image area occupied by the person's body. Accordingly, if the image area where a person's head image area overlaps with a person's body image area is 0, the calculated overlap coefficient between the person's head image area and the person's body image area is 0, indicating that the person's head image area and the person's body image area do not belong to the same person.

[0145] For each head image area, if there is an overlapping coefficient among the overlapping coefficients corresponding to the head image area, which represents that the head in the head image area and the human body in the human body image area corresponding to the overlapping coefficient belong to the same person, then the head image area can be input into a pre-trained classification model, and the classification model can classify the head image area to determine the preset behavior of the head in the head image area in the video frame.

[0146] See also Figure 6 , Figure 6 A schematic diagram of a process for screening a human head image area provided in an embodiment of the present application. Figure 6 In the method, for a head image area, the overlap coefficient between the head image area and each human body image area can be calculated. If the overlap coefficient meets the preset conditions, it means that the overlap coefficient indicates that the head in the head image area and the human body in the human body image area corresponding to the overlap coefficient belong to the same person, then the subsequent classification process can be entered. Otherwise, the head image area is filtered out.

[0147] For any scene, a plurality of different preset behaviors can be preset. Accordingly, an initial detection result of a head image region indicates that the behavior of the head in the head image region is one of the plurality of preset behaviors.

[0148] For example, in the aforementioned industrial production and construction environments, it is necessary to detect whether the workers in the environment are wearing helmets. Accordingly, a variety of different preset behaviors are: wearing helmets, wearing other items except helmets, and not wearing anything.

[0149] In another implementation, in the aforementioned road traffic environment, it is necessary to detect whether a motorcycle driver in the environment is wearing a helmet. Accordingly, a plurality of different preset behaviors are: wearing a helmet, wearing other items except a helmet, and not wearing any items.

[0150] The classification model can be obtained by training based on the ResNet (residual network) model. The specific process of training the classification model will be described in the subsequent embodiments.

[0151] In some embodiments, see Figure 7 , Figure 7A third flow chart of the behavior detection method provided in the embodiment of the present application, in Figure 1 On the basis of this, step S104 comprises:

[0152] S1041: Input the human head image region into a pre-trained classification model, perform feature extraction on the human head image region based on a specified network layer in the pre-trained classification model, and obtain image features corresponding to the human head image region.

[0153] The specified network layer includes: a convolutional layer and a pooling layer.

[0154] S1042: Input the image features corresponding to the human head image region into the normalization layer of the trained classification model to obtain the normalized image features corresponding to the human head image region.

[0155] S1043: Input the normalized image features corresponding to the human head image area into the cosine classifier in the trained classification model to obtain the cosine similarity between the normalized image features corresponding to the human head image area and the standard image features of each preset behavior.

[0156] The standard image feature of a preset behavior is determined based on a second sample image representing the preset behavior.

[0157] S1044: If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, the products of each cosine similarity corresponding to the head image region and the temperature coefficient are calculated respectively.

[0158] Wherein, a target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of a preset behavior corresponding to the target cosine similarity.

[0159] S1045: The calculated products are processed by the activation function in the pre-trained classification model to obtain a classification result representing the probability of the presence of each preset behavior of the head in the head image area as the initial detection result of the head image area.

[0160] In this embodiment, for the head image region input to the pre-trained classification model, feature extraction can be performed on the head image region based on the specified network layer in the pre-trained classification model to obtain image features corresponding to the head image region. The specified network layer includes at least one convolution layer and a pooling layer. Furthermore, the image features corresponding to the head image region can be input to the normalization layer of the trained classification model to perform normalization processing to obtain the normalized image features corresponding to the head image region.

[0161] For each preset behavior, a standard image feature of the preset behavior may be determined in advance based on a second sample image representing the preset behavior.

[0162] For example, when the preset behaviors include: wearing a safety helmet, wearing items other than a safety helmet, and not wearing anything, the standard image features of the preset behavior indicating wearing a safety helmet can be determined based on the second sample image representing a person wearing a safety helmet, and the standard image features of the preset behavior indicating wearing items other than a safety helmet can be determined based on the second sample image representing a person wearing items other than a safety helmet, and the standard image features of the preset behavior indicating not wearing anything can be determined based on the second sample image representing a person not wearing anything.

[0163] Furthermore, the normalized image features corresponding to the head image region may be input into a cosine classifier in a trained classification model to calculate the cosine similarity between the normalized image features corresponding to the head image region and the standard image features of each preset behavior.

[0164] The cosine similarity between the image feature corresponding to the normalized head image area and the standard image feature of a preset behavior can be calculated by the following formula:

[0165]

[0166] Among them, Cos sim Representing image features With image features The cosine similarity of , ‖‖ represents the modulus. Image features is the image feature corresponding to the normalized head image area, image feature A standard image feature for a preset behavior.

[0167] When the preset behaviors include wearing a safety helmet, wearing other items except the safety helmet, and not wearing any items, for an image feature corresponding to a normalized head image area, the cosine similarity between the image feature corresponding to the normalized head image area and a standard image feature indicating wearing a safety helmet according to the preset behavior (which may be called a first cosine similarity), the cosine similarity between the image feature corresponding to the normalized head image area and a standard image feature indicating wearing other items except the safety helmet according to the preset behavior (which may be called a second cosine similarity), and the cosine similarity between the image feature corresponding to the normalized head image area and a standard image feature indicating not wearing any items according to the preset behavior (which may be called a third cosine similarity).

[0168] Furthermore, a screening can be performed based on the cosine similarities corresponding to the human head image region. Accordingly, for the cosine similarities between the normalized image features corresponding to the human head image region and the standard image features of each preset behavior, if each cosine similarity is less than the similarity threshold of the preset behavior corresponding to the cosine similarity, it can be determined that the human head image region does not contain a human head with any preset behavior.

[0169] For example, when the preset behaviors include wearing a safety helmet, wearing other items besides a safety helmet, and not wearing anything, the similarity threshold for the preset behavior of a human head wearing a safety helmet (which may be referred to as the first similarity threshold) may be 0.3, the similarity threshold for the preset behavior of a human head wearing other items besides a safety helmet (which may be referred to as the second similarity threshold) may be 0.4, and the similarity threshold for the preset behavior of a human head not wearing anything (which may be referred to as the third similarity threshold) may be 0.2.

[0170] If a cosine similarity is not less than the similarity threshold of the preset behavior corresponding to the cosine similarity, it indicates that the cosine similarity is a target cosine similarity. If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the head image area and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the head image area and the temperature coefficient is calculated respectively.

[0171] For example, for a head image region, if the first cosine similarity of the head image region is 0.5, the second cosine similarity is 0.2, and the third cosine similarity is 0.3, then the first cosine similarity of the head image region is not less than the first similarity threshold, the second cosine similarity of the head image region is less than the second similarity threshold, and the third cosine similarity of the head image region is not less than the third similarity threshold. If there is at least one target cosine similarity (i.e., the first cosine similarity and the third cosine similarity) in the cosine similarities between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, then the products of the cosine similarities corresponding to the head image region and the temperature coefficient can be calculated respectively.

[0172] If the first cosine similarity is not less than the first similarity threshold, and / or the second cosine similarity is not less than the second similarity threshold, and / or the third cosine similarity is not less than the third similarity threshold, then the product of the first cosine similarity and the temperature coefficient, the product of the second cosine similarity and the temperature coefficient, and the product of the third cosine similarity and the temperature coefficient corresponding to the head image area can be calculated respectively. Wherein, the temperature coefficient is pre-set by the technician according to actual needs, for example, the temperature coefficient can be 8. Accordingly, the calculated products are processed by the activation function in the pre-trained classification model to obtain the probability (i.e., the classification result) of the head in the head image area having each preset behavior. Wherein, for each preset behavior, the probability of the head in the head image area having the preset behavior is between [0, 1], and the sum of the probabilities of the head in the head image area having each preset behavior is 1. Furthermore, the preset behavior with the highest probability can be determined as the preset behavior of the head in the head image area as the initial detection result of the head image area.

[0173] In some embodiments, when the preset behaviors of a person's head in any head image area are: wearing a helmet, wearing other items in addition to a helmet, or not wearing anything, it is necessary to focus on monitoring whether the person's head is wearing a helmet. That is to say, in this scene, when the person's head is wearing other items in addition to a helmet, or not wearing anything, it is considered that the preset behavior of the person's head does not meet the requirement of wearing a helmet.

[0174] Accordingly, after obtaining the cosine similarity between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, it is possible to determine whether the head in the head image region has the behavior of wearing a helmet according to the cosine similarity between the image features corresponding to the head image region and the standard image features of wearing items other than a helmet (i.e., the second cosine similarity), and the cosine similarity between the image features corresponding to the head image region and the standard image features of not wearing any items (i.e., the third cosine similarity). That is, when the corresponding second cosine similarity in the head image region is less than the second similarity threshold, and the distance of the third cosine similarity is less than the third similarity threshold, it can be determined that the head in the head image region has the behavior of wearing a helmet, which is used as the initial detection result of the head image region.

[0175] For example, for a head image area, if the second cosine similarity corresponding to the head image area is 0.2, and the third cosine similarity is 0.1, at this time, the second cosine similarity of the head image area is less than the second similarity threshold (0.4), and the third cosine similarity of the head image area is less than the third similarity threshold (0.2), accordingly, it can be determined that the head in the head image area is wearing a helmet.

[0176] like Figure 8 As shown, Figure 8 A schematic diagram of a process for obtaining classification results of a head image area provided in an embodiment of the present application. Figure 8 In the embodiment, the electronic device can input an image (i.e., a head image region) into the classification model, and correspondingly, the designated network layer in the classification model can perform feature extraction on the head image region to obtain image features corresponding to the head image region. The normalization layer of the classification model can perform normalization processing on the image features corresponding to the head image region to obtain normalized image features corresponding to the head image region. The cosine similarity between the normalized image features corresponding to the head image region and the standard image features of each preset behavior can be obtained by the cosine classifier in the classification model. If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the head image region and the temperature coefficient is calculated respectively. The calculated products are processed by the activation function in the pre-trained classification model to obtain a classification result representing the probability of the head in the head image region having each preset behavior.

[0177] Based on the above processing, the cosine classifier can be used to calculate the cosine similarity between the image features corresponding to a head image area and the standard image features of each preset behavior. In this way, the angle between the two image features can be measured, and the similarity between the two image features can be reflected by the calculated cosine value. Then, the head image area can be classified, which can improve the accuracy of classification.

[0178] In some embodiments, the step of training the classification model includes:

[0179] Step 1: Obtain second sample images and labels of each second sample image.

[0180] The label of a second sample image indicates a preset behavior of the presence of a human head in the second sample image.

[0181] Step 2: Input each second sample image into the classification model of the initial structure, perform feature extraction on the second sample image based on the specified network layer in the classification model of the initial structure, and obtain image features corresponding to the second sample image.

[0182] Step 3: Input the image features corresponding to the second sample image into the normalization layer of the classification model of the initial structure to obtain the normalized image features corresponding to the head image area.

[0183] Step 4: Input the normalized image features corresponding to the second sample image into the cosine classifier in the classification model of the initial structure to obtain the cosine similarity between the normalized image features corresponding to the head image area and the standard image features of the current preset behaviors.

[0184] Step 5: If there is at least one target cosine similarity among the cosine similarities between the normalized image features corresponding to the second sample image and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the second sample image and the temperature coefficient is calculated respectively.

[0185] Wherein, a target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of a preset behavior corresponding to the target cosine similarity.

[0186] Step 6: Process the calculated products through the activation function in the classification model of the initial structure to obtain a classification result representing the probability of the human head in the second sample image having each preset behavior.

[0187] Step 7: Calculate the loss value based on the difference between the classification result of the second sample image and the label of the second sample image.

[0188] Step 8: Based on the obtained loss value, adjust the model parameters of the classification model of the initial structure until convergence is achieved.

[0189] In this embodiment, when the preset behaviors include wearing a safety helmet, wearing items other than a safety helmet, and not wearing anything, the second sample image includes: a sample image representing a person wearing a safety helmet, a sample image representing a person wearing items other than a safety helmet, and a sample image representing a person not wearing anything.

[0190] During the training process of the classification model, the standard image features of each preset behavior can be adjusted. For each preset behavior, the initial standard image features of the preset behavior can be obtained by random initialization. After obtaining the second sample image representing the preset behavior, the second sample image representing the preset behavior can be feature extracted based on the specified network layer in the classification model to obtain the image features of the second sample image representing the preset behavior. Furthermore, the multiple image features can be fused, and the initial standard image features of the preset behavior can be adjusted according to the obtained fusion result to obtain the current standard image features of the preset behavior. For example, the weighted sum of the multiple image features can be calculated as the fusion result of the multiple image features.

[0191] The process of classifying a second sample image based on the classification model of the initial structure to obtain a classification result of a preset behavior indicating the presence of a human head in the second sample image can correspond to the relevant description of the above-mentioned steps S1041-S1045, that is, the process of inputting a human head image area into the trained classification model to obtain a classification result of a preset behavior indicating the presence of a human head in the human head image area, which will not be elaborated here.

[0192] After obtaining the classification result of the preset behavior of the human head in the second sample image, the loss value can be calculated based on the difference between the classification result in the second sample image and the label of the second sample image. For example, the first classification sub-loss value can be calculated based on the Margin Loss function, the second classification sub-loss value can be calculated based on the Mixup Loss function, and the third classification sub-loss value can be calculated based on the Cross-Entropy Loss function, and the sum of the first classification sub-loss value, the second classification sub-loss value and the third classification sub-loss value is calculated as the loss value of this training.

[0193] Furthermore, the model parameters of the classification model can be adjusted based on the loss value of this training.

[0194] For step S105, for a head image area in the current video frame to be detected, after obtaining the initial detection result of the head image area, the preset behavior of the head of the first person (which can be called the head of the first person) in the first head image area can be combined with the preset behavior existing in other video frames to determine the current preset behavior of the head of the first person.

[0195] That is to say, the video frames (which may be referred to as historical detection video frames) that have been subjected to behavior detection according to the above steps S101 to S104 before the preset number of current video frames to be detected may be determined. Accordingly, the initial detection results of other head image regions containing the head of the first person in the historical detection video frames may be obtained. The preset number is set by the technician according to actual needs.

[0196] In some embodiments, step S105 includes:

[0197] In at least one video frame before the current video to be detected in the video to be detected, the initial detection results of other head image areas containing the head in the head image area, and in the initial detection results of the head image area, if the proportion of initial detection results indicating that the preset behavior of the head in the head image area is a specified preset behavior is greater than a third preset threshold, it is determined that the head in the head image area currently has the specified preset behavior.

[0198] In the present application, for the head of the first person, the initial detection results of the head image area occupied by the head of the first person in each historical detection video frame and the initial detection results of the head image area occupied by the head of the first person in the current video frame to be detected can be recorded in a smooth queue.

[0199] Furthermore, the initial detection results in which the preset behavior indicating the presence of a head is the specified preset behavior can be determined in each initial detection result, and the ratio of the number of initial detection results in which the preset behavior indicating the presence of a head is the specified preset behavior to the total number of initial detection results can be calculated. That is, if the number of initial detection results in which the preset behavior indicating the presence of a head is the specified preset behavior in the smooth queue satisfies the following formula, it can be determined that the head of the first person currently has the specified preset behavior:

[0200] Len cls ≥Len sQ ×R s

[0201] Among them, Len cls To smooth the number of initial detection results of the preset behavior indicating the presence of a person in the queue, Len sQ refers to the smoothed queue length, R s Refers to the smoothing ratio (ie, the third preset threshold).

[0202] For example, when the specified preset behavior represents wearing a safety helmet, and there are 5 initial detection results of the head image area occupied by the first person's head recorded in the smooth queue, if the third preset threshold is 0.6, it can be calculated that 5×0.6=3, that is, if there are at least three initial detection results indicating that the first person's head has the behavior of wearing a safety helmet in the head image area, it can be determined that the head in the head image area currently has the specified preset behavior, that is, the first person's head is currently wearing a safety helmet.

[0203] Based on the above processing, the initial detection results of other head image areas that contain a head in the head image area in at least one video frame before the current head image area in the video to be detected, as well as the initial detection result of the head image area, can be combined to determine whether the head in the head image area currently has a specified preset behavior. In this way, it is possible to avoid incorrect detection results due to errors in the classification results of a single head image area, thereby improving the accuracy of behavior detection and avoiding missed detections and false detections.

[0204] Based on the same inventive concept, the present application embodiment provides a behavior detection device, see Fig. 9 , Fig. 9 A structural diagram of a model training device provided in an embodiment of the present application. The device comprises:

[0205] The video frame acquisition module 901 is used to acquire the current video frame to be detected from the video to be detected;

[0206] The target detection module 902 is used to detect the head and the body in the current video frame to be detected based on the pre-trained target detection model, and obtain the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected;

[0207] The overlap coefficient calculation module 903 is used to calculate the overlap coefficient between each detected head image region and each detected human body image region; wherein the overlap coefficient between a head image region and a human body image region represents: the area ratio of the image region where the head image region intersects with the human body image region to the head image region or the human body image region;

[0208] The classification module 904 is used for inputting the human head image region into a pre-trained classification model to obtain an initial detection result of the human head image region if there is an overlap coefficient indicating that the human head in the human head image region and the human body in the human body image region corresponding to the overlap coefficient belong to the same person; wherein the initial detection result of the human head image region indicates: a preset behavior of the human head in the human head image region;

[0209] The behavior detection module 905 is used to determine the final detection result indicating the preset behavior currently existing in the human head in the head image area based on the initial detection results of other head image areas containing the head in the head image area in at least one video frame before the current video to be detected in the video to be detected, and the initial detection result of the head image area.

[0210] In some embodiments, the training step of the target detection model includes:

[0211] Acquire multiple first sample images required for this round of training, and labels of each first sample image; wherein the first sample images include a first positive sample image and a first negative sample image, the first positive sample image includes a human head and / or a human body; the first negative sample image does not include a human head and a human body; a label of a first sample image indicates a human head image area occupied by a human head, and / or a human body image area occupied by a human body in the first sample image;

[0212] Based on the current target detection model, detect the human head and human body in each first sample image required for this round of training to obtain the prediction result of the first sample image; wherein the prediction result of a first sample image represents: the human head image area occupied by each human head and the human body image area occupied by each human body in the first sample image;

[0213] Based on the difference between the label of the first sample image and the prediction result of the first sample image, a loss value of this round is calculated;

[0214] Based on the loss value of this round, the model parameters of the current object detection model are adjusted, and the step of obtaining a plurality of first sample images required for this round of training and a label of each first sample image is returned to execute until convergence is reached;

[0215] Among them, the first sample image required for each round of training includes the specified first negative sample image used in the previous round of training; the specified first negative sample image used in a round of training is: the first negative sample image in which the presence of a head image area and / or a human body image area is detected during the training process.

[0216] In some embodiments, based on the current target detection model, detecting the head and body in each first sample image required for the current round of training to obtain the prediction result of the first sample image includes:

[0217] Preprocessing each first sample image to obtain a preprocessed first sample image; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.;

[0218] Input each preprocessed first sample image into the current target detection model to obtain the first parameter, second parameter and position information of each first candidate detection frame in the first sample image; wherein the first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the maximum value of the probability that the image area corresponding to the first candidate detection frame contains a human head and the probability that the image area contains a human body;

[0219] For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame;

[0220] The first candidate detection frame whose confidence is greater than the first preset threshold is used as the second candidate detection frame;

[0221] Based on a non-maximum suppression algorithm, selecting at least one from the second candidate detection frames as a predicted detection frame;

[0222] For each predicted detection frame, if the second parameter of the predicted detection frame represents the probability that the image region corresponding to the predicted detection frame contains a human head, then determining that the image region corresponding to the predicted detection frame in the first sample image is a human head image region occupied by a human head;

[0223] If the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, it is determined that the image area corresponding to the predicted detection frame in the first sample image is a human body image area occupied by a human body.

[0224] In some embodiments, the target detection module 902 is specifically used to:

[0225] Preprocessing the current video frame to be detected to obtain the preprocessed current video frame to be detected; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.;

[0226] The preprocessed current video frame to be detected is input into a pre-trained target detection model to obtain the first parameter, the second parameter and the position information of each first candidate detection frame in the current video frame to be detected; wherein the first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the maximum value of the probability that the image area corresponding to the first candidate detection frame contains a human head and the probability that the image area contains a human body;

[0227] For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame;

[0228] The first candidate detection frame whose confidence is greater than the second preset threshold is used as the second candidate detection frame;

[0229] Based on a non-maximum suppression algorithm, selecting at least one from the second candidate detection frames as a predicted detection frame;

[0230] For each predicted detection frame, if the second parameter of the predicted detection frame represents the probability that the image region corresponding to the predicted detection frame contains a human head, it is determined that the image region corresponding to the predicted detection frame in the current video frame to be detected is a human head image region occupied by a human head;

[0231] If the second parameter of the predicted detection frame represents the probability that the image area corresponding to the predicted detection frame contains a human body, it is determined that the image area corresponding to the predicted detection frame in the current video frame to be detected is a human image area occupied by a human body.

[0232] In some embodiments, the classification module 904 is specifically used to:

[0233] The head image region is input into a pre-trained classification model, and features of the head image region are extracted based on a specified network layer in the pre-trained classification model to obtain image features corresponding to the head image region; wherein the specified network layer includes: a convolution layer and a pooling layer;

[0234] Inputting the image features corresponding to the human head image region into the normalization layer of the trained classification model to obtain the normalized image features corresponding to the human head image region;

[0235] Inputting the normalized image features corresponding to the human head image region into a cosine classifier in a pre-trained classification model to obtain the cosine similarity between the normalized image features corresponding to the human head image region and the standard image features of each preset behavior; wherein the standard image features of a preset behavior are: determined based on a second sample image representing the preset behavior;

[0236] If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the head image region and the temperature coefficient is calculated respectively; wherein one target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity;

[0237] The calculated products are processed by the activation function in the pre-trained classification model to obtain a classification result representing the probability of the presence of each preset behavior of the head in the head image area as the initial detection result of the head image area.

[0238] In some embodiments, the step of training the classification model includes:

[0239] Acquire second sample images and labels of each second sample image; wherein the label of a second sample image indicates a preset behavior of the presence of a human head in the second sample image;

[0240] Input each second sample image into the classification model of the initial structure, extract features of the second sample image based on the specified network layer in the pre-trained classification model, and obtain image features corresponding to the second sample image; wherein the specified network layer includes: a convolution layer and a pooling layer;

[0241] Inputting the image features corresponding to the second sample image into the normalization layer of the trained classification model to obtain the normalized image features corresponding to the human head image area;

[0242] Inputting the normalized image features corresponding to the second sample image into the cosine classifier in the classification model of the initial structure to obtain the cosine similarity between the normalized image features corresponding to the human head image area and the standard image features of each current preset behavior;

[0243] If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the second sample image and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the second sample image and the temperature coefficient is calculated respectively; wherein one target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity;

[0244] Processing the calculated products through the activation function in the classification model of the initial structure to obtain a classification result representing the probability of the human head in the second sample image having each preset behavior;

[0245] Calculating a loss value based on a difference between a classification result of the second sample image and a label of the second sample image;

[0246] Based on the obtained loss value, the model parameters of the classification model of the initial structure are adjusted until convergence is achieved.

[0247] In some embodiments, the behavior detection module 905 is specifically used to:

[0248] In at least one video frame before the current video to be detected in the video to be detected, the initial detection results of other head image areas containing the head in the head image area, and in the initial detection results of the head image area, if the proportion of initial detection results indicating that the preset behavior of the head in the head image area is a specified preset behavior is greater than a third preset threshold, it is determined that the head in the head image area currently has the specified preset behavior.

[0249] In some embodiments, the video frame acquisition module 901 is specifically used to:

[0250] Based on the preset sampling interval, a video frame is sampled from the video to be detected in the order of the video frames as the current video frame to be detected.

[0251] In some embodiments, the preset behaviors of the presence of a human head in any human head image area are: wearing a helmet, wearing other items except the helmet, or not wearing any items.

[0252] The present application also provides an electronic device, such as Fig.10As shown, it includes a processor 1001 , a communication interface 1002 , a memory 1003 and a communication bus 1004 , wherein the processor 1001 , the communication interface 1002 , and the memory 1003 communicate with each other via the communication bus 1004 .

[0253] Memory 1003, used for storing computer programs;

[0254] The processor 1001 is used to implement the steps of any behavior detection method in the above embodiments when executing the program stored in the memory 1003.

[0255] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0256] The communication interface is used for communication between the above electronic device and other devices.

[0257] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0258] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0259] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned behavior detection methods are implemented.

[0260] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any behavior detection method in the above embodiments.

[0261] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0262] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0263] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0264] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A behavior detection method, characterized in that: The method comprises: Obtain the current video frame to be detected from the video to be detected; Based on the pre-trained target detection model, the human head and human body in the current video frame to be detected are detected to obtain the head image area occupied by each human head and the human body image area occupied by each human body in the current video frame to be detected; For each detected head image region, the overlap coefficient between the head image region and each detected human body image region is calculated; wherein the overlap coefficient between a head image region and a human body image region represents: the area ratio of the image region where the head image region intersects with the human body image region to the head image region or the human body image region; If there is an overlap coefficient indicating that the head in the head image area and the human body in the human body image area corresponding to the overlap coefficient belong to the same person, the head image area is input into a pre-trained classification model to obtain an initial detection result of the head image area; wherein the initial detection result of the head image area indicates: a preset behavior of the head in the head image area; Based on the initial detection results of other head image areas that contain a head in the head image area in at least one video frame before the current head image area in the video to be detected, and the initial detection result of the head image area, a final detection result indicating a preset behavior currently existing in the head in the head image area is determined.

2. The method according to claim 1, characterized in that The training steps of the object detection model include: Acquire multiple first sample images required for this round of training, and labels of each first sample image; wherein the first sample images include a first positive sample image and a first negative sample image, the first positive sample image includes a human head and / or a human body; the first negative sample image does not include a human head and a human body; a label of a first sample image indicates a human head image area occupied by a human head, and / or a human body image area occupied by a human body in the first sample image; Based on the current target detection model, detect the human head and human body in each first sample image required for this round of training to obtain the prediction result of the first sample image; wherein the prediction result of a first sample image represents: the human head image area occupied by each human head and the human body image area occupied by each human body in the first sample image; Based on the difference between the label of the first sample image and the prediction result of the first sample image, a loss value of this round is calculated; Based on the loss value of this round, the model parameters of the current object detection model are adjusted, and the step of obtaining a plurality of first sample images required for this round of training and a label of each first sample image is returned to execute until convergence is reached; Among them, the first sample image required for each round of training includes the specified first negative sample image used in the previous round of training; the specified first negative sample image used in a round of training is: the first negative sample image in which the presence of a head image area and / or a human body image area is detected during the training process.

3. The method according to claim 2, characterized in that The method of detecting the head and the body in each first sample image to be used in this round of training based on the current target detection model to obtain the prediction result of the first sample image includes: Preprocessing each first sample image to obtain a preprocessed first sample image; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.; Input each preprocessed first sample image into the current target detection model to obtain the first parameter, second parameter and position information of each first candidate detection frame in the first sample image; wherein the first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the maximum value of the probability that the image area corresponding to the first candidate detection frame contains a human head and the probability that the image area contains a human body; For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame; The first candidate detection frame whose confidence is greater than the first preset threshold is used as the second candidate detection frame; Based on a non-maximum suppression algorithm, selecting at least one from the second candidate detection frames as a predicted detection frame; For each predicted detection frame, if the second parameter of the predicted detection frame represents the probability that the image region corresponding to the predicted detection frame contains a human head, then determining that the image region corresponding to the predicted detection frame in the first sample image is a human head image region occupied by a human head; If the second parameter of the predicted detection frame indicates the probability that the image area corresponding to the predicted detection frame contains a human body, it is determined that the image area corresponding to the predicted detection frame in the first sample image is a human body image area occupied by a human body.

4. The method according to claim 3, characterized in that: The method detects the head and body in the current video frame to be detected based on the pre-trained target detection model, and obtains the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected, including: Preprocessing the current video frame to be detected to obtain the preprocessed current video frame to be detected; wherein the preprocessing includes at least one of the following processing: denoising, scaling, etc.; The preprocessed current video frame to be detected is input into a pre-trained target detection model to obtain the first parameter, the second parameter and the position information of each first candidate detection frame in the current video frame to be detected; wherein the first parameter of a first candidate detection frame represents the probability that the image area corresponding to the first candidate detection frame is not the background, and the second parameter of a first candidate detection frame represents the maximum value of the probability that the image area corresponding to the first candidate detection frame contains a human head and the probability that the image area contains a human body; For each first candidate detection frame, calculate the product of the first parameter and the second parameter of the first candidate detection frame as the confidence of the first candidate detection frame; The first candidate detection frame whose confidence is greater than the second preset threshold is used as the second candidate detection frame; Based on a non-maximum suppression algorithm, selecting at least one from the second candidate detection frames as a predicted detection frame; For each predicted detection frame, if the second parameter of the predicted detection frame represents the probability that the image region corresponding to the predicted detection frame contains a human head, it is determined that the image region corresponding to the predicted detection frame in the current video frame to be detected is a human head image region occupied by a human head; If the second parameter of the predicted detection frame represents the probability that the image area corresponding to the predicted detection frame contains a human body, it is determined that the image area corresponding to the predicted detection frame in the current video frame to be detected is a human image area occupied by a human body.

5. The method according to claim 1, characterized in that The step of inputting the head image region into a pre-trained classification model to obtain an initial detection result of the head image region includes: The head image region is input into a pre-trained classification model, and features of the head image region are extracted based on a specified network layer in the pre-trained classification model to obtain image features corresponding to the head image region; wherein the specified network layer includes: a convolution layer and a pooling layer; Inputting the image features corresponding to the human head image region into the normalization layer of the trained classification model to obtain the normalized image features corresponding to the human head image region; Inputting the normalized image features corresponding to the human head image region into a cosine classifier in a pre-trained classification model to obtain the cosine similarity between the normalized image features corresponding to the human head image region and the standard image features of each preset behavior; wherein the standard image features of a preset behavior are: determined based on a second sample image representing the preset behavior; If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the head image region and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the head image region and the temperature coefficient is calculated respectively; wherein one target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity; The calculated products are processed by the activation function in the pre-trained classification model to obtain a classification result representing the probability of the presence of each preset behavior of the head in the head image area as the initial detection result of the head image area.

6. The method according to claim 5, characterized in that The training steps for the classification model include: Acquire second sample images and labels of each second sample image; wherein the label of a second sample image indicates a preset behavior of the presence of a human head in the second sample image; Input each second sample image into the classification model of the initial structure, perform feature extraction on the second sample image based on a specified network layer in the classification model of the initial structure, and obtain image features corresponding to the second sample image; Inputting the image features corresponding to the second sample image into the normalization layer of the classification model of the initial structure to obtain the normalized image features corresponding to the human head image region; Inputting the normalized image features corresponding to the second sample image into the cosine classifier in the classification model of the initial structure to obtain the cosine similarity between the normalized image features corresponding to the human head image area and the standard image features of each current preset behavior; If there is at least one target cosine similarity in the cosine similarities between the normalized image features corresponding to the second sample image and the standard image features of each preset behavior, the product of each cosine similarity corresponding to the second sample image and the temperature coefficient is calculated respectively; wherein one target cosine similarity represents: a cosine similarity that is not less than a similarity threshold of the preset behavior corresponding to the target cosine similarity; Processing the calculated products through the activation function in the classification model of the initial structure to obtain a classification result representing the probability of the human head in the second sample image having each preset behavior; Calculating a loss value based on a difference between a classification result of the second sample image and a label of the second sample image; Based on the obtained loss value, the model parameters of the classification model of the initial structure are adjusted until convergence is achieved.

7. The method according to claim 1, characterized in that The determining, based on the initial detection results of other head image areas containing a head in the head image area in at least one video frame before the current head image area in the video to be detected, and the initial detection result of the head image area, a final detection result indicating a preset behavior currently existing in the head image area, comprises: In at least one video frame before the current video to be detected in the video to be detected, the initial detection results of other head image areas containing the head in the head image area, and in the initial detection results of the head image area, if the proportion of initial detection results indicating that the preset behavior of the head in the head image area is a specified preset behavior is greater than a third preset threshold, it is determined that the head in the head image area currently has the specified preset behavior.

8. The method according to claim 1, characterized in that The step of obtaining a current video frame to be detected from the video to be detected includes: Based on the preset sampling interval, a video frame is sampled from the video to be detected in the order of the video frames as the current video frame to be detected.

9. The method according to any one of claims 1 to 8, characterized in that: The preset behaviors of a human head in any human head image area are: wearing a helmet, wearing other items except the helmet, or not wearing anything.

10. A behavior detection device, characterized in that: The device comprises: A video frame acquisition module is used to acquire the current video frame to be detected from the video to be detected; The target detection module is used to detect the head and body in the current video frame to be detected based on the pre-trained target detection model, and obtain the head image area occupied by each head and the body image area occupied by each body in the current video frame to be detected; The overlap coefficient calculation module is used to calculate the overlap coefficient between each detected head image region and each detected human body image region; wherein the overlap coefficient between a head image region and a human body image region represents: the area ratio of the image region where the head image region intersects with the human body image region to the head image region or the human body image region; The classification module is used for inputting the human head image area into a pre-trained classification model to obtain an initial detection result of the human head image area if there is an overlap coefficient indicating that the human head in the human head image area and the human body in the human body image area corresponding to the overlap coefficient belong to the same person; wherein the initial detection result of the human head image area indicates: a preset behavior of the presence of the human head in the human head image area; The behavior detection module is used to determine a final detection result indicating a preset behavior currently existing in the human head in the head image area based on the initial detection results of other head image areas containing a head in the head image area in at least one video frame before the current video to be detected in the video to be detected, and the initial detection result of the head image area.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.

Images