A method and device for identifying human behavior based on multimodal large model

Abstract

The present application relates to the field of computer vision technology, and in particular to a method and device for identifying human behavior based on a multimodal large model. First, based on a first detection small model, determine whether there is a face in each first image frame in the video to be analyzed, and then for each first image frame with a face, determine the state information of the preset object of the face in the first image frame based on the behavior recognition small model. According to the state information corresponding to each first image frame with a face and a first quantity threshold, determine whether there is a violation in the video to be analyzed. Only when the number of first image frames with a certain state information meets the requirement of the first quantity threshold, it is determined that there is a violation in the video to be analyzed, and output a prompt message indicating the existence of the violation, thereby improving the accuracy of human behavior recognition.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for identifying human behavior based on a multimodal large model. Background Art

[0002] The existing personnel behavior analysis technology on the market mainly relies on a small model of a single convolutional neural network (CNN) architecture to analyze a single static image. However, personnel behavior analysis based only on a single static image often fails to accurately capture the dynamic changes in personnel behavior, resulting in frequent false detections and missed detections.

[0003] Therefore, how to improve the accuracy of personnel behavior recognition has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method and device for identifying human behavior based on a multimodal large model, so as to solve the problem of low accuracy of human behavior identification in the prior art.

[0005] The present application provides a method for identifying human behavior based on a multimodal large model, the method comprising:

[0006] Determine whether there is a human face in each first image frame in the video to be analyzed based on the first detection model;

[0007] For each first image frame containing a human face, determining state information of a preset object of the human face in the first image frame based on the behavior recognition model;

[0008] According to the state information corresponding to each first image frame with a human face and the first quantity threshold, it is determined whether there is any violation in the video to be analyzed, and if so, prompt information indicating the existence of the violation is output.

[0009] The above technical scheme first determines whether there is a human face in each first image frame in the video to be analyzed based on the first detection small model, and then determines the status information of the preset object of the human face in each first image frame with a human face based on the behavior recognition small model. According to the status information corresponding to each first image frame with a human face and the first quantity threshold, it is determined whether there is any violation in the video to be analyzed. Only when the number of first image frames with certain status information meets the requirement of the first quantity threshold, it is determined that there is a violation in the video to be analyzed, and a prompt information of the existence of violation is output, thereby improving the accuracy of personnel behavior recognition.

[0010] Furthermore, before determining whether a human face exists in each first image frame in the video to be analyzed based on the first detection model, the method further includes:

[0011] Determine whether there is a human body in each image frame in the video to be analyzed based on the second detection model;

[0012] Determine the image frame in which the human body exists as the first image frame;

[0013] The determining whether there is a human face in each first image frame in the video to be analyzed based on the first detection model includes:

[0014] Based on the first detection model, determine whether there is a face in the detection frame corresponding to the human body in each first image frame.

[0015] The above technical solution, before performing face recognition, also identifies whether there is a human body in each image frame in the video to be analyzed, and subsequently only performs face recognition on the first image frame where a human body exists. The recognition is performed in a hierarchical manner and the corresponding expert model is used for recognition, which can further improve the accuracy of human behavior recognition.

[0016] Furthermore, the method further comprises:

[0017] If the number of consecutive image frames without human bodies is greater than a second number threshold, a violation prompt message indicating that the person has left his post is output.

[0018] The above technical solution can accurately identify the absence of work behavior by identifying whether there is a human body in the image frame to determine whether there is an absence of work behavior in the video to be analyzed.

[0019] Furthermore, the method further comprises:

[0020] If the number of consecutive first image frames without human faces is greater than a third number threshold, a prompt message of human occlusion is output.

[0021] In the above technical solution, if no face is recognized when recognizing a face, it means that occlusion may have occurred. Through this method, the occlusion can be accurately identified.

[0022] Furthermore, after determining whether a human body exists in each image frame in the video to be analyzed based on the second detection model, and before determining the image frame in which a human body exists as the first image frame, the method further includes:

[0023] For each image frame in which a human body exists, an area threshold is determined based on the width, height and a first preset ratio of the image frame; if the area of ​​the detection frame corresponding to the human body in the image frame is smaller than the area threshold, the detection frame corresponding to the human body in the image frame is deleted.

[0024] The above technical solution, based on the principle that objects are larger when they are closer and smaller when they are farther away, deletes the detection frame of the human body that is too small in the image frame, which can effectively improve the accuracy of human behavior recognition, because if the human body image is too small, it is impossible to recognize the facial details well.

[0025] Furthermore, after determining whether a human body exists in each image frame in the video to be analyzed based on the second detection model, and before determining the image frame in which a human body exists as the first image frame, the method further includes:

[0026] For each image frame in which a human body exists, a central image area corresponding to the image frame is determined based on the width of the image frame and a second preset ratio; a center point of a detection frame corresponding to the human body in the image frame is determined, and a detection frame of a human body whose center point does not exist in the central image area corresponding to the detection frame is deleted.

[0027] The above technical solution removes the objects located on the left and right sides of the image and eliminates the interference of people on the left and right sides.

[0028] Furthermore, the method further comprises:

[0029] If there are multiple human bodies in any image frame in which a human body exists, for each human body, a weight value corresponding to the human body is determined according to the width and height of the detection frame corresponding to the human body, and the width and height of the image frame;

[0030] Delete the detection box corresponding to the human body with non-maximum weight value.

[0031] In the above technical solution, if there are multiple human bodies in any image frame containing a human body, the best target is selected by determining the weight value corresponding to each human body. On the one hand, the target should be large, and on the other hand, the target should be located closest to the center of the picture, which further improves the accuracy of human behavior recognition.

[0032] Furthermore, before determining whether a human face exists in each first image frame in the video to be analyzed based on the first detection model, the method further includes:

[0033] Acquire a preset number of image frames in the video to be analyzed as preprocessing image frames;

[0034] Based on the light detection model, determine whether there is a backlight phenomenon in the pre-processed image frame. If so, output a prompt message that the light of the image frame to be analyzed is abnormal, and end the personnel behavior recognition.

[0035] The above technical solution uses a small model based on light detection to determine whether there is a backlight phenomenon in the preprocessed image frame, and does not identify the video with the backlight phenomenon. The accuracy of personnel behavior recognition is further improved through preliminary data screening.

[0036] Furthermore, after determining that there is a violation in the video to be analyzed and before outputting a prompt message indicating that there is a violation, the method further includes:

[0037] Acquire abnormal image frames containing the illegal behavior in the video to be analyzed;

[0038] If it is determined based on the multimodal large model that the illegal behavior exists in the abnormal image frame, then the subsequent step of outputting prompt information indicating the existence of the illegal behavior is continued.

[0039] The above technical solution can more accurately understand and analyze human behavior by re-examining abnormal image frames with violations through a multimodal large model, thereby improving the accuracy and efficiency of human behavior recognition.

[0040] The present application provides a human behavior recognition device based on a multimodal large model, the device comprising:

[0041] A face detection module, used for determining whether there is a face in each first image frame in the video to be analyzed based on the first detection model;

[0042] A behavior detection module, for determining, for each first image frame containing a human face, state information of a preset object of the human face in the first image frame based on a small behavior recognition model;

[0043] The confirmation module is used to determine whether there is any violation in the video to be analyzed based on the status information corresponding to each first image frame with a face and the first quantity threshold, and if so, output a prompt message indicating the existence of the violation.

[0044] Furthermore, the device also includes:

[0045] A human body detection module, used for determining whether there is a human body in each image frame in the video to be analyzed based on a second detection model; and determining an image frame in which a human body exists as the first image frame;

[0046] The face detection module is specifically used to determine whether there is a face in the detection frame corresponding to the human body in each first image frame based on the first detection model.

[0047] Furthermore, the confirmation module is also used to output a violation prompt message indicating that the person has left his post if the number of consecutive image frames in which no human body exists is greater than a second number threshold.

[0048] Furthermore, the confirmation module is also used to output a prompt message of human occlusion if the number of consecutive first image frames without human faces is greater than a third number threshold.

[0049] Furthermore, the device also includes:

[0050] The screening module is used to determine an area threshold for each image frame containing a human body according to the width, height and a first preset ratio of the image frame; if the area of ​​the detection frame corresponding to the human body in the image frame is smaller than the area threshold, the detection frame corresponding to the human body in the image frame is deleted.

[0051] Furthermore, the screening module is also used to determine, for each image frame in which a human body exists, a central image area corresponding to the image frame according to the width of the image frame and a second preset ratio; determine the center point of the detection frame corresponding to the human body in the image frame, and delete the detection frame of the human body whose center point does not exist in the central image area corresponding to the central image area.

[0052] Furthermore, the screening module is also used to determine, if there are multiple human bodies in any image frame in which a human body exists, a weight value corresponding to each human body based on the width and height of the detection frame corresponding to the human body, and the width and height of the image frame; and delete the detection frame corresponding to the human body with a non-maximum weight value.

[0053] Furthermore, the screening module is also used to obtain a preset number of image frames in the video to be analyzed as pre-processed image frames; based on the light detection small model, determine whether there is a backlight phenomenon in the pre-processed image frames. If so, output a prompt message of the light abnormality of the image frame to be analyzed, and end the personnel behavior recognition.

[0054] Furthermore, the device also includes:

[0055] A recheck module, used for obtaining abnormal image frames containing the illegal behavior in the video to be analyzed;

[0056] If it is determined based on the multimodal large model that the illegal behavior exists in the abnormal image frame, then the subsequent step of outputting prompt information indicating the existence of the illegal behavior is continued.

[0057] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned methods for identifying personnel behavior based on a multimodal large model when executing a computer program stored in a memory.

[0058] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for identifying personnel behavior based on a multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A schematic diagram of a process of identifying human behavior based on a multimodal large model provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of a personnel behavior recognition process provided in an embodiment of the present application;

[0062] Figure 3a A schematic diagram of a human behavior recognition result page provided in an embodiment of the present application;

[0063] Figure 3b A schematic diagram of a human behavior recognition result details page provided in an embodiment of the present application;

[0064] Figure 3c A schematic diagram of a manual review page for human behavior recognition results provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the structure of a personnel behavior recognition device based on a multimodal large model provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0068] In the field of modern intelligent monitoring and security, personnel behavior analysis is a crucial research direction. Traditional CNN-based models perform well in image recognition and classification, but in practical applications, especially when it comes to video analysis, there are inherent limitations. These limitations are mainly manifested in the large number of false detections and missed detections, which affect the accuracy and reliability of personnel behavior recognition. In order to overcome these limitations, the embodiment of the present application provides a personnel behavior recognition method and device based on a multimodal large model, in which the method determines whether there is a face in each first image frame in the video to be analyzed based on a first detection small model; for each first image frame with a face, the state information of the preset object of the face in the first image frame is determined based on the behavior recognition small model; according to the state information corresponding to each first image frame with a face and the first quantity threshold, it is determined whether there is a violation in the video to be analyzed, and if so, a prompt message of the violation is output.

[0069] Figure 1 A flowchart of a personnel behavior recognition process based on a multimodal large model is provided in an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0070] S101: Determine whether there is a human face in each first image frame in the video to be analyzed based on the first detection model.

[0071] The personnel behavior recognition method based on a multimodal large model provided in the present application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0072] When identifying the behavior of a person, in an embodiment of the present application, it can be determined whether there is a face in each first image frame in the video to be analyzed based on the first detection model. The video to be analyzed can be a video captured by an image acquisition device installed at a preset position, wherein the preset position can be a desk, a classroom, an entrance, and the like. Exemplarily, in order to monitor the working status of the staff, an image acquisition device can be installed on the desk, and the image acquisition device can capture the status of the corresponding staff at work. By analyzing the video, it can be determined whether the staff has any illegal behavior, such as leaving the post, sleeping on the post, smoking, making phone calls, etc. The video to be analyzed can be an offline video or an online video.

[0073] In the embodiment of the present application, each image frame in the video to be analyzed can be determined as the first image frame, or some image frames in the video can be determined as the first image frame. Exemplarily, an image frame can be determined as the first image frame at every set time length in the video to be analyzed, and the set time length can be 1 second, 5 seconds, 10 seconds, etc.

[0074] In the embodiment of the present application, the first detection model can be any model with face detection function. Optionally, the first detection model can be a Fast MTCNN (Multi-task Cascaded Convolutional Networks) algorithm. Fast MTCNN is an efficient and accurate face detection method, and its main process is as follows:

[0075] Image pyramid construction: First, an image pyramid is constructed for the input image to detect faces at different scales.

[0076] Candidate window generation (P-Net): The first convolutional neural network (Proposal Network, P-Net) generates candidate windows at each layer of the image pyramid and performs preliminary screening. P-Net outputs the bounding box regression and face confidence of the candidate window.

[0077] Bounding box calibration and non-maximum suppression (NMS): The candidate windows generated by P-Net are calibrated, and non-maximum suppression technology is used to remove candidate windows with high overlap and retain windows with higher confidence.

[0078] Refinement screening (R-Net): The candidate windows that have been initially screened are input into the second convolutional neural network (RefineNetwork, R-Net) for further refinement screening. R-Net outputs more accurate bounding box regression and face confidence.

[0079] Recalibration and non-maximum suppression (NMS): The candidate windows generated by R-Net are recalibrated and non-maximum suppression technology is used to remove candidate windows with high overlap.

[0080] Final detection (O-Net): The carefully screened candidate windows are input into the third convolutional neural network (Output Network, O-Net) for final face detection and key point location. O-Net outputs the final face bounding box and the coordinates of the five key points (eyes, nose and mouth).

[0081] Output detection results: Finally output the detected face bounding box and its key point information.

[0082] The Fast MTCNN algorithm achieves efficient and accurate face detection and key point location through the architecture of multi-task cascade convolutional networks. This feature provides reliable support as the basic face detection module in the personnel behavior analysis of this patent.

[0083] Of course, those skilled in the art may use other first detection small models for face recognition.

[0084] S102: For each first image frame containing a human face, determine state information of a preset object of the human face in the first image frame based on a small behavior recognition model.

[0085] If the first detection model detects a face, it means that there is a person in the corresponding first image frame. In order to determine the current state of the person, in the embodiment of the present application, for each first image frame with a face, the state information of the preset object of the face in the first image frame can be determined based on the behavior recognition model. The preset object can be eyes, mouth, eyebrows, etc. The state information of the preset object can be eyes open, eyes closed, mouth open, mouth closed, etc.

[0086] The behavior recognition model can be any model that has the ability to recognize the state of a preset object. In the embodiment of the present application, a multi-label recognition algorithm can be used to recognize the state of the eyes and mouth of the face. In the embodiment of the present application, a multi-label classification algorithm based on deep learning is first introduced, and its main process is as follows:

[0087] Data labeling: Classify the attribute labels. The attribute labels have two digits. The first digit is the eye state: 0 for open eyes and 1 for closed eyes. The second digit is the mouth state: 0 for closed mouth and 1 for open mouth. So there are four types of attribute labels: 00, 01, 10, 11. Among them, 00 means open eyes and closed mouth, 01 means open eyes and open mouth, 10 means closed eyes and closed mouth, and 11 means closed eyes and open mouth.

[0088] Input image preprocessing: The input face image is preprocessed, including image scaling, normalization, and data augmentation, to meet the input requirements of the model.

[0089] Feature extraction: Extract the features of face images through convolutional neural networks (CNN). Commonly used network architectures include ResNet, VGG, etc. These networks can capture the detailed features of face images.

[0090] Multi-task learning: Based on feature extraction, multi-task learning is used to build multiple parallel classifiers, each of which is responsible for identifying a specific attribute. For example, one classifier is used to identify gender, another classifier is used to identify age, and so on.

[0091] Loss function design: In order to optimize multiple classification tasks at the same time, a multi-task loss function is designed. A common practice is to weight the cross entropy loss of each task, and the weight can be adjusted according to the importance of the task.

[0092] Model training: Use a face dataset with various attributes to train the model. Use the back propagation algorithm to optimize the network parameters so that it can accurately identify various facial attributes.

[0093] Prediction and output: In the inference phase, the preprocessed face image is input into the model, and the model outputs the prediction results of multiple attributes. The prediction result category probability of each attribute.

[0094] This multi-label recognition algorithm based on multi-task learning can efficiently recognize multiple attributes of a face, and based on a shared feature extraction network, it reduces computational overhead and improves recognition accuracy and real-time performance. This type of behavior recognition model provides comprehensive and reliable support for human behavior recognition.

[0095] S103: Determine whether there is any violation in the video to be analyzed based on the state information corresponding to each first image frame with a face and the first quantity threshold, and if so, output prompt information indicating the violation.

[0096] When a person makes an action, it usually lasts for a certain period of time, for example, yawning lasts for about ten seconds, and sleeping lasts for a longer period of time. Therefore, in the embodiment of the present application, it is possible to determine whether there is any violation in the video to be analyzed based on the state information corresponding to each first image frame with a human face and the first quantity threshold.

[0097] Specifically, the number of first image frames in which the eyes are closed continuously appears in the first image frame with a face. If the number is greater than a first threshold, it means that the eyes are closed for a long time, and it can be considered that there is a violation in the video to be analyzed.

[0098] After determining that there is a violation in the video to be analyzed, a prompt message indicating the violation may be output.

[0099] In a possible implementation, in order to make it clear to the user what violations have occurred, in an embodiment of the present application, when outputting the prompt information, any first image frame containing the violation can also be output for user preview.

[0100] In the embodiment of the present application, it is first determined based on the first detection small model whether there is a human face in each first image frame in the video to be analyzed, and then for each first image frame with a human face, the state information of the preset object of the human face in the first image frame is determined based on the behavior recognition small model. According to the state information corresponding to each first image frame with a human face and the first quantity threshold, it is determined whether there is a violation in the video to be analyzed. Only when the number of first image frames with a certain state information meets the requirement of the first quantity threshold, it is determined that there is a violation in the video to be analyzed, and a prompt information of the existence of the violation is output, thereby improving the accuracy of personnel behavior recognition.

[0101] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, before determining whether there is a face in each first image frame in the video to be analyzed based on the first detection model, the method also includes:

[0102] Determine whether there is a human body in each image frame in the video to be analyzed based on the second detection model;

[0103] Determine the image frame in which the human body exists as the first image frame;

[0104] The determining whether there is a human face in each first image frame in the video to be analyzed based on the first detection model includes:

[0105] Based on the first detection model, determine whether there is a face in the detection frame corresponding to the human body in each first image frame.

[0106] In the embodiment of the present application, before determining whether there is a face in each first image frame in the video to be analyzed based on the first detection model, it is also possible to determine whether there is a human body in each image frame in the video to be analyzed based on the second detection model. The image frame with a human body is determined as the first image frame. In other words, human body detection is performed on each image frame in the video to be analyzed, and after the human body is detected, face detection is performed.

[0107] The second detection model can be any model with human body recognition function, and technicians in this field can configure a suitable model as needed. Exemplarily, the second detection model can be a YOLO (You Only Look Once) series algorithm. The YOLO series algorithm is an efficient target detection method, and its main process is as follows:

[0108] Input image preprocessing: Scale the input image to the specified size and normalize it to meet the input requirements of the YOLO model.

[0109] Feature extraction: Extract image features through convolutional neural networks (CNN). The YOLO series of algorithms use deep convolutional networks, such as Darknet, to extract feature information from images.

[0110] Grid partitioning: The image is divided into SxS grids, each grid is responsible for detecting one or more objects.

[0111] Bounding box prediction: Each grid predicts multiple bounding boxes and outputs the coordinates, confidence, and category probability of each bounding box. The confidence indicates the probability that the bounding box contains an object, and the category probability indicates the probability that the object belongs to a certain category.

[0112] Non-maximum suppression (NMS): In order to reduce redundant detection results, non-maximum suppression technology is used to retain the bounding box with the highest confidence and remove other bounding boxes with high overlap.

[0113] Output detection results: Finally output the detected human body bounding box and its category information.

[0114] When performing face detection on image frames containing human bodies, in an embodiment of the present application, it can be determined based on the first detection model whether there is a face in the detection frame corresponding to the human body in each first image frame.

[0115] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0116] If the number of consecutive image frames without human bodies is greater than a second number threshold, a violation prompt message indicating that the person has left his post is output.

[0117] According to common sense, the relevant personnel are generally at their workstations during work. Therefore, in the embodiment of the present application, the number of consecutive image frames without human bodies can be determined. If the number is greater than the second number threshold, it can be considered that the corresponding personnel are not at their workstations, and a prompt message indicating that the personnel have left their posts can be output.

[0118] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, the method further includes:

[0119] If the number of consecutive first image frames without human faces is greater than a third number threshold, a prompt message of human occlusion is output.

[0120] If the presence of a human body is detected based on the second detection model, but no human face is detected when the first detection model is used for face detection, it means that the face of the corresponding person is blocked. In the embodiment of the present application, the number of consecutive first image frames without human faces can be determined. If the number is greater than the third number threshold, it means that the face of the corresponding person is blocked, and a prompt message of person blocking can be output.

[0121] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, after determining whether there is a human body in each image frame in the video to be analyzed based on the second detection model, and before determining the image frame with a human body as the first image frame, the method further includes:

[0122] For each image frame in which a human body exists, an area threshold is determined based on the width, height and a first preset ratio of the image frame; if the area of ​​the detection frame corresponding to the human body in the image frame is smaller than the area threshold, the detection frame corresponding to the human body in the image frame is deleted.

[0123] In actual application scenarios, in images of personnel on duty, the monitored personnel are generally closer to the image acquisition device, and the human body occupies a relatively large proportion in the picture. Therefore, in the embodiment of the present application, after determining whether there is a human body in each image frame, the area threshold can be determined for each image frame with a human body according to the width, height and first preset ratio of the image frame.

[0124] Specifically, when determining the area threshold, the product of the width and height of the image frame may be determined, and then the product is multiplied by a first preset ratio to obtain the area threshold, wherein the first preset ratio may be any decimal less than 1, for example, the first preset ratio is 0.1, 0.25, etc.

[0125] After determining the area threshold, the area of ​​the detection frame corresponding to the human body in the image frame can be determined. If the area is less than the area threshold, it means that the human body occupies a small proportion in the picture and may not be a person who needs to be monitored. Therefore, in the embodiment of the present application, the detection frame corresponding to the human body can be deleted in the image frame. If the detection frame corresponding to the human body is deleted and there are no other detection frames in the image frame, it means that there is no content that needs to be further identified in the image frame, and the image frame can be marked as not containing a human body, and face recognition will no longer be performed on the image frame in the future.

[0126] Specifically, when removing the detection boxes, the detection boxes with S (area) = w1 * h1 < W * H * T1 can be removed. Wherein, w1 represents the width corresponding to the detection box of the human body, h1 represents the height corresponding to the detection box of the human body, W represents the width of the image frame, H represents the height of the image frame, and T1 represents the first preset ratio.

[0127] To further improve the accuracy of human behavior recognition, based on the above embodiments, in the embodiments of the present application, after determining whether there is a human body in each image frame of the video to be analyzed based on the second detection sub-model, and before determining the image frame with a human body as the first image frame, the method further includes:

[0128] For each image frame with a human body, according to the width of the image frame and the second preset ratio, determine the central image area corresponding to the image frame; determine the center point of the detection box corresponding to the human body in the image frame, and delete the detection box of the human body whose center point does not exist in the central image area.

[0129] In the embodiments of the present application, it is only to detect whether the monitored person has any illegal behavior, but in the actual scenario, it is very easy to have a situation where there are other people in the picture besides the monitored person. To improve the accuracy of human behavior recognition, in the embodiments of the present application, the detection boxes corresponding to the human bodies on the left and right sides of the picture can be removed, and the subsequent detection of this person will no longer be performed, excluding the interference of other people on the left and right sides.

[0130] In the embodiments of the present application, after determining whether there is a human body in each image frame, for each image frame with a human body, according to the width of the image frame and the second preset ratio, the central image area corresponding to the image frame can also be determined.

[0131] Specifically, assuming the second preset ratio is 20%, then the area between 20% of the width of the image frame and (1 - 20%) of the width can be determined as the central image area.

[0132] After determining the central image area of the image frame, the center point of the detection box corresponding to the human body in the image frame can be determined. If the center point of the detection box does not lie in the central image area, the corresponding detection box can be deleted.

[0133] Specifically, if the coordinates of the center point of a certain detection box satisfy the following formula, it can be considered that the center point of the detection box is located in the central image area, and the detection box can not be deleted.

[0134] W * T2 < x1center < W * (1 - T2)

[0135] Wherein, W represents the width of the image frame; T2 represents the second preset ratio; X1 represents the width of the detection frame corresponding to the human body in the image frame, and x1center represents the center point of the width of the detection frame.

[0136] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, the method further includes:

[0137] If there are multiple human bodies in any image frame in which a human body exists, for each human body, a weight value corresponding to the human body is determined according to the width and height of the detection frame corresponding to the human body, and the width and height of the image frame;

[0138] Delete the detection box corresponding to the human body with non-maximum weight value.

[0139] If there are multiple human bodies in an image frame containing a human body, in order to select the best target, the target should be large and located closest to the center of the picture. In the embodiment of the present application, for each human body in the image frame, the weight value corresponding to the human body can be determined according to the width and height of the detection frame corresponding to the human body and the width and height of the image frame.

[0140] Specifically, the weight value corresponding to each human body can be calculated based on the following formula:

[0141]

[0142] Among them, Value represents the weight value corresponding to a human body in the image frame; w1 represents the width of the detection frame corresponding to the human body; h1 represents the height of the detection frame corresponding to the human body; X1center represents the horizontal coordinate of the midpoint of the detection frame corresponding to the human body; y1center represents the vertical coordinate of the midpoint of the detection frame corresponding to the human body; Xcenter represents the horizontal coordinate of the midpoint of the image frame; Ycenter represents the vertical coordinate of the midpoint of the image frame.

[0143] After obtaining the weight value corresponding to each human body in the image frame, the human body corresponding to the maximum weight value can be determined as the best target, and the behavior detection of the human body is subsequently performed. In the embodiment of the present application, the detection frame corresponding to the human body with non-maximum weight value can be deleted in the image frame.

[0144] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, before determining whether there is a face in each first image frame in the video to be analyzed based on the first detection model, the method also includes:

[0145] Acquire a preset number of image frames in the video to be analyzed as preprocessing image frames;

[0146] Based on the light detection model, determine whether there is a backlight phenomenon in the pre-processed image frame. If so, output a prompt message that the light of the image frame to be analyzed is abnormal, and end the personnel behavior recognition.

[0147] In actual application scenarios, poor video quality will directly affect the recognition effect. In order to avoid misrecognition caused by data quality issues, in an embodiment of the present application, the quality of the video can be effectively monitored in advance.

[0148] In an embodiment of the present application, a preset number of image frames can be obtained from the video to be analyzed as preprocessed image frames. Exemplarily, all image frames in the video to be analyzed can be determined as preprocessed image frames, or the first preset number of image frames in the video to be analyzed can be determined as preprocessed image frames, or a preset number of image frames can be selected from the video to be analyzed at a preset time interval as preprocessed image frames.

[0149] After obtaining the preprocessed image frame, it can be determined whether there is a backlight phenomenon in the preprocessed image frame based on the light detection model. If so, a prompt message indicating that the light of the image frame to be analyzed is abnormal is output, and the personnel behavior recognition is terminated. In an embodiment of the present application, the light detection model can be a classification model, and each image frame is input into the model, and the model determines whether the light in the image frame is normal light or backlight.

[0150] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, the determination of whether there is any violation in the video to be analyzed according to the state information corresponding to each first image frame with a face and the first quantity threshold includes:

[0151] If the number of first image frames in which the eyes are continuously closed is greater than the first number threshold, it is determined that there is a sleeping behavior in the video to be analyzed;

[0152] If the number of first image frames in which the mouth is continuously opened is greater than the first number threshold, it is determined that yawning behavior exists in the video to be analyzed.

[0153] In order to determine what specific violation a person has committed, in an embodiment of the present application, when determining whether there is a violation in the video to be analyzed based on the status information corresponding to each first image frame with a face and the first quantity threshold, the number of first image frames in which the eyes are continuously closed can be judged. If the number is greater than the first quantity threshold, it can be determined that there is sleeping on the post in the video to be analyzed.

[0154] In the embodiment of the present application, the number of first image frames in which the mouth is continuously open can also be determined. If the number is greater than a first number threshold, it can be determined that there is post-sleeping behavior in the video to be analyzed.

[0155] In order to further improve the accuracy of personnel behavior recognition, based on the above embodiment, in the embodiment of the present application, after determining that there is a violation in the video to be analyzed, before outputting a prompt message indicating the presence of the violation, the method further includes:

[0156] Acquire abnormal image frames containing the illegal behavior in the video to be analyzed;

[0157] If it is determined based on the multimodal large model that the illegal behavior exists in the abnormal image frame, then the subsequent step of outputting prompt information indicating the existence of the illegal behavior is continued.

[0158] Although the algorithm flow based on the CNN architecture seems to be relatively complicated, the processing speed of several small models in series can be maintained at about 0.25s per frame. Although the processing speed meets the requirements, the inherent limitations of the small model cannot be solved by the small model alone. For example, when the monitored person lowers his head to write or read documents, he may be misjudged as closed-eye state because the small model cannot correctly recognize the state information of the eyes. For another example, if the monitored person is talking on a handheld walkie-talkie, there may be a problem of facial occlusion, or when the monitored person is operating a computer with his head supported by his hand, there may also be a problem of facial occlusion. For these scenes, the state of the face cannot be captured, and these scenes are frequent in actual scenes. In the face of these scenes, the face cannot be detected at all, and the state information corresponding to the preset face cannot be recognized. This is the inherent limitation of the small model. In addition, in the scene of mouth state information recognition, the small model can only recognize the static state of the mouth opening, but many mouth opening states are not yawning. For example, when the monitored person is talking, because his mouth is open, it is easy to misjudge it as yawning. Due to the limitation of small models, it can only detect the static state of the mouth and cannot understand the dynamic process of yawning. Due to the above limitations, the recognition effect of actual application scenarios is not ideal and users are dissatisfied.

[0159] Based on the limitations of the above-mentioned CNN algorithm, it can be seen that the importance of timing information and the understanding of the process before and after the event can effectively reduce the situation of misjudgment and missed detection. By introducing timing information and combining the efficiency of the CNN architecture, the accuracy and efficiency of personnel behavior recognition can be significantly improved. In an embodiment of the present application, after determining that there is a violation in the video to be analyzed, before outputting the prompt information of the violation, a re-inspection can be performed based on a large multimodal model.

[0160] In an embodiment of the present application, abnormal image frames containing illegal behaviors can be obtained in the video to be analyzed. Specifically, assuming that it is determined that there is off-duty behavior in the video to be analyzed, all image frames that do not include human bodies can be determined as abnormal image frames, or a set number of image frames that do not include human bodies can be randomly selected as abnormal image frames.

[0161] After the abnormal image frame is acquired, the abnormal image frame can be input into the multimodal large model, and the multimodal large model determines whether there is any violation in the abnormal image frame. If the multimodal large model determines that the violation exists in the abnormal image frame, the recognition result can be considered correct, and the subsequent step of outputting a prompt message indicating the existence of the violation can be continued.

[0162] The current capabilities of the multimodal large model can indeed solve the problem of understanding various scenes. The small model lacks the understanding of time series information. The small model can only understand based on a static picture, but many scenes or behavioral actions require a series of actions to determine whether the event occurs, such as yawning. The multimodal large model can accurately understand scenes such as occlusion and bowing, but the processing speed and the hardware cost of algorithm deployment are extremely high, so we also take advantage of the advantages of the efficiency of the small model. Compared with the prior art, the embodiment of the present application can more accurately capture the dynamic changes of behavior and reduce false detection and missed detection by introducing time series information. The accuracy and reliability of personnel behavior recognition are significantly improved. In addition, the efficiency of the CNN architecture is combined to solve the time-consuming problem of the multimodal large model in time series understanding, and efficient and accurate personnel behavior analysis is achieved. Although the traditional CNN architecture has limitations in some aspects, its efficiency cannot be ignored. The present application combines the efficiency of CNN with the understanding of time series information, which significantly improves the processing speed of the system. Through experiments on different types of graphics processing units (GPUs), the processing time of a small model in the personnel behavior recognition process provided by this application is about 0.1 seconds, and the processing time of a multimodal large model for an image is about 2 seconds. This application combines time series information, image information and text guidance to analyze and identify personnel behavior, accurately analyze personnel dynamic behavior, effectively solve the misjudgment of static scenes, and realize the scene expansion and algorithm extension of personnel behavior analysis.

[0163] In order to further improve the accuracy of personnel behavior recognition, in the embodiment of the present application, the multimodal large model for re-examination can be a model obtained by fine-tuning training. In the embodiment of the present application, fine-tuning training can be performed based on the instruction data set.

[0164] When constructing an instruction data set, corresponding sample videos can be collected according to the violations that need to be identified. After obtaining the sample video, a preset number of sample image frames can be extracted from the sample video. In an embodiment of the present application, different frame extraction strategies can be set for different violations. Exemplarily, for screen occlusion, leaving the post, and sleeping on the post, which are all continuous events with stable images, the frame extraction strategy can be key frame extraction or uniform frame extraction, and the number of frames extracted ranges from 4 to 10 frames. For fluctuating events, such as yawning, it is necessary to capture the changing process of the action, extract key frames, and obtain the detailed process of the event.

[0165] After obtaining each sample image frame, the sample image frame can be annotated. The existing large models have good performance in recognizing human attributes and actions, but they are still prone to misjudgment in some scenes. We can use GPT4o to perform initial data annotation, and then use manual verification to refine the annotation results to ensure data quality.

[0166] Some scenarios require algorithms to identify multiple behaviors or attributes, and require a multimodal large model to have the ability to identify multiple behaviors, and to determine whether there are behaviors or attributes related to the user through one input. In this case, the multimodal large model is required to have the ability to make multiple selections. In addition, while identifying the behavior of personnel, the multimodal large model must be able to perceive the people and objects in some duty scenes in the picture, and then recognize the relationship between them and analyze the behavior after perceiving the target. And frame the location of the personnel with the above specific behaviors. In other words, the multimodal large model is required to have the ability of multiple selections, scene description, task detection, retrieval, and reference generation. Based on this, in order to enable the multimodal large model to have the above capabilities, several prompt texts are designed in the embodiments of the present application, and the following instruction set can be constructed based on these prompt texts and sample image frames.

[0167] A. Multiple choice:

[0168] Prompt text: Whether the staff member has any of the following behaviors: a) sleeping, b) yawning, c) normal duty status, d) blocking the screen, e) not at work.

[0169] Answer from the multimodal large model: The staff member exhibited the following behaviors: b) yawning.

[0170] Prompt text: Check whether the staff member wearing a red vest in the picture is engaging in the following behaviors: a) sleeping, b) yawning, c) normal duty status, d) blocking the picture, e) not at the post.

[0171] The answer of the multimodal large model: The staff in red vests have the following behaviors: a) Sleeping.

[0172] B. Scene description:

[0173] Prompt text: Describe what the workers in this video are doing?

[0174] Prompt text: What is the on-duty personnel in the picture doing?

[0175] C. Detection tasks:

[0176] Hint text: Frame the sleeping person in the picture.

[0177] Answers from multimodal large models: coordinates of detection boxes [24, 56, 47, 89].

[0178] Hint text: Frame the yawning person in the picture.

[0179] Answers from multimodal large models: coordinates of detection boxes [24, 146, 47, 129].

[0180] D. Search:

[0181] Prompt text: Image1, image2, image3. Which on-duty personnel in the above images are sleeping?

[0182] Prompt text: Image1, image2, image3. Which staff members are yawning in the above images?

[0183] Prompt text: Image1, image2, image3. Which on-duty personnel in the above images have left their posts?

[0184] The above-mentioned Image1, image2, and image3 represent three sample image frames.

[0185] E. Reference generation:

[0186] Hint text: What is the worker at [24,56,47,89] doing in the picture? Describe it.

[0187] Hint text: What is the red vested officer [24,56,47,89] doing in the picture? Describe it.

[0188] The multimodal large model selects a model that can support video tasks or multi-frame data input. The labeled data is used for model fine-tuning training.

[0189] The following is a description of the personnel behavior recognition process with reference to a specific embodiment. Figure 2 A flow chart of a personnel behavior recognition process provided in an embodiment of the present application is as follows: Figure 2As shown, after obtaining the video to be analyzed, the video to be analyzed can be subjected to frame extraction. Human body detection is performed on the extracted image frames, and target filtering is performed on the image frames with human bodies. The target filtering process is the process of determining whether to delete the corresponding detection frame according to the area of ​​the detection frame of the human body as described in the above embodiments.

[0190] After the target filtering is completed, it is determined whether there is a human body in each image frame. If there is no human body, the cumulative time of no human body is recorded. If the cumulative time meets the preset time threshold, it means that there is an off-duty event in the video to be detected. In other words, when no human target that meets the conditions is detected in multiple consecutive frames, and the cumulative time meets the preset time threshold, it is considered that an off-duty event has occurred. An event picture can be saved, and the duration of the event can be continued to be detected. When the event ends, the start and end time of the time are saved.

[0191] For image frames with human bodies, face detection continues. If there is no face in the image frame, the cumulative time without face is recorded. If the cumulative time meets the preset time threshold, the multimodal large model is called for re-judgment, so as to determine whether there is an occlusion event based on the re-judgment result of the multimodal model. In other words, when a human frame that meets the conditions is detected, but no face information is detected, and the cumulative time meets the preset time threshold, it is considered that a human occlusion event has occurred, and an event picture is saved. The duration of the event continues to be detected. When the event ends, the start and end time of the time are saved.

[0192] Continue to perform behavior recognition for image frames with faces. After identifying the state information of each preset object of the face, determine whether the cumulative time of the state reaches the preset time threshold for different state information. When the time threshold is met, use the multimodal large model for re-judgment to determine whether there is any violation in the video to be detected. Among them, the violation determined based on the state information of the preset object can be yawning, sleeping, and closing eyes. That is to say, when both the human body frame and the face frame can be detected, the state information of the preset object of the face is identified to obtain the results of whether the eyes are closed and whether the mouth is open for yawning. If the cumulative eye-closing time meets the preset time threshold, it is considered that a sleeping event has occurred, an event picture is saved, and the duration of the event is continued to be detected. The event ends and the start and end time of the time is saved. If the cumulative eye-closing time meets the preset time threshold, it is considered that a yawning event has occurred, an event picture is saved, and the duration of the event is continued to be detected. The event ends and the start and end time of the time is saved.

[0193] Since the duration of different actions is different, in the embodiment of the present application, different time thresholds are set for different violations. If the number threshold of image frames is set for different violations, then the corresponding time threshold can be determined according to the number threshold and the frame rate of the video to be analyzed. After determining that there is a violation in the video to be analyzed, it can continue to determine whether all of the video to be analyzed has been analyzed. If all have been analyzed, that is to say, the video ends, then the identification of this personnel behavior can be ended. If the video does not end, then it can continue to extract frames and identify in the remaining video until the video ends.

[0194] The specific process of re-judgment of the multimodal large model is described below.

[0195] Re-judgment of head-down scenes. When processing the data of head-down scenes, the small model will basically misjudge it as sleeping. The image frames that the small model thinks are detected as sleeping within the T time are input into the multimodal large model by evenly extracting 4-10 frames of data, so that the multimodal large model can understand the events of the scene. If it is sleeping, it is processed according to the logic of the small model. If it is not sleeping, then the data of this time period is removed from the sleeping list, that is, the corresponding information of illegal behavior is deleted. If the illegal behavior considered by the multimodal large model is yawning, the frame number of this time period can be added to the yawning list, and the cumulative time of yawning is counted later. If the time threshold Tmin is met, the prompt information of the yawning behavior can be output. If the illegal behavior considered by the multimodal large model is leaving the post or occlusion, because the scene must have detected a human body, but now the multimodal large model believes that there is an occlusion or leaving the post event, the confidence of the human body detected by the second detection small model can be obtained at this time. If the confidence is higher than the threshold Tp, it is considered that there is no leaving the post or occlusion event. If the confidence level is low, trust the judgment of the multimodal large model and output the corresponding event.

[0196] Re-judgment of scenes where no face is detected. Many scenes where no face is detected are caused by occlusions during work. The real occlusion is when the entire camera is covered. Therefore, such scenes generally cannot detect human bodies. In order to distinguish them from off-duty events, a multimodal large model can be used for re-judgment. If it is an occlusion, the prompt information of the occlusion event is output. If it is off-duty, the prompt information of the off-duty behavior is output. If the multimodal large model considers the violation to be sleeping on duty, the frame number of the time period can be added to the list of sleeping on duty. When the cumulative time of sleeping on duty is counted later, if the sleeping threshold T is met, the prompt information of the behavior of sleeping on duty is output. If the multimodal large model considers the violation to be yawning, the frame number of the time period can be added to the yawning list. When the cumulative time of yawning is counted later, if the time threshold Tmin is met, the prompt information of the behavior of yawning can be output. If the multimodal large model considers the violation to be leaving the post or occlusion, because the scene must have detected a human body, but now the multimodal large model believes that there is an occlusion or leaving the post event, the confidence of the human body detected by the second detection small model can be obtained. If the confidence is higher than the threshold Tp, it is considered that there is no leaving the post or occlusion event. If the confidence is low, trust the judgment of the multimodal large model and output the corresponding event.

[0197] Re-judgment of yawning behavior. There are many misjudgments of yawning events, so a multimodal large model is used to re-judge yawning behavior. If it is a yawning event, a prompt message indicating the presence of yawning behavior is output. If it is not a yawn, the frame number of this time period in the yawning list is removed.

[0198] For the re-evaluation of the off-duty scene, there are two possibilities for the scene where no person is detected: one is that the person is off-duty, and the other is that the camera is blocked. Therefore, a multi-modal large model can be used for re-evaluation to distinguish them.

[0199] After testing, the personnel behavior recognition method based on a multimodal large model provided by this application performs well in multiple false alarm scenarios and can effectively reduce the false alarm rate. Especially in specific scenarios such as sleeping on duty, the algorithm can accurately identify relevant behaviors. In addition, by introducing timing information, the algorithm's ability to understand the timing of the scene is improved, thereby effectively improving the accuracy of personnel behavior recognition. The initial screening process of the small model also significantly improves the overall operating efficiency of the algorithm, enabling it to achieve real-time processing while ensuring high precision. These improvements and optimizations make the personnel behavior analysis system of this patent perform well in practical applications, with higher reliability and practicality.

[0200] Inference speed: On the A800 server, the single-frame inference time is about 0.1s per frame (one frame is processed in one second, or one frame is processed in two seconds). The frequency of calling the multimodal large model is once every 4 minutes, and the processing speed is about 1.7s each time.

[0201] Figure 3a A schematic diagram of a human behavior recognition result page provided in an embodiment of the present application, such as Figure 3a As shown, users can view the overview information of violations on this page. Specifically, this page can display thumbnails, task IDs, video names, video submission time, station personnel, affiliated workshops, analysis completion time, AI analysis, manual review, operations and other options. Users can jump to the details page or manual review page by clicking the button below the operation. Figure 3b A schematic diagram of a detailed page of a person behavior recognition result provided in an embodiment of the present application, such as Figure 3b As shown, users can observe more detailed information about the violation on this page. Figure 3c A schematic diagram of a manual review page for human behavior recognition results provided in an embodiment of the present application, such as Figure 3c As shown, users can view images with violations on this page, and can modify the recognition results by clicking to confirm the anomaly or correct it.

[0202] This application significantly improves the accuracy and efficiency of personnel behavior recognition by introducing time series information and combining the efficiency of CNN architecture. Compared with the existing technology, when processing continuous video frames, it can more accurately capture the dynamic changes of behavior, reduce false detection and missed detection, solve the time-consuming problem of multimodal large models in time series understanding, and achieve efficient and accurate personnel behavior analysis.

[0203] It should be noted that the personnel behavior recognition method based on a multimodal large model provided by this application can be quickly expanded to other scenarios and newly added detection behaviors, and the algorithm can be quickly iterated. For example, for some behaviors related to production, smart buildings, and railway duty safety: smoking behavior detection, mobile phone behavior detection, phone call behavior detection, no mask detection, irregular dress detection, etc., the algorithm scenario can be expanded and quickly launched.

[0204] Smoking behavior recognition and detection are required in many public places, especially in factory safety production situations, where smoking behavior needs to be strictly controlled. In such situations, the idea of ​​​​fusion of large and small models can be combined to quickly go online. Solution 1: First, use a small model to detect the suspected smoking target human body. For targets with low confidence, intercept the target human body image and use a multimodal large model for re-judgment to improve the accuracy of recognition. For targets with high confidence, trust the judgment of the small model to improve the recognition efficiency of the overall event. Solution 2: Detect faces and smoke targets. When the two overlap, extract frames of the video during this period and input them into the multimodal large model to analyze whether it is smoking behavior.

[0205] Many workplaces prohibit employees from playing with their phones. To deal with the behaviors of playing with phones and making phone calls, we can detect faces and phone targets in the picture. If there is an intersection, we will extract frames from the video during this period and input them into the multimodal large model to understand whether the scene is a phone call. We can detect human targets and phone targets in the picture. If there is an intersection, we will extract frames from the video during this period and input them into the large model to understand whether the scene is playing with the phone.

[0206] For masks and irregular clothing, the face coordinates detected by the small model and the mask coordinate frame can be used to determine the position, and the overlapping relationship can be input into the multimodal large model to analyze whether a mask is worn. For dress code, the standard dress and the detection target can be input into the multimodal large model at the same time, allowing the multimodal large model to compare the differences in dress and analyze irregularities.

[0207] Based on the same inventive concept, the embodiment of the present application also provides a personnel behavior recognition device based on a multimodal large model. Figure 4 A schematic diagram of the structure of a personnel behavior recognition device based on a multimodal large model provided in an embodiment of the present application, the device comprises:

[0208] A face detection module 401 is used to determine whether there is a face in each first image frame in the video to be analyzed based on a first detection model;

[0209] The behavior detection module 402 is used to determine, for each first image frame containing a human face, state information of a preset object of the human face in the first image frame based on the behavior recognition model;

[0210] The confirmation module 403 is used to determine whether there is any violation in the video to be analyzed based on the state information corresponding to each first image frame with a face and the first quantity threshold, and if so, output a prompt message indicating the existence of the violation.

[0211] In a possible implementation, the device further includes:

[0212] A human body detection module 404 is used to determine whether there is a human body in each image frame in the video to be analyzed based on the second detection model; and determine the image frame with a human body as the first image frame;

[0213] The face detection module 401 is specifically used to determine whether there is a face in the detection frame corresponding to the human body in each first image frame based on the first detection model.

[0214] In a possible implementation, the confirmation module 403 is further configured to output a violation prompt message indicating that the person has left his post if the number of consecutive image frames in which no human body exists is greater than a second number threshold.

[0215] In a possible implementation, the confirmation module 403 is further configured to output prompt information of human occlusion if the number of consecutive first image frames without human faces is greater than a third number threshold.

[0216] In a possible implementation, the device further includes:

[0217] The screening module 405 is used to determine an area threshold for each image frame containing a human body according to the width, height and a first preset ratio of the image frame; if the area of ​​the detection frame corresponding to the human body in the image frame is smaller than the area threshold, the detection frame corresponding to the human body in the image frame is deleted.

[0218] In a possible implementation, the screening module 405 is further used to determine, for each image frame in which a human body exists, a central image area corresponding to the image frame according to the width of the image frame and a second preset ratio; determine the center point of the detection frame corresponding to the human body in the image frame, and delete the detection frame of the human body whose center point does not exist in the central image area corresponding to the central image area.

[0219] In a possible implementation, the screening module 405 is also used to determine, if there are multiple human bodies in any image frame in which a human body exists, for each human body, a weight value corresponding to the human body according to the width and height of the detection frame corresponding to the human body, and the width and height of the image frame; and delete the detection frame corresponding to the human body with a non-maximum weight value.

[0220] In a possible implementation, the screening module 405 is also used to obtain a preset number of image frames in the video to be analyzed as preprocessed image frames; based on the light detection model, determine whether there is a backlight phenomenon in the preprocessed image frames. If so, output a prompt message of the light abnormality of the image frame to be analyzed, and end the personnel behavior recognition.

[0221] In a possible implementation, the device further includes:

[0222] The re-inspection module 406 is used to obtain abnormal image frames containing the illegal behavior in the video to be analyzed; if it is determined based on the multimodal large model that the illegal behavior exists in the abnormal image frames, then continue to execute the subsequent step of outputting prompt information indicating the existence of the illegal behavior.

[0223] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5As shown, it includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0224] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 executes the steps of the method for identifying human behavior based on a multimodal large model described in the above embodiments.

[0225] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 502 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0226] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0227] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the personnel behavior recognition method based on a multimodal large model described in the above embodiments.

[0228] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0229] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0232] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for identifying human behavior based on a multimodal large model, characterized in that: The method comprises: Determine whether there is a human face in each first image frame in the video to be analyzed based on the first detection model; For each first image frame containing a human face, determining state information of a preset object of the human face in the first image frame based on the behavior recognition model; According to the state information corresponding to each first image frame with a human face and the first quantity threshold, it is determined whether there is any violation in the video to be analyzed, and if so, prompt information indicating the existence of the violation is output.

2. The method according to claim 1, characterized in that Before determining whether there is a human face in each first image frame in the video to be analyzed based on the first detection model, the method further includes: Determine whether there is a human body in each image frame in the video to be analyzed based on the second detection model; Determine the image frame in which the human body exists as the first image frame; The determining whether there is a human face in each first image frame in the video to be analyzed based on the first detection model includes: Based on the first detection model, determine whether there is a face in the detection frame corresponding to the human body in each first image frame.

3. The method according to claim 2, characterized in that The method further comprises: If the number of consecutive image frames without human bodies is greater than a second number threshold, a violation prompt message indicating that the person has left his post is output.

4. The method according to claim 2, characterized in that: The method further comprises: If the number of consecutive first image frames without human faces is greater than a third number threshold, a prompt message of human occlusion is output.

5. The method according to claim 3, characterized in that: After determining whether there is a human body in each image frame in the video to be analyzed based on the second detection model, and before determining the image frame in which the human body exists as the first image frame, the method further includes: For each image frame in which a human body exists, an area threshold is determined based on the width, height and a first preset ratio of the image frame; if the area of ​​the detection frame corresponding to the human body in the image frame is smaller than the area threshold, the detection frame corresponding to the human body in the image frame is deleted.

6. The method according to claim 3 or 5, characterized in that: After determining whether there is a human body in each image frame in the video to be analyzed based on the second detection model, and before determining the image frame in which the human body exists as the first image frame, the method further includes: For each image frame in which a human body exists, a central image area corresponding to the image frame is determined based on the width of the image frame and a second preset ratio; a center point of a detection frame corresponding to the human body in the image frame is determined, and a detection frame of a human body whose center point does not exist in the central image area corresponding to the detection frame is deleted.

7. The method according to claim 6, characterized in that The method further comprises: If there are multiple human bodies in any image frame in which a human body exists, for each human body, a weight value corresponding to the human body is determined according to the width and height of the detection frame corresponding to the human body, and the width and height of the image frame; Delete the detection box corresponding to the human body with non-maximum weight value.

8. The method according to claim 1, characterized in that: Before determining whether there is a human face in each first image frame in the video to be analyzed based on the first detection model, the method further includes: Acquire a preset number of image frames in the video to be analyzed as preprocessing image frames; Based on the light detection model, determine whether there is a backlight phenomenon in the pre-processed image frame. If so, output a prompt message that the light of the image frame to be analyzed is abnormal, and end the personnel behavior recognition.

9. The method according to any one of claims 1-5, 7-8, characterized in that: After determining that there is a violation in the video to be analyzed and before outputting prompt information indicating that there is a violation, the method further includes: Acquire abnormal image frames containing the illegal behavior in the video to be analyzed; If it is determined based on the multimodal large model that the illegal behavior exists in the abnormal image frame, then the subsequent step of outputting prompt information indicating the existence of the illegal behavior is continued.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the method for identifying human behavior based on a multimodal large model as claimed in any one of claims 1 to 9 when executing a computer program stored in a memory.

Images