A method and system for identifying abnormal behavior combined with laboratory video data analysis

Abstract

The present invention relates to an abnormal behavior recognition method and system combined with laboratory video data analysis, which relates to the field of video recognition, including: extracting key images from multiple part monitoring image sequences to obtain multiple key part monitoring image sequences, extracting key timestamps from monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; performing action recognition based on multiple key part monitoring image sequences to obtain multiple behavior recognition results, calculating multiple change speeds based on multiple key monitoring timestamp sequences; calculating the abnormal behavior probability based on multiple behavior recognition results and multiple change speeds, and obtaining recognition results through discrimination. The present invention can solve the problem that traditional methods cannot quickly and accurately identify potential dangerous behaviors in dynamic and complex laboratory environments, and have slow response and low recognition accuracy; it can significantly improve the efficiency and accuracy of abnormal behavior recognition and effectively prevent the occurrence of safety hazards.

Description

Technical Field

[0001] The present invention relates to the field of video recognition, and in particular to a method and system for identifying abnormal behavior in combination with laboratory video data analysis. Background Art

[0002] As the scale of university laboratories gradually expands, the equipment, personnel and operational complexity within the laboratories are also increasing, resulting in severe challenges for laboratory safety management. Especially in the process of scientific research and teaching, the laboratory, as a dynamic and complex environment, has a large number of dangerous operations and potential safety hazards.

[0003] The laboratory environment is complex and the behavior of people is changeable. The motion characteristics of different people vary greatly. Traditional methods rely too much on static scenes and simple motion patterns in the recognition process and cannot handle various potential risks in dynamic and complex environments. For example, behaviors such as fighting, chasing, and falling may be misjudged as normal operations or difficult to be effectively captured in real-time video streams.

[0004] In summary, traditional abnormal behavior recognition methods based on video analysis cannot quickly and accurately identify potentially dangerous behaviors in dynamic and complex laboratory environments, and have technical problems such as slow response and low recognition accuracy. Summary of the Invention

[0005] The present invention addresses the technical problems that traditional abnormal behavior recognition methods based on video analysis cannot quickly and accurately identify potentially dangerous behaviors in dynamic and complex laboratory environments, and have slow response and low recognition accuracy. This invention provides an abnormal behavior recognition method and system combined with laboratory video data analysis to solve the problem.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] In a first aspect, the present invention provides a method for identifying abnormal behavior in combination with laboratory video data analysis, comprising: during laboratory use, obtaining monitoring video through monitoring equipment, dividing a monitoring image sequence, identifying and dividing multiple limb part images of users in the laboratory, and obtaining multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring timestamp sequences; performing key image recognition and extraction on the multiple part monitoring image sequences to obtain multiple key part monitoring image sequences, and performing key monitoring timestamp extraction on the monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; performing action recognition based on the multiple key part monitoring image sequences to obtain multiple behavior recognition results, and calculating multiple change speeds based on the multiple key monitoring timestamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; calculating the probability of abnormal behavior based on the multiple behavior recognition results and the multiple change speeds, and discriminating to obtain abnormal behavior recognition results.

[0008] In a second aspect, the present invention provides an abnormal behavior recognition system combined with laboratory video data analysis, including: a part monitoring image sequence acquisition module, which is used to obtain monitoring videos through monitoring equipment during laboratory use, divide the monitoring image sequence, identify and divide multiple limb part images of users in the laboratory, and obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring timestamp sequences; a key information extraction module, which is used to perform key image recognition and extraction on the multiple part monitoring image sequences, obtain multiple key part monitoring image sequences, and perform key monitoring timestamp extraction on the monitoring timestamp sequences, obtain multiple key monitoring timestamp sequences; a key information analysis module, which is used to perform action recognition based on the multiple key part monitoring image sequences, obtain multiple behavior recognition results, and calculate multiple change speeds based on the multiple key monitoring timestamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; an abnormal behavior recognition module, which is used to calculate the probability of abnormal behavior based on the multiple behavior recognition results and the multiple change speeds, and discriminate the abnormal behavior recognition results.

[0009] In a third aspect, the present invention further provides an electronic device, comprising:

[0010] At least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the steps of any one of the methods described in the first aspect above.

[0011] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the steps of the method described in any one of the first aspects are implemented.

[0012] The beneficial effects of the present invention are as follows: by dividing the monitoring video to obtain a monitoring image sequence, the recognition and division of multiple body part images of the user in the laboratory are performed to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to the monitoring timestamp sequences; then, key image recognition and extraction are performed on the multiple body part monitoring image sequences to obtain multiple key body part monitoring image sequences; on the other hand, key monitoring timestamp extraction is performed on the monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; then, action recognition is performed based on the multiple key body part monitoring image sequences to obtain multiple behavior recognition results, wherein the behavior recognition results include normal behavior or abnormal behavior; based on the multiple key monitoring timestamp sequences, multiple change speeds are calculated; finally, the abnormal behavior probability is calculated based on the multiple behavior recognition results and the multiple change speeds, and the abnormal behavior recognition result is obtained based on the abnormal behavior probability; the above method can quickly locate key abnormal actions in a dynamic and complex laboratory environment, efficiently and accurately identify potential dangerous behaviors, significantly improve the efficiency and accuracy of abnormal behavior recognition, thereby effectively preventing the occurrence of safety hazards in the laboratory and improving the overall safety protection level. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of an abnormal behavior identification method combined with laboratory video data analysis provided by the present invention;

[0014] Figure 2 This is a schematic diagram of the structure of an abnormal behavior recognition system combined with laboratory video data analysis provided by the present invention;

[0015] Figure 3 A schematic structural diagram of the electronic device provided by the present invention;

[0016] Figure 4 A schematic structural diagram of a computer-readable storage medium provided by the present invention.

[0017] In the accompanying drawings, the components represented by the reference numerals are described as follows:

[0018] Part monitoring image sequence acquisition module 01, key information extraction module 02, key information analysis module 03, abnormal behavior recognition module 04, electronic device 500, memory 510, processor 520, first computer program 511, computer readable storage medium 600, second computer program 611. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0021] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0022] Example 1, as Figure 1 As shown, an embodiment of the present invention provides a method for identifying abnormal behavior by combining laboratory video data analysis, which specifically includes the following steps:

[0023] S100: During use in the laboratory, monitoring video is obtained through monitoring equipment, and a monitoring image sequence is obtained by dividing and identifying images of multiple body parts of users in the laboratory to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to monitoring timestamp sequences.

[0024] Furthermore, step S100 of the present invention further includes:

[0025] S110: During laboratory use, the user's area is monitored by monitoring equipment to obtain a monitoring video, wherein the monitoring video includes monitoring images under multiple timestamps, and the multiple timestamps form a monitoring timestamp sequence; S120: According to the monitoring timestamp sequence, the monitoring images in the monitoring video are divided to obtain a monitoring image sequence.

[0026] Specifically, during the use of the laboratory, monitoring equipment (such as cameras, intelligent monitoring systems) is used to continuously monitor the activity areas of people in the laboratory and generate a real-time video stream. The video stream records all dynamics in the laboratory, including people's movements, position changes, etc. Among them, each frame of the image in the video acquisition process corresponds to a specific timestamp. By automatically marking the timestamp, each image is associated with its corresponding acquisition time. Through continuous video monitoring, a series of monitoring images arranged in chronological order are formed, namely monitoring videos; at the same time, these images are marked with timestamps to form a complete monitoring timestamp sequence. The monitoring timestamp sequence provides information support in the time dimension for subsequent data analysis, behavior recognition, etc., and can track and analyze user behavior at each moment (such as students fighting, chasing, etc.).

[0027] Each frame in a surveillance video corresponds to a specific timestamp. Timestamps are a mapping between images and time, allowing each frame to be uniquely identified by its timestamp. Next, the surveillance images within the surveillance video are segmented according to the monitoring timestamp sequence. Specifically, the surveillance video is segmented by frame based on the unit time step of the timestamp (e.g., 0.1 second). The segmented images are then arranged in timestamp order to generate a surveillance image sequence.

[0028] S130: Identify and divide the multiple body part images of the user in each monitoring image to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to monitoring time stamp sequences.

[0029] Furthermore, step S130 of the present invention further includes:

[0030] S131: Based on the user monitoring data within a historical period, a set of sample monitoring images is collected, and the arm area image, leg area image and torso area image of the user in each sample monitoring image are divided and identified to obtain a set of sample limb part image division results; S132: Based on semantic segmentation, a network structure of a monitoring image division channel is constructed, wherein the network structure of the monitoring image division channel includes an encoder and a decoder; S133: Using the sample monitoring image set and the sample limb part image division result set, the monitoring image division channel is supervised trained until the training converges to obtain a monitoring image division channel; S134: The multiple monitoring images in the monitoring image sequence are respectively input into the monitoring image division channel, and multiple limb part image division results are obtained by identification and division; S135: The images of multiple limb parts in the multiple limb part image division results are extracted to obtain multiple part monitoring image sequences.

[0031] Specifically, based on historical user monitoring data, multiple sample monitoring images are collected. Sample monitoring images are a representative set of images selected from the video according to timestamps or event trigger conditions, typically covering a variety of scenes or changes in user behavior. A sample monitoring image set is constructed based on these multiple sample monitoring images. Next, the user's arm area (including upper arms, forearms, hands, etc.), leg area (including thighs, calves, feet, etc.), and torso area (including chest, abdomen, and back) images within each sample monitoring image are segmented and labeled to obtain a sample limb area image segmentation result set.

[0032] Based on semantic segmentation, a network structure of a monitoring image segmentation channel is constructed. The monitoring image segmentation channel is used to automatically segment the user's limb areas (such as arms, legs, and torso) from the original monitoring image, and generate an independent image channel for each limb part. The network structure of the monitoring image segmentation channel includes an encoder and a decoder. The encoder is used to extract image features and usually includes multiple convolutional layers and pooling layers, which help the model learn the spatial information and semantic features in the image layer by layer. For example, the convolutional layer extracts local features of the image through convolution operations, such as the user's limb contours; the decoder is used to restore the spatial information of the image and perform fine segmentation of the limb area, including upsampling layers, transposed convolution layers, etc. The upsampling layer is used to gradually restore the spatial resolution of the feature map to the size of the original image; the transposed convolution layer is used to restore the feature map to the spatial dimension of the image through transposed convolution operations, and simultaneously classify the limb parts so that each pixel can be classified as a different limb area (such as arms, legs, and torso).

[0033] Then, the sample monitoring image set and the sample limb part image segmentation result set are used as training data, the sample monitoring image is used as input, and the sample limb part image segmentation result is used as output, and the monitoring image segmentation channel is supervised and trained. First, the monitoring image is input into the network, and after the encoder extracts the features, the decoder performs upsampling to restore the image resolution, and finally a segmentation map of the limb part is generated; then, the segmentation map generated by the network is compared with the true label, and the loss value is calculated; further, a gradient descent optimization algorithm (such as the Adam optimizer) is used to update the network weights through back propagation, reduce the loss, and gradually improve the performance of the network; then, the above process is repeated until the network converges, that is, the loss value tends to be stable, indicating that the network has learned how to accurately divide the limb area from the input image, and a trained monitoring image segmentation channel is obtained.

[0034] Then, the multiple monitoring images in the monitoring image sequence are respectively input into the monitoring image division channel for identification and division to obtain multiple limb part image division results, each result corresponding to a limb part identified in the input image; finally, the images of multiple limb parts (such as arms, legs, and torso) in the multiple limb part image division results are extracted and arranged according to the monitoring timestamps to obtain multiple part monitoring image sequences, wherein the part monitoring image sequence corresponds to the monitoring timestamp sequence.

[0035] By dividing the monitoring images according to body parts, it is helpful to accurately extract the dynamic features of each part and individually identify the actions and movement trajectories of each part, so that the user's specific behavior can be judged more carefully and accurately, and the accuracy of abnormal behavior identification can be improved.

[0036] S200: performing key image recognition and extraction on the plurality of part monitoring image sequences to obtain a plurality of key part monitoring image sequences, and performing key monitoring timestamp extraction on the monitoring timestamp sequences to obtain a plurality of key monitoring timestamp sequences.

[0037] Furthermore, step S200 of the present invention further includes:

[0038] S210: Extracting a first part monitoring image from the first part monitoring image sequence as a first key part monitoring image.

[0039] Specifically, any one part monitoring image sequence is randomly selected from the multiple part monitoring image sequences and set as the first part monitoring image sequence; then the first part monitoring image is extracted from the first part monitoring image sequence and set as the first key part monitoring image, wherein the first part monitoring image refers to the monitoring image of the first part (such as the arm area or the leg area) corresponding to the initial monitoring timestamp.

[0040] S220: extracting a second part monitoring image in the first part monitoring image sequence, analyzing the similarity with the first key part monitoring image, obtaining a similarity parameter, and determining whether the similarity parameter is greater than a similarity threshold.

[0041] Furthermore, step S220 of the present invention further includes:

[0042] S221: According to the limb part corresponding to the first part monitoring image sequence, based on the limb part monitoring image data in the historical time, multiple sample part monitoring image combinations are collected, and the similarity between the two sample part monitoring images in each sample part monitoring image combination is identified to obtain a sample similarity parameter set; S222: Based on the twin network, a part monitoring image recognition channel is constructed; S223: Using the multiple sample part monitoring image combinations and the sample similarity parameter set, the part monitoring image recognition channel is supervised trained until the training converges; S224: The first key part monitoring image and the second part monitoring image are input into the part monitoring image recognition channel to identify and obtain similarity parameters.

[0043] Specifically, first, the limb part (e.g., arm area) corresponding to the first part monitoring image sequence is obtained. Based on the limb part (arm area) monitoring image data over a historical period, multiple sample part monitoring image combinations are collected, where each sample part monitoring image combination includes two sample part monitoring images. Similarity calculations are then performed on the two sample part monitoring images within each sample part monitoring image combination to obtain multiple image similarities. For example, image similarity calculations are performed using Euclidean distance, which measures image differences by calculating the straight-line distance between image pixels. This calculation can be performed based on image feature points (e.g., edges or color distribution). Multiple sample part monitoring image combinations are then identified based on the multiple image similarities to obtain a set of sample similarity parameters, where the sample similarity parameters correspond one-to-one to the sample part monitoring image combinations.

[0044] A part monitoring image recognition channel is constructed based on a twin network. The twin network consists of two identical neural networks that share weights and accept input images as input. By calculating the output features of the two networks, the twin network can determine the similarity between the input images, which is used to process the comparative recognition of limb parts (such as arms, legs, and torso) in the monitoring images. Next, the part monitoring image recognition channel is supervised trained using the multiple sample part monitoring image combinations and the sample similarity parameter set as training data, with the sample part monitoring image combinations as input and the sample similarity parameters as output. First, two sample images are input into the two sub-networks of the twin network respectively. After processing through convolution, pooling, and fully connected layers, corresponding feature vectors are output. Next, the distance between the two feature vectors is calculated (usually using Euclidean distance or cosine similarity) to obtain a similarity score between the images. Then, a loss value is calculated based on the comparison of the similarity score with the sample label. Further, based on the loss value, backpropagation is performed to update the network parameters. Through multiple iterations and adjustments until the network loss converges, the network can accurately judge the similarity of new images, thus obtaining a trained part monitoring image recognition channel. By constructing a part monitoring image recognition channel based on the twin network, the efficiency and accuracy of similarity comparison analysis of part monitoring images can be improved, providing support for subsequent key image extraction.

[0045] Then, a second part monitoring image is extracted from the first part monitoring image sequence, where the second part monitoring image is the part monitoring image corresponding to the next monitoring timestamp immediately preceding the first key part monitoring image. The first key part monitoring image and the second part monitoring image are then input into the part monitoring image recognition channel for similarity recognition, and a similarity parameter is output.

[0046] Obtain a similarity threshold. The similarity threshold can be set based on the accuracy of abnormal behavior recognition. The higher the abnormal behavior recognition accuracy, the more data needs to be analyzed, and the larger the similarity threshold. The similarity parameters are then determined based on the similarity threshold. For example, the similarity threshold is 80%.

[0047] S230: If yes, then the second part monitoring image is not used as the key part monitoring image, and the first key part monitoring image is continued as the basis for similarity analysis and judgment of the third part monitoring image; S240: If not, then the second part monitoring image is used as the second key part monitoring image, and the second key part monitoring image is continued as the basis for similarity analysis and judgment of the third part monitoring image; S250: Continue to identify and extract to obtain the first key part monitoring image sequence, and perform key image identification and extraction on the other multiple part monitoring image sequences to obtain multiple key part monitoring image sequences; S260: Obtain the monitoring timestamps corresponding to the key part monitoring images in the multiple key part monitoring image sequences in the monitoring timestamp sequence, and extract to obtain multiple key monitoring timestamp sequences.

[0048] Specifically, if the similarity parameter is greater than the similarity threshold, indicating that the deviation between the second part monitoring image and the first key part monitoring image is small, that is, the image changes are not large and may describe similar states or actions, then the second part monitoring image will not be used as the key part monitoring image; then, the first key part monitoring image will be used as the basis for similarity comparison to perform similarity analysis and judgment on the third part monitoring image (that is, the next part monitoring image adjacent to the second part monitoring image).

[0049] If the similarity parameter is less than or equal to the similarity threshold, indicating that the difference between the two images is large, indicating that the action or state described by the two images has changed significantly, the second part monitoring image is used as the second key part monitoring image, and based on the second key part monitoring image, the similarity analysis and judgment of the third part monitoring image are performed.

[0050] The same method is then used to extract key part monitoring images from the first part monitoring image sequence to obtain a first key part monitoring image sequence. Key image recognition and extraction are then performed on multiple other part monitoring image sequences (e.g., the leg region and the torso region) to obtain multiple key part monitoring image sequences. Furthermore, the corresponding monitoring timestamps of the key part monitoring images within the multiple key part monitoring image sequences within the monitoring timestamp sequence are obtained to generate multiple key monitoring timestamp sequences.

[0051] Through precise image sequence segmentation, key image extraction and timestamp matching, abnormal behavior actions in laboratory environments can be efficiently and accurately identified and located, thereby improving the efficiency and accuracy of abnormal behavior identification.

[0052] S300: performing action recognition based on the plurality of key part monitoring image sequences to obtain a plurality of behavior recognition results, and calculating a plurality of change speeds based on the plurality of key monitoring timestamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior.

[0053] Furthermore, step S300 of the present invention further includes:

[0054] S310: Based on the multiple key part monitoring image sequences, action recognition of multiple limb parts is performed respectively to obtain multiple behavior recognition results, wherein a set of sample key part monitoring image sequences and a set of sample behavior recognition results are collected, and a motion behavior recognition channel is trained based on a convolutional neural network to perform action recognition, and each behavior recognition result includes normal behavior or abnormal behavior; S320: Extract the number of multiple key monitoring timestamps in the multiple key monitoring timestamp sequences, and calculate the ratio of the number of each key monitoring timestamp to the sum of the number of multiple key monitoring timestamps as multiple change speeds.

[0055] Specifically, an action behavior recognition channel is constructed based on a convolutional neural network, for example, an action behavior recognition channel is constructed based on a convolutional neural network and a SlowFast network. The SlowFast network is a two-stream network architecture designed specifically for video action recognition, including a Slow branch and a Fast branch. The Slow branch processes video information at a low frame rate to capture slower movements (for example, large movements, stable movements, etc.), and the Fast branch processes video information at a high frame rate to capture fast detailed movements (for example, fast movements, subtle changes, etc.). Then, a set of sample key part monitoring image sequences and a set of sample behavior recognition results are collected, wherein the behavior recognition results include normal behavior or abnormal behavior; then, the sample key part monitoring images are used as input and the sample behavior recognition results are used as output, and the action behavior recognition channel is supervised and trained through the sample key part monitoring image sequence set and the sample behavior recognition result set. First, the sample key part monitoring image sequence is input into the network, and each image sequence is input into the neural network as a time series. The convolution layer is usually used for feature extraction, and then the time series data is processed using models such as RNN or LSTM; then, the model outputs a predicted label for each sample, and the predicted label is "normal behavior" or "abnormal behavior"; then, the loss between the predicted result and the true label is calculated, and according to the calculated loss, the weight of the network is adjusted through the back propagation algorithm so that the model can be gradually optimized; the above process is repeated until the training process converges, and the model can better predict the behavior label from the key part monitoring image sequence, and a trained action behavior recognition channel is obtained.

[0056] The multiple key body monitoring image sequences are input into the action behavior recognition channel to perform action recognition on multiple body parts, outputting multiple action recognition results (normal or abnormal). Optionally, a convolutional neural network can be used to perform action recognition on multiple key body monitoring image sequences, outputting multiple action recognition results.

[0057] On the other hand, the number of multiple key monitoring timestamps in the multiple key monitoring timestamp sequences is extracted, and then the ratio between the number of each key monitoring timestamp and the total number of all key monitoring timestamps is calculated, wherein the larger the ratio, the more frequent the changes in limb movements, the faster the movement speed, and may be abnormal behavior; and the ratio is set as the change speed, and multiple change speeds are obtained, wherein, when the change speed is small, it indicates that the user's movements are relatively stable and belong to normal behavior, such as slow movements such as standing and walking; when the change speed is large, it indicates that the user's movements are relatively fast or violent, and may be abnormal behavior, such as fast running, fighting, chasing, etc.

[0058] By calculating the rate of change of the ratio of the number of key timestamps to the total number, abnormal behaviors in the laboratory, such as fighting and running fast, can be quickly identified, which helps to improve the response speed and accuracy of the laboratory monitoring system, and timely discover and deal with potential safety hazards.

[0059] S400: Calculating an abnormal behavior probability based on the multiple behavior recognition results and the multiple change speeds, and determining an abnormal behavior recognition result.

[0060] Furthermore, step S400 of the present invention further includes:

[0061] S410: Based on the multiple change speeds, multiple identification weights are allocated; S420: The multiple behavior identification results are classified to obtain multiple normal behaviors and multiple abnormal behaviors; S430: The identification weights corresponding to the multiple normal behaviors are added to obtain the normal behavior probability, and the identification weights corresponding to the multiple abnormal behaviors are added to obtain the abnormal behavior probability; S440: Determine whether the abnormal behavior probability is greater than or equal to a preset abnormal behavior probability threshold to obtain the abnormal behavior identification result.

[0062] Specifically, identification weights are set based on the multiple change rates to obtain multiple identification weights, where the change rate and the identification weight are positively correlated, i.e., the greater the change rate, the greater the weight, and the sum of the multiple identification weights is 1. The relationship between the change rate and the weight can be mapped using a nonlinear function, such as an exponential function or a logarithmic function. Furthermore, the multiple behavior identification results are classified to obtain multiple normal behaviors and multiple abnormal behaviors. Furthermore, multiple identification weights corresponding to the multiple normal behaviors are obtained and summed, with the summed result being used as the probability of normal behavior. The identification weights corresponding to the multiple abnormal behaviors are summed to obtain the probability of abnormal behavior.

[0063] Obtain a preset abnormal behavior probability threshold, which is a critical value used to distinguish normal behavior from abnormal behavior and can be set according to the actual scenario; then judge the abnormal behavior probability according to the preset abnormal behavior probability threshold. If the abnormal behavior probability is less than or equal to the preset abnormal behavior probability threshold, it is determined to be normal behavior; if the abnormal behavior probability is greater than the preset abnormal behavior probability threshold, it is determined to be abnormal behavior and an early warning is issued.

[0064] The embodiment of the present invention provides a method for identifying abnormal behavior by combining laboratory video data analysis, which has at least the following technical effects:

[0065] By dividing the monitoring video to obtain a monitoring image sequence, the images of multiple body parts of the user in the laboratory are identified and divided to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to the monitoring timestamp sequences; then, key image recognition and extraction are performed on the multiple body part monitoring image sequences to obtain multiple key body part monitoring image sequences; on the other hand, key monitoring timestamps are extracted from the monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; then, action recognition is performed based on the multiple key body part monitoring image sequences to obtain multiple behavior recognition results, wherein the behavior recognition results include normal behavior or abnormal behavior; based on the multiple key monitoring timestamp sequences, multiple change speeds are calculated; finally, the abnormal behavior probability is calculated based on the multiple behavior recognition results and the multiple change speeds, and the abnormal behavior recognition result is obtained based on the abnormal behavior probability; the above method can quickly locate key abnormal actions in a dynamic and complex laboratory environment, efficiently and accurately identify potential dangerous behaviors, and significantly improve the efficiency and accuracy of abnormal behavior recognition, thereby effectively preventing safety hazards in the laboratory and improving the overall safety protection level.

[0066] Example 2, as Figure 2As shown, based on the same inventive concept as the abnormal behavior recognition method combined with laboratory video data analysis provided in Example 1, an embodiment of the present invention further provides an abnormal behavior recognition system combined with laboratory video data analysis, including: a part monitoring image sequence acquisition module 01, for obtaining a monitoring video through monitoring equipment during laboratory use, dividing the monitoring image sequence, and performing recognition and division of multiple body part images of the user in the laboratory to obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring timestamp sequences; a key information extraction module 02, for performing key image recognition and extraction on the multiple part monitoring image sequences to obtain multiple key part monitoring image sequences, and performing key monitoring timestamp extraction on the monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; a key information analysis module 03, for performing action recognition based on the multiple key part monitoring image sequences to obtain multiple behavior recognition results, and calculating multiple change speeds based on the multiple key monitoring timestamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; and an abnormal behavior recognition module 04, for calculating the probability of abnormal behavior based on the multiple behavior recognition results and the multiple change speeds, and discriminating the abnormal behavior recognition results.

[0067] Furthermore, the abnormal behavior identification system combined with laboratory video data analysis also includes: during laboratory use, monitoring the user's area through monitoring equipment to obtain a monitoring video, wherein the monitoring video includes monitoring images under multiple timestamps, and the multiple timestamps form a monitoring timestamp sequence; according to the monitoring timestamp sequence, the monitoring images in the monitoring video are divided to obtain a monitoring image sequence; multiple limb part images of the user in each monitoring image are identified and divided to obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to the monitoring timestamp sequence.

[0068] Furthermore, the abnormal behavior recognition system combined with laboratory video data analysis also includes: collecting a sample monitoring image set based on user monitoring data in a historical period, and dividing and identifying the user's arm area image, leg area image and torso area image in each sample monitoring image to obtain a sample limb part image division result set; constructing a network structure of a monitoring image division channel based on semantic segmentation, wherein the network structure of the monitoring image division channel includes an encoder and a decoder; using the sample monitoring image set and the sample limb part image division result set, the monitoring image division channel is supervised trained until the training converges to obtain a monitoring image division channel; inputting multiple monitoring images in the monitoring image sequence into the monitoring image division channel respectively, identifying and dividing to obtain multiple limb part image division results; extracting images of multiple limb parts in the multiple limb part image division results to obtain multiple part monitoring image sequences.

[0069] Furthermore, the abnormal behavior recognition system combined with laboratory video data analysis also includes: extracting the first part monitoring image in the first part monitoring image sequence as the first key part monitoring image; extracting the second part monitoring image in the first part monitoring image sequence, analyzing the similarity with the first key part monitoring image, obtaining a similarity parameter, and judging whether it is greater than a similarity threshold; if so, not taking the second part monitoring image as the key part monitoring image, and continuing to perform similarity analysis and judgment of the third part monitoring image based on the first key part monitoring image; if not, taking the second part monitoring image as the second key part monitoring image, and performing similarity analysis and judgment of the third part monitoring image based on the second key part monitoring image; continuing to identify and extract to obtain the first key part monitoring image sequence, and performing key image identification and extraction on multiple other part monitoring image sequences to obtain multiple key part monitoring image sequences; obtaining the monitoring timestamps corresponding to the key part monitoring images in the multiple key part monitoring image sequences in the monitoring timestamp sequence, and extracting multiple key monitoring timestamp sequences.

[0070] Furthermore, the abnormal behavior recognition system combined with laboratory video data analysis also includes: according to the limb part corresponding to the first part monitoring image sequence, based on the limb part monitoring image data in historical time, collecting multiple sample part monitoring image combinations, and identifying the similarity between the two sample part monitoring images in each sample part monitoring image combination to obtain a sample similarity parameter set; based on the twin network, constructing a part monitoring image recognition channel; using the multiple sample part monitoring image combinations and the sample similarity parameter set, supervised training is performed on the part monitoring image recognition channel until the training converges; the first key part monitoring image and the second part monitoring image are input into the part monitoring image recognition channel to identify and obtain similarity parameters.

[0071] Furthermore, the abnormal behavior recognition system combined with laboratory video data analysis also includes: performing action recognition of multiple limb parts respectively according to the multiple key part monitoring image sequences to obtain multiple behavior recognition results, wherein a set of sample key part monitoring image sequences and a set of sample behavior recognition results are collected, and an action behavior recognition channel is trained based on a convolutional neural network to perform action recognition, and each behavior recognition result includes normal behavior or abnormal behavior; extracting the number of multiple key monitoring timestamps in the multiple key monitoring timestamp sequences, and calculating the ratio of the number of each key monitoring timestamp to the sum of the number of multiple key monitoring timestamps as multiple change speeds.

[0072] Furthermore, the abnormal behavior identification system combined with laboratory video data analysis also includes: allocating multiple identification weights based on the multiple change speeds; classifying the multiple behavior identification results to obtain multiple normal behaviors and multiple abnormal behaviors; adding the identification weights corresponding to the multiple normal behaviors to obtain the normal behavior probability, and adding the identification weights corresponding to the multiple abnormal behaviors to obtain the abnormal behavior probability; judging whether the abnormal behavior probability is greater than or equal to a preset abnormal behavior probability threshold to obtain an abnormal behavior identification result.

[0073] For example three, please refer to Figure 3 , Figure 3 Schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. Figure 3As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, the following steps are implemented: during use in a laboratory, monitoring video is obtained through monitoring equipment, and monitoring image sequences are obtained by dividing the images, and recognition and division of multiple body part images of a user in the laboratory are performed to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to monitoring time stamp sequences; key image recognition and extraction are performed on the multiple body part monitoring image sequences to obtain multiple key body part monitoring image sequences, and key monitoring time stamp extraction is performed on the monitoring time stamp sequences to obtain multiple key monitoring time stamp sequences; action recognition is performed based on the multiple key body part monitoring image sequences to obtain multiple behavior recognition results, and multiple change speeds are calculated based on the multiple key monitoring time stamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; abnormal behavior probabilities are calculated based on the multiple behavior recognition results and the multiple change speeds, and abnormal behavior recognition results are obtained by discrimination.

[0074] For example 4, please refer to Figure 4 , Figure 4 Schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 600, on which a second computer program 611 is stored. When the second computer program 611 is executed by a processor, the following steps are implemented: during laboratory use, monitoring video is obtained through monitoring equipment, and a monitoring image sequence is obtained by dividing the monitoring image sequence, and identification and division of multiple body part images of the user in the laboratory are performed to obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring timestamp sequences; key image recognition and extraction are performed on the multiple part monitoring image sequences to obtain multiple key part monitoring image sequences, and key monitoring timestamp extraction is performed on the monitoring timestamp sequences to obtain multiple key monitoring timestamp sequences; action recognition is performed based on the multiple key part monitoring image sequences to obtain multiple behavior recognition results, and multiple change speeds are calculated based on the multiple key monitoring timestamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; based on the multiple behavior recognition results and the multiple change speeds, the probability of abnormal behavior is calculated, and the abnormal behavior recognition result is obtained by discrimination.

[0075] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0076] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0080] Although preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts become known.

[0081] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for identifying abnormal behavior by combining laboratory video data analysis, characterized in that: Methods include: During use in the laboratory, monitoring video is obtained through monitoring equipment, and a monitoring image sequence is obtained by segmentation. Multiple body part images of the user in the laboratory are identified and segmented to obtain multiple body part monitoring image sequences, wherein the multiple body part monitoring image sequences correspond to monitoring time stamp sequences; Performing key image recognition and extraction on the plurality of part monitoring image sequences to obtain a plurality of key part monitoring image sequences, and performing key monitoring time stamp extraction on the monitoring time stamp sequences to obtain a plurality of key monitoring time stamp sequences; Performing action recognition based on the multiple key part monitoring image sequences to obtain multiple behavior recognition results, and calculating multiple change speeds based on the multiple key monitoring time stamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior, including: Based on the multiple key part monitoring image sequences, action recognition of multiple limb parts is performed respectively to obtain multiple behavior recognition results, wherein a set of sample key part monitoring image sequences and a set of sample behavior recognition results are collected, and a motion behavior recognition channel is trained based on a convolutional neural network to perform action recognition, and each behavior recognition result includes normal behavior or abnormal behavior; Extracting multiple numbers of key monitoring time stamps in the multiple key monitoring time stamp sequences, and calculating a ratio of each number of key monitoring time stamps to the sum of the numbers of the multiple key monitoring time stamps as multiple change speeds; According to the multiple behavior recognition results and the multiple change speeds, the probability of abnormal behavior is calculated and the abnormal behavior recognition result is obtained by discrimination.

2. The abnormal behavior identification method combined with laboratory video data analysis according to claim 1 is characterized in that: During use in the laboratory, monitoring video is obtained through monitoring equipment, and monitoring image sequences are obtained by segmentation. Multiple body part images of users in the laboratory are identified and segmented to obtain multiple body part monitoring image sequences, including: During use in the laboratory, the user's area is monitored by a monitoring device to obtain a monitoring video, wherein the monitoring video includes monitoring images at multiple timestamps, and the multiple timestamps form a monitoring timestamp sequence; Dividing the monitoring images in the monitoring video according to the monitoring timestamp sequence to obtain a monitoring image sequence; Multiple limb part images of the user in each monitoring image are identified and divided to obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring time stamp sequences.

3. The abnormal behavior identification method combined with laboratory video data analysis according to claim 2 is characterized in that: Identify and segment multiple body part images of the user in each monitoring image to obtain multiple body part monitoring image sequences, including: Based on the user monitoring data within the historical period, a set of sample monitoring images is collected, and the user's arm area image, leg area image, and torso area image in each sample monitoring image are divided and labeled to obtain a set of sample limb area image division results; Based on semantic segmentation, a network structure of a monitoring image segmentation channel is constructed, wherein the network structure of the monitoring image segmentation channel includes an encoder and a decoder; Using the sample monitoring image set and the sample limb part image segmentation result set, supervised training is performed on the monitoring image segmentation channel until the training converges to obtain the monitoring image segmentation channel; Inputting the plurality of monitoring images in the monitoring image sequence into the monitoring image segmentation channel respectively, identifying and segmenting to obtain a plurality of limb part image segmentation results; Images of multiple limb parts within the multiple limb part image segmentation results are extracted to obtain multiple part monitoring image sequences.

4. The abnormal behavior identification method combined with laboratory video data analysis according to claim 1 is characterized in that: Performing key image recognition and extraction on the plurality of part monitoring image sequences to obtain a plurality of key part monitoring image sequences, and performing key monitoring timestamp extraction on the monitoring timestamp sequences, including: Extracting a first part monitoring image from the first part monitoring image sequence as a first key part monitoring image; Extracting a second part monitoring image from the first part monitoring image sequence, analyzing the similarity with the first key part monitoring image, obtaining a similarity parameter, and determining whether the similarity parameter is greater than a similarity threshold; If so, the second part monitoring image is not used as the key part monitoring image, and the similarity analysis and judgment of the third part monitoring image is continued based on the first key part monitoring image; If not, the second part monitoring image is used as the second key part monitoring image, and based on the second key part monitoring image, the third part monitoring image is analyzed and judged for similarity; Continue to identify and extract to obtain a first key part monitoring image sequence, and perform key image identification and extraction on multiple other part monitoring image sequences to obtain multiple key part monitoring image sequences; Acquire monitoring timestamps corresponding to the key part monitoring images in the plurality of key part monitoring image sequences in the monitoring timestamp sequence, and extract and obtain a plurality of key monitoring timestamp sequences.

5. The abnormal behavior identification method combined with laboratory video data analysis according to claim 4 is characterized in that: Extracting a second part monitoring image from the first part monitoring image sequence, analyzing the similarity with the first key part monitoring image, and obtaining a similarity parameter, including: According to the limb part corresponding to the first part monitoring image sequence, based on the limb part monitoring image data in a historical period, a plurality of sample part monitoring image combinations are collected, and the similarity between two sample part monitoring images in each sample part monitoring image combination is marked to obtain a sample similarity parameter set; Based on the twin network, a part monitoring image recognition channel is constructed; Performing supervised training on the part monitoring image recognition channel using the plurality of sample part monitoring image combinations and the sample similarity parameter set until the training converges; The first key part monitoring image and the second part monitoring image are input into the part monitoring image recognition channel, and the similarity parameters are obtained by recognition.

6. The abnormal behavior identification method combined with laboratory video data analysis according to claim 1 is characterized in that: Calculating the abnormal behavior probability based on the multiple behavior recognition results and the multiple change speeds, and determining the abnormal behavior recognition result, including: allocating and obtaining a plurality of recognition weights according to the plurality of change speeds; Classifying the multiple behavior recognition results to obtain multiple normal behaviors and multiple abnormal behaviors; Adding the recognition weights corresponding to the multiple normal behaviors to obtain a normal behavior probability, and adding the recognition weights corresponding to the multiple abnormal behaviors to obtain an abnormal behavior probability; Determine whether the abnormal behavior probability is greater than or equal to a preset abnormal behavior probability threshold to obtain an abnormal behavior recognition result.

7. An abnormal behavior recognition system combined with laboratory video data analysis, characterized in that: The steps for implementing the abnormal behavior identification method combined with laboratory video data analysis as described in any one of claims 1 to 6 include: a part monitoring image sequence acquisition module, configured to, during laboratory use, obtain monitoring videos through monitoring equipment, divide and obtain monitoring image sequences, identify and divide multiple body part images of users in the laboratory, and obtain multiple part monitoring image sequences, wherein the multiple part monitoring image sequences correspond to monitoring time stamp sequences; a key information extraction module, configured to perform key image recognition and extraction on the plurality of part monitoring image sequences to obtain a plurality of key part monitoring image sequences, and to perform key monitoring timestamp extraction on the monitoring timestamp sequences to obtain a plurality of key monitoring timestamp sequences; a key information analysis module, configured to perform action recognition based on the plurality of key part monitoring image sequences to obtain a plurality of behavior recognition results, and to calculate a plurality of change speeds based on the plurality of key monitoring time stamp sequences, wherein the behavior recognition results include normal behavior or abnormal behavior; The abnormal behavior recognition module is used to calculate the probability of abnormal behavior based on the multiple behavior recognition results and the multiple change speeds, and to determine the abnormal behavior recognition results.

8. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the steps of the abnormal behavior identification method combined with laboratory video data analysis as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a computer software program, which, when executed by a processor, implements the steps of the abnormal behavior identification method combined with laboratory video data analysis as described in any one of claims 1 to 6.

Images