Method for detecting abnormal behaviors of personnel around river based on deep learning image processing

Abstract

The invention discloses a river surrounding personnel abnormal behavior detection method based on deep learning image processing, and relates to the technical field of image processing, and the method comprises the steps: obtaining first information and second information; the first information is preprocessed; training a preset self-supervised key point extraction network according to the preprocessed data; inputting the preprocessed data into a multi-modal self-supervised learning network, outputting a behavior vector library, and obtaining a distance discrimination set according to the tag and the behavior vector library; preprocessing the second information, inputting the preprocessed second information into the trained self-supervised key point extraction network, outputting a corresponding real-time embedding vector, calculating the distance between the real-time embedding vector and an embedding vector corresponding to an abnormal behavior, calculating the sorting proportion of the distance in a distance discrimination set, comparing the sorting proportion with a preset alarm threshold value, and determining the abnormal behavior. And if the sorting proportion is smaller than or equal to an alarm threshold value, carrying out abnormal behavior alarm. The method improves the detection accuracy, and enables the abnormal behavior detection of the personnel around the river to be more comprehensive.

Description

Technical Field

[0001] The present invention relates to the technical field of graphic data reading and processing, and in particular to a method for detecting abnormal behavior of people around a river based on deep learning image processing. Background Art

[0002] With the acceleration of urbanization, safety issues in river areas have become increasingly prominent. As an important part of public safety, abnormal behaviors around rivers (such as climbing guardrails, falling into rivers, etc.) not only threaten people's lives, but may also cause secondary disasters or accidents. Traditional river safety management methods usually rely on manual patrols and real-time observation by monitoring personnel, but such methods are inefficient, limited by human resources, and prone to underreporting due to negligence, making it difficult to meet the needs of modern safety management. With the breakthrough of deep learning technology in the field of computer vision, automated abnormal behavior detection methods based on deep learning have begun to become a key technology to solve this problem.

[0003] In recent years, the rapid development of deep learning technology, especially breakthroughs in the field of computer vision, has provided new technical support for security monitoring. Deep learning models based on target detection have been widely used in many fields such as traffic monitoring and security systems due to their efficient end-to-end detection capabilities. They can achieve high-precision target recognition and positioning in complex backgrounds, providing a technical basis for real-time detection of abnormal behaviors around rivers. However, the complexity of river scenes (such as light changes, frequent occlusions, water surface reflections, etc.) puts higher requirements on the robustness of the detection model. It is difficult to accurately identify dynamic behavior characteristics and potential abnormal patterns by relying solely on single-frame target detection. Therefore, traditional target detection methods face the challenge of adapting to complex scenes and dynamic changes. Although there are some abnormal behavior recognition methods based on target detection, they usually rely on pre-defined behavior categories and cannot effectively identify unknown behavior types that are not in the training set, resulting in low detection accuracy. Therefore, a method for detecting abnormal behavior of people around rivers based on deep learning image processing is developed to solve the above problems. Summary of the invention

[0004] The present invention proposes a method for detecting abnormal behavior of people around a river based on deep learning image processing, so as to solve the problem that the prior art cannot effectively identify unknown behavior types that are not in the training set, resulting in low detection accuracy.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The present invention provides a method for detecting abnormal behavior of people around a river based on deep learning image processing, comprising:

[0007] Acquire first information and second information, wherein the first information is a training data set with labels, the training data set includes historical monitoring image data around the river, the labels include normal behavior labels and abnormal behavior labels, and the second information includes real-time monitoring video data around the river;

[0008] Preprocessing the first information to obtain preprocessed data;

[0009] Training a preset self-supervised key point extraction network according to the preprocessed data, including inputting the preprocessed data into a preset target detection model, outputting the personnel target features of the surrounding area, and training the target detection model based on an intersection-over-union loss function, inputting the preprocessed data into a preset key point detection model, outputting the personnel dynamic features, and training the key point detection model based on a key point coordinate loss function, inputting the personnel target features and the personnel dynamic features as input samples into a preset multimodal self-supervised learning network to generate an embedding vector for each data, and training the multimodal self-supervised learning network based on a contrastive learning method;

[0010] Input the preprocessed data into the multimodal self-supervised learning network, output a behavior vector library, classify and label the embedded vectors in the behavior vector library according to the label, calculate the distance between the embedded vector corresponding to each normal behavior and the embedded vector corresponding to each abnormal behavior, and sort all the distances from small to large to obtain a distance discriminant set;

[0011] The second information is preprocessed and then input into the trained self-supervised key point extraction network, the corresponding real-time embedding vector is output, the distance between the real-time embedding vector and the embedding vector corresponding to the abnormal behavior is calculated, the ranking ratio of the distance in the distance discrimination set is calculated, and the ranking ratio is compared with a preset alarm threshold. If the ranking ratio is less than or equal to the alarm threshold, an abnormal behavior alarm is issued.

[0012] Further, obtaining the second information includes:

[0013] Obtain video data recorded continuously for 72 hours by two high-definition infrared surveillance cameras, with a resolution of 2560×1440, a viewing angle of 110°, an installation height of 3.5 meters, and an inclination of 15°;

[0014] Extract key frames from the video data based on the FFmpeg program module, extract 2 frames of images per second, and generate a standardized image data set of 640×640 pixels;

[0015] Gaussian filtering and denoising are performed on the standardized image data set, and the target is annotated using the LabelImg tool to generate an XML annotation file to obtain the training data set with labels.

[0016] Furthermore, the pre-processing step comprises:

[0017] Performing image correlation analysis on initial image data to remove low-quality data to obtain a key frame image sequence, wherein the initial image data is the first information or the second information;

[0018] The Laplace operator variance method is used to calculate the clarity score of each image in the key frame image sequence, and the images whose clarity scores are lower than a preset threshold are marked as low-quality frames and removed to obtain clear image data;

[0019] Using a contrast-limited adaptive histogram equalization method to optimize each frame of the clear image data to obtain an optimized image;

[0020] The optimized image is processed based on an illumination normalization model of a convolutional neural network.

[0021] Further, according to the pre-processed data, the preset target detection model is input, the target features of the people in the surrounding area are output, and the target detection model is trained based on the intersection-over-union loss function, including:

[0022] The preprocessed data is divided into a training set and a validation set in a ratio of 80%:20%;

[0023] Construct a target detection model, which includes a resnet50 backbone network, an FPN feature pyramid network and a detection head. The target detection model is used to first extract features from the input data through the resnet50 backbone network, then perform multi-scale feature fusion on the extracted features through the FPN feature pyramid network, and finally input the fused multi-scale features into the detection head to perform target positioning and output the detection results;

[0024] According to the training set, the constructed target detection model is trained to a specified round based on the intersection-over-union loss function and the data enhancement strategy, and the intersection-over-union loss function is:

[0025]

[0026]

[0027] in, and are the predicted bounding box and the true bounding box respectively, is the number of detected targets, is the intersection-and-union ratio, is the weight factor, Area of ​​Overlap represents the overlapping area of ​​the predicted bounding box and the true bounding box, Area of ​​Union represents the total area of ​​the predicted bounding box and the true bounding box, and the IoU value range is [0, 1].

[0028] Furthermore, the key point coordinate loss function is:

[0029]

[0030] in, is the detected human key point coordinates, The coordinates of the key points of the annotation.

[0031] Furthermore, the personnel target features and the personnel dynamic features are input as input samples into a preset multimodal self-supervised learning network to generate an embedding vector for each data, and the multimodal self-supervised learning network is trained based on a contrastive learning method, including:

[0032] Constructing a multimodal self-supervised learning network, the multimodal self-supervised learning network includes an image feature extraction module, a key point coordinate extraction module, a feature fusion module, and an embedding vector module, the image feature extraction module is used to extract image feature vectors for the personnel target features, the key point coordinate extraction module is used to extract key point feature vectors for the personnel dynamic features, the feature fusion module is used to fuse the image feature vectors and the key point feature vectors, the embedding vector module is used to embed features for the fused vectors, and the embedding vector module includes two fully connected layers;

[0033] Based on the data augmentation method, the input samples are enhanced in multiple different ways, including rotation, translation, flipping and random occlusion;

[0034] Performing multiple different augmentations on each input sample to generate a positive sample pair, and all other samples in the training set constitute negative sample pairs with it, and optimizing the multimodal self-supervised learning network using a contrastive loss function;

[0035] The contrastive loss function is as follows:

[0036]

[0037] in, and are the embedding vectors of the positive sample pairs, is the embedding vector of the negative sample pair, represents the cosine similarity between embedding vectors, is the temperature hyperparameter, which controls the sensitivity of distance;

[0038] During the training process, when the contrast loss function reaches convergence, the multimodal self-supervised learning network training is completed.

[0039] Furthermore, the distance between the embedding vector corresponding to each normal behavior and the embedding vector corresponding to each abnormal behavior is calculated. The calculation formula is as follows:

[0040]

[0041]

[0042] Embedding vector representing normal behavior, represents the embedding vector of abnormal behavior, j represents the index of the abnormal behavior embedding vector, and M represents the total number of abnormal behavior embedding vectors.

[0043] The beneficial effects of the present invention are:

[0044] The method for detecting abnormal behavior of people around rivers based on deep learning image processing proposed in the present invention adopts a self-supervised algorithm, combined with a multimodal feature extraction network, and analyzes the image and dynamic behavior characteristics of people through feature embedding. This method uses the behavior vector library and distance discriminant set constructed during the training process to realize the judgment of all behavior types, regardless of whether they belong to known categories. Through self-supervised learning, the behavior category is judged by the behavior embedding vector instead of the known behavior labeled in the training set, thereby enhancing the generalization ability and adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a method for detecting abnormal behavior of people around a river based on deep learning image processing according to the present invention.

[0046] Figure 2 This is the algorithm flow of step 3 in the embodiment of the present invention.

[0047] Figure 3 This is the self-supervisory model structure of step 3 in the embodiment of the present invention.

[0048] Figure 4 This is the algorithm flow of step 5 in the embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0052] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0053] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, a flow chart of a method for detecting abnormal behavior of people around a river based on deep learning image processing of the present invention specifically includes:

[0055] S1: Acquire first information and second information, wherein the first information is a training data set with labels, the training data set includes historical monitoring image data around the river, the labels include normal behavior labels and abnormal behavior labels, and the second information includes real-time monitoring video data around the river;

[0056] S2: preprocessing the first information to obtain preprocessed data;

[0057] S3: training a preset self-supervised key point extraction network according to the preprocessed data, including inputting the preprocessed data into a preset target detection model, outputting the personnel target features of the surrounding area, and training the target detection model based on the intersection-over-union loss function, inputting the preprocessed data into a preset key point detection model, outputting the personnel dynamic features, and training the key point detection model based on the key point coordinate loss function, inputting the personnel target features and the personnel dynamic features as input samples into a preset multimodal self-supervised learning network to generate an embedding vector for each data, and training the multimodal self-supervised learning network based on a contrastive learning method;

[0058] S4: inputting the preprocessed data into the multimodal self-supervised learning network, outputting a behavior vector library, classifying and labeling the embedded vectors in the behavior vector library according to the labels, calculating the distance between the embedded vector corresponding to each normal behavior and the embedded vector corresponding to each abnormal behavior, and sorting all the distances from small to large to obtain a distance discriminant set;

[0059] S5: Preprocess the second information and input it into the trained self-supervised key point extraction network, output the corresponding real-time embedding vector, calculate the distance between the real-time embedding vector and the embedding vector corresponding to the abnormal behavior, calculate the ranking ratio of the distance in the distance discrimination set, compare the ranking ratio with the preset alarm threshold, and if the ranking ratio is less than or equal to the alarm threshold, issue an abnormal behavior alarm.

[0060] The specific implementation steps according to the present invention are as follows:

[0061] Step 1: Select two Hikvision DS-2CD2T47G1 high-definition cameras with a resolution of 2560×1440, supporting night infrared monitoring and a viewing angle of 110°. Each camera is equipped with a waterproof housing and a shockproof bracket to ensure long-term outdoor environment. The camera is installed at a height of 3.5 meters and the tilt angle is set to 15°. The laser rangefinder is used to ensure that the camera covers all blind spots. 72 hours of uninterrupted video recording is performed in MP4 format with a frame rate of 25 frames per second and stored on a 2TB SSD hard drive. FFmpeg is used to extract key frames in the video, extract 2 frames of images per second, and generate a standardized image dataset of 640×640 pixels. Gaussian filtering and denoising are performed on the image, and the LabelImg tool is used to annotate the target and generate an XML annotation file. Finally, about 100,000 frames of image data are generated, providing a basis for subsequent model training.

[0062] Step 2: Preprocess the collected raw video data to improve data quality and convert the data into a standard format suitable for model training. Specifically, it includes:

[0063] Step 2.1: Remove low-quality data based on the correlation of the collected images. By analyzing the collected key frame image sequence, first use the image clarity evaluation algorithm to score the clarity of each frame. The Laplace operator variance method is used to calculate the clarity score C of the image. The clarity is lower than the preset threshold ( ) will be marked as low-quality frames and removed. The image quality is improved by using the image optimization algorithm of contrast analysis. The contrast-limited adaptive histogram equalization (CLAHE) method is used to optimize each frame of the image, enhancing the image contrast while avoiding the loss of details caused by over-enhancement.

[0064] Step 2.2: Build a de-illumination normalization neural network to eliminate the interference of illumination changes. To address the problem of non-uniform illumination caused by strong light or shadows in some images, a convolutional neural network (CNN)-based illumination normalization model is used to process the data using an existing publicly available trained algorithm to enhance the contrast and clarity of the target area.

[0065] Step 3: If Figure 2 As shown in the figure, the self-supervised key point extraction network is used to locate the person in the image and embed the behavior features, and train the target detection model. Specifically, it includes:

[0066] Step 3.1: Based on the image dataset and annotation files generated in the first two steps, the data is divided into a training set and a validation set at a ratio of 80%:20%. The target detection network consists of a resnet50 backbone network, an FPN feature pyramid network, and a detection head. Key point detection uses the trained human key points to extract the skeleton key points of the target area. After that, a multimodal feature extraction network is used to obtain the behavior vector embedding.

[0067] During training, the loss function of the target detection algorithm is as follows:

[0068] Bounding Box Loss:

[0069]

[0070] in, and The predicted bounding box and the true bounding box respectively; is the number of targets detected; is a weight factor; used to assign different weights to different targets; is the intersection over union ratio (Intersection over Union), and the calculation formula is as follows:

[0071]

[0072] Among them, Area of ​​Overlap represents the area of ​​the overlapping area between the predicted bounding box and the true bounding box, Area of ​​Union represents the total area of ​​the predicted bounding box and the true bounding box (excluding the overlapping part), and the value range of IoU is [0, 1]. The closer the value is to 1, the more similar the two boxes are.

[0073] Data enhancement strategies (such as rotation, cropping, brightness adjustment, and random occlusion simulation) are added during the training process to improve the model's generalization ability for complex scenes.

[0074] Use the training data to train for the specified number of rounds to make the model converge and obtain a trained target detection algorithm model.

[0075] The loss function of the key point detection model is:

[0076]

[0077] in, are the detected human key point coordinates (head, hands, center point, feet), The coordinates of the key points of the annotation.

[0078] Step 3.2: If Figure 3 As shown, considering that the data obtained by the above model includes both personnel image data and key point coordinate data, a single model is difficult to process these two data types. Therefore, the present invention constructs a multimodal self-supervised learning network for feature extraction and embedding. The structure of the network is as follows:

[0079] Image feature extraction module: ResNet18, input 224*224*3 data, output 64-dimensional features ;

[0080] Key point coordinate extraction module: Input the key point feature vector with a length of 12, and after passing through two layers of fully connected layers (the first layer outputs 32 dimensions, using ReLU activation; the second layer outputs 64 dimensions), a 64-dimensional feature vector is obtained. ;

[0081] Feature fusion module: image feature vector and the key point feature vector Splice to get a 128-dimensional fusion feature vector .

[0082] Embedding vector module: input 128-dimensional fused feature vector , through two layers of fully connected layers (the first layer outputs 128 dimensions and uses ReLU activation; the second layer outputs 256 dimensions), the final embedding vector e is obtained to represent the feature embedding of the input image and key point data.

[0083] In order to construct an effective self-supervised learning task, the present invention adopts data enhancement methods, including rotation, translation, flipping and random occlusion, to perform multiple different enhancements on each input sample to generate positive sample pairs. All other samples are used as negative sample pairs, and contrastive loss is used to optimize the embedding space. The specific contrastive loss function is as follows:

[0084]

[0085] in, and are the embedding vectors of the positive sample pairs, is the embedding vector of the negative sample pair, represents the cosine similarity between embedding vectors, is the temperature hyperparameter that controls the sensitivity to distance.

[0086] During the training process, when the loss function reaches convergence, that is, when the model no longer decreases further, the training is completed. At this time, the trained model is used to process the data obtained in step 4.1 again, and all the generated embedding vectors are saved. At the same time, the embedding vectors are annotated with the labels in step 1, and finally the behavior vector library is obtained.

[0087] Step 3.3: Calculate the L2 distance between the embedding vector of each normal behavior and the embedding vector of all abnormal behaviors, and take the average of the L2 distances of all abnormal behaviors. The specific steps are as follows:

[0088] Assume that the behavior vector library of normal behavior is , the behavior vector library of abnormal behavior is .

[0089] Embedding vector for each normal behavior , calculate its embedding vector with all abnormal behaviors The L2 distance between:

[0090]

[0091] in, Represents the L2 norm, i.e., the Euclidean distance, between two embedding vectors.

[0092] For each normal behavior embedding vector, calculate the average L2 distance from the embedding vectors of all abnormal behaviors:

[0093]

[0094] After calculating the average L2 distances from the embedding vectors of all normal behaviors to the embedding vectors of abnormal behaviors, these average distances are sorted in ascending order to obtain a ranking set of average distances from normal behaviors to abnormal behaviors:

[0095]

[0096] Finally, the average L2 distance between each normal behavior embedding vector and all abnormal behavior embedding vectors is obtained and arranged in ascending order as the distance discriminant set.

[0097] Step 4: Deploy the optimized target detection and human key point detection models to the monitoring equipment, integrate them with the real-time video stream, improve the detection and analysis efficiency through hardware acceleration or edge computing, and ensure real-time response. After training, the optimized target detection model and human key point detection module are exported to the ONNX (Open Neural Network Exchange) format so that they can run efficiently on hardware devices. During the deployment process, the model is loaded into the NVIDIA Jetson Xavier NX edge computing device, which is equipped with a 6-core CPU, a 384-core GPU, and 8GB of memory, which can meet the high-performance requirements of real-time reasoning and key point analysis. The present invention integrates a real-time video stream processing module, and the video stream is transmitted via the RTSP protocol and decoded using OpenCV.

[0098] Step 5: Figure 4 As shown in the figure, the deployed model is used to detect, analyze and embed people in the real-time monitoring screen, generate behavioral feature data, and compare it with the behavior vector library in the database to determine whether the behavior is abnormal. The deployed target detection model receives the real-time monitoring video stream, and each frame is detected by the target detection model to generate the bounding box coordinates of the target. ,in and is the center point coordinate of the target bounding box, and is the width and height of the bounding box. At the same time, the human key point detection module extracts the key point coordinates , where each Indicates key points and Then, the image of the person and the coordinates of the key points of the human body are input into the self-supervised learning network designed by the invention to generate a real-time embedding vector and compare it with the behavior vector library obtained in step 3.2 The L2 distance is calculated by embedding vectors of all abnormal behaviors in . The specific L2 distance calculation formula is:

[0099]

[0100] in, ' is the real-time embedding vector, is the embedding vector of abnormal behavior.

[0101] Calculate the distance In the distance discriminant set The sorting ratio in is compared with a preset alarm threshold θ. If the sorting ratio is less than or equal to the alarm threshold, the behavior is determined to be abnormal behavior and an alarm is triggered.

[0102] Step 6: All relevant data of abnormal behavior is saved in real time, including the location of the person , Person Image , behavioral feature embedding vector ', key point coordinates , Detection timestamp , and abnormal behavior determination results (in Indicates abnormal behavior. Indicates normal behavior) According to the abnormal judgment results and the real-time monitoring screen, the saved data is carefully reviewed and marked to see whether it is indeed abnormal behavior. The annotations in the original dataset will be updated and saved together with the original data as a new dataset. Each annotation result will contain detailed review comments, corrected behavior categories, and relevant time information to ensure the accuracy and completeness of the data. Ultimately, the corrected and annotated results will form a new dataset. , the dataset will be used for subsequent model training, verification, optimization and continuous performance improvement.

[0103] Step 7: Regularly summarize and analyze the detection results to identify and summarize the time and location distribution of abnormal behaviors. For example, by analyzing the time period when abnormal behaviors occur and location area , identifying which time periods and location areas frequently experience abnormal behaviors. This information will be used to troubleshoot the influence of non-human factors, such as changes in ambient lighting, equipment failures, or other external factors, thereby optimizing the stability and accuracy of the self-supervised key point extraction network.

[0104] At the same time, when the number of new data sets reaches a certain level, these data are used to fine-tune the multimodal self-supervised learning network to further improve the accuracy and adaptability of the model. Specifically, the new data sets It will be input into the self-supervised network for updating and training, re-execute the embedding vector generation process in step 3.2, and recalculate the behavior vector library and abnormal behavior embedding vector based on the new data. Then, update the distance discriminant set in step 3.3 , by sorting and adjusting the threshold, the detection accuracy and generalization ability of abnormal behavior are optimized. Through this periodic fine-tuning and updating process, the self-supervised key point extraction network will continuously improve its abnormal behavior detection ability, ensure that it can adapt to changes in different environments and behavior patterns, and thus enhance the recognition accuracy of unknown or unseen abnormal behaviors.

[0105] In summary, the present invention realizes accurate detection of human targets in the river surrounding area, and identifies abnormal behaviors by dynamic analysis of skeleton key points and calculation of behavior embedding vectors. By using the target detection model and the behavior embedding vector generated by self-supervised learning, the present invention can perform real-time abnormal behavior detection and alarm in complex river surrounding scenes. Through the combination of target detection and behavior embedding, not only can the target be accurately located, but also the movement pattern, behavior characteristics and dynamic changes of the target can be captured in real time, and abnormal behaviors can be effectively identified. This technology shows high robustness in the face of complex environments such as light changes, target occlusion and rapid movement, meeting the dual requirements of real-time and accuracy. Through multiple optimization strategies such as key frame extraction, image preprocessing and key point sequence analysis, the data quality and model adaptability are further improved. Combined with the self-supervised learning method of behavior embedding, it can continuously adapt to new abnormal behavior patterns by learning the embedding vector of normal behavior, ensuring the stability and accuracy of the self-supervised key point extraction network in practical applications. This technology provides efficient and stable technical support for abnormal behavior identification, and can reliably perform real-time alarms and data recording in complex environments.

[0106] The present invention proposes a complete technical solution for safety management around rivers, which comprehensively uses technologies such as software and hardware collaborative design, real-time video stream processing, target detection, dynamic behavior analysis, and behavior embedding vectors generated by self-supervised learning, significantly improving the stability and efficiency of the self-supervised key point extraction network. Combined with the intelligent recognition of the dynamic characteristics of human key points and the automatic alarm mechanism, the self-supervised key point extraction network can efficiently adapt to the needs of different scenarios. It has important value in reducing safety hazards and improving the level of public safety management, and can cope with common complex factors in the river environment (such as light changes, obstructions, etc.). The present invention not only improves the intelligent and refined management capabilities of the monitoring system, but also continuously expands the recognition capabilities of new abnormal behaviors to ensure the comprehensiveness and accuracy of public safety management. This technical solution is widely applicable to fields such as river safety monitoring and public safety protection, and provides a strong technical guarantee for the realization of intelligent and refined safety management.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting abnormal behavior of people around rivers based on deep learning image processing, characterized in that: include: Acquire first information and second information, wherein the first information is a training data set with labels, the training data set includes historical monitoring image data around the river, the labels include normal behavior labels and abnormal behavior labels, and the second information includes real-time monitoring video data around the river; Preprocessing the first information to obtain preprocessed data; Training a preset self-supervised key point extraction network according to the preprocessed data, including inputting the preprocessed data into a preset target detection model, outputting the personnel target features of the surrounding area, and training the target detection model based on the intersection-over-union loss function, inputting the preprocessed data into a preset key point detection model, outputting the personnel dynamic features, and training the key point detection model based on the key point coordinate loss function, inputting the personnel target features and the personnel dynamic features as input samples into a preset multimodal self-supervised learning network to generate an embedding vector for each data, and training the multimodal self-supervised learning network based on a contrastive learning method; Input the preprocessed data into the multimodal self-supervised learning network, output a behavior vector library, classify and label the embedded vectors in the behavior vector library according to the label, calculate the distance between the embedded vector corresponding to each normal behavior and the embedded vector corresponding to each abnormal behavior, and sort all the distances from small to large to obtain a distance discriminant set; The second information is preprocessed and then input into the trained self-supervised key point extraction network, the corresponding real-time embedding vector is output, the distance between the real-time embedding vector and the embedding vector corresponding to the abnormal behavior is calculated, the ranking ratio of the distance in the distance discrimination set is calculated, and the ranking ratio is compared with a preset alarm threshold. If the ranking ratio is less than or equal to the alarm threshold, an abnormal behavior alarm is issued.

2. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: Acquiring the second information includes: Obtain video data recorded continuously for 72 hours by two high-definition infrared surveillance cameras, with a resolution of 2560×1440, a viewing angle of 110°, an installation height of 3.5 meters, and an inclination of 15°; Extract key frames from the video data based on the FFmpeg program module, extract 2 frames of images per second, and generate a standardized image data set of 640×640 pixels; Gaussian filtering and denoising are performed on the standardized image data set, and the target is annotated using the LabelImg tool to generate an XML annotation file to obtain the training data set with labels.

3. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: The pre-processing step comprises: Performing image correlation analysis on initial image data to remove low-quality data to obtain a key frame image sequence, wherein the initial image data is the first information or the second information; The Laplace operator variance method is used to calculate the clarity score of each image in the key frame image sequence, and the images whose clarity scores are lower than a preset threshold are marked as low-quality frames and removed to obtain clear image data; Using a contrast-limited adaptive histogram equalization method to optimize each frame of the clear image data to obtain an optimized image; The optimized image is processed based on an illumination normalization model of a convolutional neural network.

4. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: The preprocessed data is input into a preset target detection model, the target features of people in the surrounding area are output, and the target detection model is trained based on the intersection-over-union loss function, including: The preprocessed data is divided into a training set and a validation set in a ratio of 80%:20%; Construct a target detection model, which includes a resnet50 backbone network, an FPN feature pyramid network and a detection head. The target detection model is used to first extract features from the input data through the resnet50 backbone network, then perform multi-scale feature fusion on the extracted features through the FPN feature pyramid network, and finally input the fused multi-scale features into the detection head to perform target positioning and output the detection results; According to the training set, the constructed target detection model is trained to a specified round based on the intersection-over-union loss function and the data enhancement strategy. for: ， ， in, and are the predicted bounding box and the true bounding box respectively, is the number of detected targets, is the intersection-and-union ratio, is the weight factor, Area of ​​Overlap represents the overlapping area of ​​the predicted bounding box and the true bounding box, Area of ​​Union represents the total area of ​​the predicted bounding box and the true bounding box, and the IoU value range is [0, 1].

5. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: The key point coordinate loss function for: ， in, is the detected human key point coordinates, The coordinates of the key points of the annotation.

6. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: Inputting the personnel target features and the personnel dynamic features as input samples into a preset multimodal self-supervised learning network to generate an embedding vector for each data, and training the multimodal self-supervised learning network based on a contrastive learning method, including: Constructing a multimodal self-supervised learning network, the multimodal self-supervised learning network includes an image feature extraction module, a key point coordinate extraction module, a feature fusion module, and an embedding vector module, the image feature extraction module is used to extract image feature vectors for the personnel target features, the key point coordinate extraction module is used to extract key point feature vectors for the personnel dynamic features, the feature fusion module is used to fuse the image feature vectors and the key point feature vectors, the embedding vector module is used to embed features for the fused vectors, and the embedding vector module includes two fully connected layers; Based on the data augmentation method, the input samples are enhanced in multiple different ways, including rotation, translation, flipping and random occlusion; Performing multiple different augmentations on each input sample to generate a positive sample pair, and all other samples in the training set constitute negative sample pairs with it, and optimizing the multimodal self-supervised learning network using a contrastive loss function; Contrastive loss function as follows: ， in, and are the embedding vectors of the positive sample pairs, is the embedding vector of the negative sample pair, represents the cosine similarity between embedding vectors, is the temperature hyperparameter, which controls the sensitivity of distance; During the training process, when the contrast loss function reaches convergence, the multimodal self-supervised learning network training is completed.

7. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: Calculate the distance between the embedding vector corresponding to each normal behavior and the embedding vector corresponding to each abnormal behavior The calculation formula is as follows: ， ， in, Embedding vector representing normal behavior, represents the embedding vector of abnormal behavior, j represents the index of the abnormal behavior embedding vector, and M represents the total number of abnormal behavior embedding vectors.

8. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 1 is characterized in that: It also includes real-time storage of all data determined to be abnormal behavior, including personnel location, personnel image, behavior characteristics and system time, marking whether it is indeed abnormal behavior, and saving the marking results and data as a new data set.

9. The method for detecting abnormal behavior of people around a river based on deep learning image processing according to claim 8 is characterized in that: It also includes fine-tuning the self-supervised key point extraction network using a new dataset.

Images