An intelligent behavior analysis method and system based on multi-visual recognition algorithm

Abstract

The present invention belongs to the field of image information processing technology and discloses an intelligent behavior analysis method and system based on a multi-vision recognition algorithm. The present invention uses a multi-vision acquisition module to acquire three-dimensional images, a multi-target detection module to extract feature maps of the three-dimensional images, and a data processing module to perform denoising, filtering, and normalization on the three-dimensional images of the human body, as well as image enhancement and correction. A behavior analysis module determines whether the human body exhibits abnormal behavior, a repeated behavior filtering module filters repeated abnormal behaviors, and an alarm and feedback module is used to issue alarm information for abnormal behaviors. The collected three-dimensional images are then subjected to population counting, flow analysis, and aggregation analysis. The present invention can quickly identify abnormal human behavior and issue early warning signals in a timely manner, providing comprehensive support for safety monitoring, behavior analysis, and situation prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and specifically relates to an intelligent behavior analysis method and system based on a multi-visual recognition algorithm. Background Art

[0002] In modern society, the rapid advancement of technology and the growing demand for intelligent applications have made the accurate analysis and understanding of human behavior increasingly critical in many fields. Traditional behavioral analysis techniques often rely on a single visual data source or limited feature extraction methods, which limits the accuracy and comprehensiveness of the analysis results.

[0003] In the construction site monitoring fields of water conservancy, hydropower, wind power, and solar power generation, previous monitoring systems typically relied on fixed-angle cameras to capture limited scene information, making it difficult to fully capture human behavior in complex environments. Worker monitoring often relied on manual observation or basic image recognition technology, making it difficult to promptly identify potential irregularities and safety hazards. Furthermore, existing technologies, when processing large amounts of visual data, suffer from low recognition efficiency, high error rates, and insufficient real-time performance, failing to meet the demand for rapid and accurate behavior analysis. Summary of the Invention

[0004] In order to overcome the limitations of existing technologies and improve the accuracy, comprehensiveness and real-time performance of behavior analysis, it is particularly important to develop an intelligent behavior analysis method and system based on multi-visual recognition algorithm, which has significant practical significance and application value.

[0005] With the continuous advancement of computer vision technology and deep learning algorithms, multi-object detection algorithms and multi-viewing technologies have made significant progress. Multi-object detection algorithms such as YOLO and SSD use convolutional neural networks to directly predict the bounding box and category of an object, offering the advantages of low computational complexity and good real-time performance. Multi-viewing technology, on the other hand, utilizes multiple cameras or sensors to capture multiple perspectives of a scene. Through image fusion and 3D reconstruction, it achieves comprehensive perception and understanding of the scene. Combining these two technologies can significantly improve the accuracy and reliability of behavioral analysis.

[0006] The present invention aims to solve the technical problems existing in the prior art and provide an intelligent behavior analysis method and system based on multi-visual recognition algorithms that can quickly identify abnormal human behavior and issue early warning signals in a timely manner, providing comprehensive support for safety monitoring, behavior analysis and situation prediction.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] The present invention provides an intelligent behavior analysis method based on a multi-visual recognition algorithm, comprising the following steps:

[0009] S1. Using an image acquisition device to capture images from different perspectives, extracting image feature points from different perspectives, matching the image feature points from different perspectives, and calculating disparity and depth based on the positional differences of the successfully matched feature points in different images to obtain a three-dimensional image.

[0010] S2. Normalize the three-dimensional image, extract features of the three-dimensional image, obtain a feature map of the three-dimensional image, and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0011] S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image;

[0012] S4, performing human posture judgment, gait analysis, and face recognition on the three-dimensional human body image obtained in step S3, and determining whether the human body has abnormal behavior;

[0013] S5. Detect and record abnormal behaviors of the human body in the three-dimensional image, identify repeated abnormal behaviors that occur multiple times within a preset time period, filter out repeated behaviors, and determine abnormal behaviors as abnormal;

[0014] S6. For the abnormal behavior determined to be abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded.

[0015] Preferably, in step S1, before collecting images from different perspectives, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and label the multi-visual data and record the location information, behavior labels are set for the multi-visual data, and multi-visual data with behavior labels of abnormal human behavior are screened to construct an abnormal human behavior database.

[0016] Preferably, in step S1, a scale space is constructed by a scale space feature extraction algorithm, and the scale space feature extraction algorithm is used to extract image feature points at different viewing angles.

[0017] Preferably, the scale space feature extraction algorithm is one of the SIFT algorithm, the SURF algorithm, and the ORB algorithm.

[0018] Preferably, in step S1, the image feature points under different perspectives are matched by a brute force matching algorithm, the Euclidean distance of the image feature points under different perspectives is calculated by the brute force matching algorithm to find the matching feature points, and the successfully matched feature points are obtained. The disparity and depth are calculated based on the position difference of the successfully matched feature points in different images, and the three-dimensional position of the target object corresponding to the image feature points is further determined using the triangulation formula to determine the three-dimensional image of the image feature points under different perspectives.

[0019] Preferably, in step S2, the three-dimensional image obtained in step S1 is first resized and then normalized so that the pixel values of the three-dimensional image are within a preset range. Assuming that the pixel values of the original three-dimensional image are $I(x,y)$, the pixel values after normalization are $I_{norm}(x,y)=\frac{I(x,y)}{255}$;

[0020] A deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$.

[0021] Preferably, in step S2, the target detection problem is converted into a regression problem using the YOLO algorithm, a grid is divided on the feature map, and each grid predicts the confidence of multiple bounding boxes;

[0022] The SSD algorithm is used to perform target detection on multiple feature maps. Each feature map corresponds to a different scale. The feature maps of different scales are combined to set the ground truth values of bounding boxes of different sizes for target detection to obtain a three-dimensional image of the human body.

[0023] Preferably, in step S2, the bounding boxes are sorted according to the confidence level, the bounding box with the highest confidence level is selected, and the overlap between the bounding box with the highest confidence level and other bounding boxes is calculated. If the overlap exceeds a threshold, the bounding box is removed.

[0024] Preferably, in step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image of the human body obtained in step S2.

[0025] Preferably, in step S3, the three-dimensional human body image obtained in step S2 is denoised using a median filtering algorithm, and filtered using a mean filtering algorithm to smooth the image and highlight important features, and the data value range of the denoised and filtered three-dimensional human body image is normalized.

[0026] Preferably, in step S3, the contrast of the three-dimensional image is adjusted using a histogram equalization algorithm; for adjusting the brightness of the three-dimensional image, assuming that the original brightness is $L$, the adjustment value is $\Delta L$, and the new brightness is $New\Brightness=(L+\Delta L)\mod\256$; the color saturation of the three-dimensional image is adjusted by changing the saturation component in the color space to perform image enhancement and correction.

[0027] Preferably, step S4 specifically includes:

[0028] Convolutional neural network is used for feature extraction, and coordinate prediction is performed through the fully connected layer of the convolutional neural network to determine the position of human joints;

[0029] A convolutional neural network is used to extract the motion characteristics of the human body, and the optical flow algorithm is used to calculate the motion speed and direction of pixels in the three-dimensional image to obtain the motion trajectory of the human body.

[0030] A convolutional neural network is used to extract facial features, which are then compared with known faces in the database to determine the person's identity.

[0031] The human body posture is judged based on the position of the human joints, gait analysis is performed based on the human body's movement trajectory, and face recognition is performed based on facial feature comparison. The human body posture, gait analysis and face recognition are combined to determine whether the human body has abnormal behavior.

[0032] Preferably, in step S5, the abnormal behavior of the human body is continuously monitored. Assume that the set of abnormal behavior events is

[0033] $E={e_1,e_2,…,e_n}$, identifies abnormal behavior events that occur multiple times within a preset time by comparing behavioral features. Suppose an abnormal behavior event that occurs within the preset time is $e_i$, and another abnormal behavior event is $e_j$. If the feature similarity between the abnormal behavior event $e_i$ and the other abnormal behavior event $e_j$ exceeds a preset value, then it is considered a repeated behavior;

[0034] Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of abnormal behavior events, and obtain the set of behavior events actually judged to be abnormal. The set of behavior events actually judged to be abnormal is $E'=ER$.

[0035] Preferably, the three-dimensional image collected in step S1 is subjected to population counting, flow direction analysis and aggregation analysis.

[0036] Preferably, a human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1. Assume that the video frame of the three-dimensional image is $F$, the human body detection algorithm is $D$, the number of people is $N$, $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at the coordinate $(x,y)$.

[0037] Use the human body detection algorithm to analyze the gait obtained in step S4, perform flow analysis on the movement direction and speed, and derive the overall flow trend. Assume that the position sequence of the human body is ${(x_1,y_1),(x_2,y_2),…,(x_t,y_t)}$, the movement direction is $\vec{d}=(x_t-x_1,y_t-y_1)$, and the movement speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$;

[0038] The human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1, and the crowd density is calculated using the crowd density calculation method to obtain the crowd-dense area for aggregation analysis.

[0039] The present invention also provides a system for performing intelligent behavior analysis using the intelligent behavior analysis method based on the multi-visual recognition algorithm as described above, comprising a multi-vision acquisition module, a multi-target detection module, a data processing module, a behavior analysis module, a repeated behavior filtering module, and an alarm and feedback module;

[0040] The multi-viewing acquisition module is used to collect images from different perspectives, extract feature points, match the feature points of images from different perspectives, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image;

[0041] The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0042] The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image of the human body;

[0043] The behavior analysis module is used for human posture judgment, gait analysis and face recognition in three-dimensional human images, and to determine whether the human body has abnormal behavior;

[0044] The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal;

[0045] The alarm and feedback module is used to issue alarm information for abnormal behavior.

[0046] Preferably, the multi-viewing acquisition module includes a situation prediction module, which is used to perform headcount, flow analysis and aggregation analysis on the acquired three-dimensional images.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention can quickly identify abnormal human behavior, avoid potential risk factors, issue early warning signals in the first place, and record on-site conditions, thereby achieving comprehensive monitoring and analysis of human behavior in images, thereby improving the efficiency and accuracy of security monitoring, facilitating subsequent analysis and processing, significantly improving the efficiency of security management, and significantly saving labor costs;

[0049] (2) The present invention uses an intelligent behavior analysis method based on a multi-visual recognition algorithm to achieve real-time monitoring and risk warning of construction sites such as water conservancy and hydropower, wind power, and solar power generation, ensuring the safety and order of the construction sites. In addition, the intelligent behavior analysis system based on the multi-visual recognition algorithm is not limited to construction site management, but can also be widely used in communities, streets, schools, hospitals, and many other application scenarios.

[0050] (3) The present invention integrates multi-viewing technology and multi-target detection algorithms to achieve automatic recognition, tracking and analysis of human behavior in surveillance videos, thereby providing comprehensive support for safety monitoring, behavior analysis and situation prediction. By automatically identifying targets in three-dimensional images and predicting their positions and bounding boxes, accurate tracking of targets is achieved. In addition, pedestrians can be classified according to the attributes of their helmets, for example, they can be divided into construction personnel, supervisors, owner personnel, etc., thereby achieving more detailed and accurate personnel classification management.

[0051] (4) The present invention removes noise and interference from images by performing denoising, filtering, and normalization on images during the data preprocessing stage, thereby improving the clarity and accuracy of the data. Image enhancement and correction technologies further enhance the visual effect and accuracy, ensuring that the system can more accurately identify and analyze targets and behaviors in images, thereby improving data quality and usability.

[0052] (5) The present invention uses a behavior analysis module to perform human posture judgment, gait analysis, and face recognition on three-dimensional images to determine whether the human body has abnormal behavior, thereby providing strong support for safety monitoring. In addition, the present invention uses a repeated behavior filtering module to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter repeated behaviors, effectively avoid data redundancy, maintain the latest data, and ensure that the system can reflect the latest behavior status in real time.

[0053] (6) The present invention conducts situation prediction and crowd concentration analysis on the degree and distribution of crowds, thereby avoiding the risk of accidents caused by excessive crowds and rationally allocating resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of an intelligent behavior analysis method based on a multi-visual recognition algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Example 1

[0057] Combine Figure 1 As shown, an embodiment of the present invention provides an intelligent behavior analysis method based on a multi-visual recognition algorithm, comprising the following steps:

[0058] S1. Using an image acquisition device to capture images from different perspectives, extracting image feature points from different perspectives, matching the image feature points from different perspectives, and calculating disparity and depth based on the positional differences of the successfully matched feature points in different images to obtain a three-dimensional image.

[0059] S2. Normalize the three-dimensional image, extract features of the three-dimensional image, obtain a feature map of the three-dimensional image, and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0060] S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image;

[0061] S4, performing human posture judgment, gait analysis, and face recognition on the three-dimensional human body image obtained in step S3, and determining whether the human body has abnormal behavior;

[0062] S5. Detect and record abnormal behaviors of the human body in the three-dimensional image, identify repeated abnormal behaviors that occur multiple times within a preset time period, filter out repeated behaviors, and determine abnormal behaviors as abnormal;

[0063] S6. For the abnormal behavior determined to be abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded.

[0064] Example 2

[0065] Different from Example 1, in this embodiment, in step S1, before collecting images from different perspectives, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and label the multi-visual data and record location information, behavior labels are set for the multi-visual data, and multi-visual data with behavior labels of abnormal human behavior are screened to construct a human abnormal behavior database;

[0066] In the process of building a human abnormal behavior database, it is necessary to scientifically and rationally divide the multi-visual data into training sets, validation sets, and test sets. The training set is mainly used for the initial training of the human abnormal behavior recognition model, the validation set is used for model tuning and parameter adjustment, and the test set is used to objectively evaluate the performance and effectiveness of the model.

[0067] Use advanced distributed training frameworks (such as TensorFlow, PyTorch, etc.) to accelerate the model training process. During the training process, apply data augmentation techniques to flip, rotate, and scale the data, thereby significantly increasing the diversity and richness of the data and effectively preventing overfitting problems during model training.

[0068] Perform hyperparameter tuning: Use effective methods such as grid search and random search to comprehensively and deeply optimize the model's many hyperparameters, including but not limited to key parameters such as learning rate, convolution kernel size, and number of network layers. Make full use of the validation set to accurately evaluate the model's performance under different hyperparameter combinations, and then carefully select the most ideal hyperparameter combination to maximize model performance.

[0069] Model compression and quantization can also be performed: cutting-edge technologies such as model pruning and quantization can be used to reduce the number of model parameters and computational complexity, thereby improving the model's inference speed and deployment efficiency in practical applications. Throughout the entire process, strict control is maintained to ensure that the model is lightweight within an acceptable accuracy loss range, making it more efficient and convenient for application in various practical scenarios and hardware devices.

[0070] In step S1, the image acquisition device may be a camera or sensor installed at different locations within the monitoring area to ensure coverage of the entire area;

[0071] The SIFT algorithm (Scale-Invariant Feature Transform) is used to extract image feature points from different perspectives. Other algorithms that can be used to extract image feature points from different perspectives, such as the SURF algorithm and the ORB algorithm, can also be used. The scale space is constructed by the SIFT algorithm. Let the image be $I(x,y)$, and the constructed scale space is expressed as $L(x,y,\sigma)=G(x,y,\sigma)\ast I(x,y)$, where $I(x,y)$ represents the input image, $x$ and $y$ are the horizontal and vertical coordinates on the image plane, representing the position of each pixel in the image; $L(x,y,\sigma)$ represents the constructed scale space, and similarly $x$ and $y$ are the horizontal and vertical coordinates on the image plane; $\sigma$ is the scale parameter used to control the smoothness of the Gaussian function, and different $\sigma$ values correspond to different scale spaces; $G(x,y,\sigma)$ represents the scale-variable Gaussian function, which is a two-dimensional Gaussian function. Its parameter $\sigma$ determines the shape and width of the function, which is used to smooth the image to different degrees; $\ast$ represents the convolution operation, which is to convolve the scale-variable Gaussian function; $G(x,y,\sigma)$ with the image $I(x,y)$ to obtain the scale space $L(x,y,\sigma)$;

[0072] The Brute-Force Matching algorithm is used to match the feature points of images under different viewpoints. The corresponding feature points are found between images under different viewpoints. Assume that the two feature point sets are $A = {a_1, a_2, …, a_n}$ and $B = {b_1, b_2, …, b_m}$. The Euclidean distance between the two feature points is calculated as follows:

[0073] $d(a_i,b_j)=\sqrt{(a_{i,x}-b_{j,x})^2+(a_{i,y}-b_{j,y})^2}$,

[0074] Among them, $A={a_1,a_2,\ldots,a_n}$ and $B={b_1,b_2,\ldots,b_m}$ represent two different feature point sets, $a_i$ represents the $i$th feature point in set $A$, where $i=1,2,\ldots,n$, $b_j$ represents the $j$th feature point in set $B$, where $j=1,2,\ldots,m$, $d(a_i,b_j)$ represents the Euclidean distance between feature point $a_i$ and feature point $b_j$, $a_{i,x}$ represents the feature point The horizontal coordinate value of point $a_i$, $a_{i,y}$ represents the vertical coordinate value of feature point $a_i$, $b_{j,x}$ represents the horizontal coordinate value of feature point $b_j$, $b_{j,y}$: represents the vertical coordinate value of feature point $b_j$, i∈{1,...n}, j∈{1,...,m}, a_{i,x}-b_{j,x} represents the distance between the i-th feature point x in set A and the j-th feature point x in set B, a_{i,y}-b_{j,y} represents the distance between the i-th feature point y in set A and the j-th feature point y in set B;

[0075] The three-dimensional image of the image feature points at different viewing angles is further determined using the triangulation formula, including:

[0076] First, perform depth calculation. Assume that the baseline distance between the optical centers of the two image acquisition devices is $b$, the focal length of the image acquisition device is $f$, and the parallax of the corresponding feature points in the two images is $d$. Then the depth of the feature point is $Z=\frac{fb}{d}$;

[0077] The triangulation formula is used to further determine the three-dimensional position of the target object corresponding to the image feature point, and the three-dimensional image of the image feature point under different viewing angles is determined. Assuming that the coordinates of the two image acquisition devices are $(x_1, y_1, z_1)$ and $(x_2, y_2, z_2)$, the coordinates of the feature points on the imaging planes of the two image acquisition devices are expressed as $(u_1, v_1)$ and $(u_2, v_2)$, and the three-dimensional coordinates of the feature points are expressed as $(X, Y, Z)$. $(X, Y, Z)$ is solved by the following formula:

[0078] $\begin{cases}\frac{u_1-x_1}{f}=\frac{X-x_1}{Z}\\frac{v_1-y_1}{f}=\frac{Y-y_1}{Z} \\frac{u_2-x_2}{f}=\frac{X-x_2}{Z}\\frac{v_2-y_2}{f}=\frac{Y-y_2}{Z}\end{cases}$.

[0079] Example 3

[0080] Unlike Examples 1 and 2, in this embodiment, in step S2, the three-dimensional image obtained in step S1 is first resized, and a possible scaling operation is performed to adjust the image to an appropriate size. Then, normalization processing is performed so that the pixel values of the three-dimensional image are within a preset range. Assuming that the pixel values of the original three-dimensional image are $I(x,y)$, the normalized pixel values are $I_{norm}(x,y)=\frac{I(x,y)}{255}$; a deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$.

[0081] The YOLO algorithm is used to transform the target detection problem into a regression problem. The feature map is divided into grids, and each grid predicts multiple bounding boxes and confidence levels. Assume that the predicted result of a bounding box is $(bx,by,bw,bh)$, and its true bounding box is $(tx,ty,tw,th)$. The relationship between it and the true bounding box $(tx,ty,tw,th)$ is expressed as confidence level, which can be converted by the following formula:

[0082] $b_x=\sigma(t_x)+c_x$

[0083] $b_y=\sigma(t_y)+c_y$

[0084] $b_w=p_we^{t_w}$

[0085] $b_h=p_he^{t_h}$

[0086] Among them, for the formulas $b_x=\sigma(t_x)+c_x$ and $b_y=\sigma(t_y)+c_y$: $b_x$ represents the horizontal coordinate position of the center point of the predicted bounding box, $b_y$ represents the vertical coordinate position of the center point of the predicted bounding box, $t_x$ represents the parameter predicted by the model, representing the horizontal offset of the center point of the predicted bounding box relative to the upper left corner of the grid, $t_y$ represents the parameter predicted by the model, representing the vertical offset of the center point of the predicted bounding box relative to the upper left corner of the grid, $\sigma$ represents the Sigmoid function, which is used to compress the value range of $t_x$ and $t_y$ to $(0,1)$, $c_x$ represents the horizontal coordinate position of the upper left corner of the current grid, and $c_y$ represents the vertical coordinate position of the upper left corner of the current grid; for the formula

[0087] $b_w=p_we^{t_w}$and $b_h=p_he^{t_h}$: $b_w$represents the width of the predicted bounding box, $b_h$represents the height of the predicted bounding box, $t_w$represents the parameter predicted by the model, which is used to adjust the width of the predicted bounding box, $t_h$represents the parameter predicted by the model, which is used to adjust the height of the predicted bounding box, $p_w$represents the width of the prior box, $p_h$represents the height of the prior box, and $e$represents the base of the natural logarithm.

[0088] Use the SSD algorithm to detect targets on multiple feature maps. Each feature map corresponds to a different scale. Combine feature maps of different scales to set the ground truth of bounding boxes of different sizes for target detection.

[0089] Sort the bounding boxes according to their confidence, select the bounding box with the highest confidence, calculate the overlap between the bounding box with the highest confidence and other bounding boxes, and remove them if they exceed the threshold. The overlap is $IoU$, which is calculated as:

[0090] $IoU=\frac{A\cap B}{A\cup B}$

[0091] Among them, $A$ and $B$ are the areas covered by the two bounding boxes respectively.

[0092] Example 4

[0093] Different from embodiments 1, 2 and 3, in this embodiment, in step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image obtained in step S2;

[0094] In this embodiment, the median filtering algorithm is used to denoise the three-dimensional image obtained in step S2. Assume that the image is $I(x,y)$, the filter window size is $n\times n$, and the pixel value of the filtered three-dimensional image is

[0095] $I_{filtered}(x,y)=median{I(x+i,y+j)\mid-k\leq i,j\leq k}$, where $k=\frac{n-1}{2}$;

[0096] Then, the mean filter algorithm is used to filter and smooth the image to highlight important features. The pixel value of the 3D image after the mean filter algorithm is $I_{filtered}(x,y)=\frac{1}{n^2}\sum_{i=-k}^{k}\sum_{j=-k}^{k}I(x+i,y+j)$, where $k=\frac{n-1}{2}$ and $n\times n$ is the filter window size.

[0097] The data value range of the denoised and filtered three-dimensional image is normalized to the interval [0,1]. The normalization formula is as follows:

[0098] $Normalized\Pixel\Value=\frac{Original\Pixel\Value}{Maximum\Pixel\Value}$;

[0099] The contrast of the three-dimensional image is adjusted using the histogram equalization algorithm. For the brightness adjustment of the three-dimensional image, assuming that the original brightness is $L$ and the adjustment value is $\Delta L$, the new brightness is $New\Brightness=(L+\Delta L)\mod\256$. The color saturation of the three-dimensional image is adjusted by changing the saturation component in the color space for image enhancement and correction.

[0100] Image correction eliminates or reduces distortion caused by factors such as lighting and shadows, ensuring that the objects and scenes in the image more realistically and accurately reflect the actual environment.

[0101] Example 5

[0102] Different from Embodiments 1, 2, 3 and 4, in this embodiment, step S4 specifically includes:

[0103] Normalize the three-dimensional image obtained in step S3. Assume that the pixel value of the three-dimensional image obtained in step S3 is $I(x,y)$, and after normalization, it is $I_{norm}(x,y)=\frac{I(x,y)}{255}$;

[0104] Convolutional neural networks are used for feature extraction. Specifically, other deep learning algorithms, such as the OpenPose algorithm, can also be used for feature extraction. Coordinates are predicted through the fully connected layer of the convolutional neural network to determine the positions of human joints, such as the head, shoulders, elbows, hands, hips, knees, and feet. Assuming that the input three-dimensional image is $I$, the joint point coordinates predicted by the fully connected layer of the convolutional neural network are $(x_1, y_1, x_2, y_2, …, x_n, y_n)$, where $(x_i, y_i)$ is the coordinate of the $i$th joint point.

[0105] Convolutional neural networks are used to extract the motion characteristics of the human body. Specifically, other algorithms under deep learning algorithms can also be used, such as step length, step frequency, gait period, etc. The optical flow algorithm is used to calculate the movement speed and direction of pixels in the three-dimensional image to obtain the movement trajectory of the human body. Assuming that the sequence of three-dimensional images is ${I_1,I_2,…,I_t}$, the optical flow calculation formula is:

[0106] $(\vec{v}(x,y))=(v_x,v_y)=(\frac{\partial I}{\partial t}) / (\frac{\partial I}{\partial x}+\frac{\partial I}{\partial y})$

[0107] Where $\vec{v}(x,y)$ is the optical flow vector at the pixel point $(x,y)$, $v_x$ and $v_y$ are the velocity components in the horizontal and vertical directions respectively;

[0108] A convolutional neural network is used to extract facial features. Specifically, a face recognition method based on eigenfaces or other algorithms under a deep learning algorithm can also be used. The features are then compared with known faces in a database to determine the identity of the person. Assume that the input face image is $I$, the feature vector extracted by the convolutional neural network is $\vec{f}$, and the feature vectors in the database are ${\vec{f}_1,\vec{f}_2,…,\vec{f}m}$;

[0109] Compare $\vec{f}$ with ${\vec{f}_1,\vec{f}_2,…,\vec{f}m}$ and calculate the similarity using the Euclidean distance method. The Euclidean distance calculation formula is as follows:

[0110] $d(\vec{f},\vec{f}i)=\sqrt{\sum{j=1}^{n}(f_j-f{i,j})^2}$, where $n$ is the dimension of the feature vector;

[0111] Determine human posture based on joint positions, perform gait analysis based on human motion trajectory, perform face recognition based on facial feature comparison, and determine whether a person is exhibiting abnormal behavior by combining human posture, gait analysis, and face recognition.

[0112] For example, human posture estimation can be used to determine the position and movement of a person's hands, and combined with the smoke characteristics in the image (which can be detected through color, shape, etc.) to determine whether the person is smoking; for the behavior of using a mobile phone to make a call, human posture estimation can be used to determine the position of the person's hands and ears, as well as the characteristics of the mobile phone (such as shape, color, etc.) to determine whether the person is making a call; for the behavior of not wearing a helmet in an environment where a helmet is required, human posture estimation can be used to determine the position of the person's head, and combined with the characteristics of the helmet (such as color, shape, etc.) to make a judgment.

[0113] Example 6

[0114] Unlike embodiments 1, 2, 3, 4, and 5, in this embodiment, in step S5, behavioral events of the human body are continuously monitored. Assuming that the set of behavioral events is $E={e_1, e_2, …, e_n}$, where $e_i$ represents a behavioral event, behavioral events that occur multiple times within a preset time are identified by comparing behavioral features. Assuming that behavioral event $e_i$ and behavioral event $e_j$ occur within the preset time, if the feature similarity between behavioral event $e_i$ and behavioral event $e_j$ exceeds a preset value, then it is determined to be a repeated behavior.

[0115] Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of behavior events, and obtain a new set of behavior events, $E'=ER$. The new set of behavior events is the abnormal behaviors determined to be abnormal.

[0116] When processing data, by identifying and filtering repeated behaviors and obtaining truly abnormal behaviors that are abnormal, we can ensure the latest and accuracy of data and improve data processing efficiency and quality;

[0117] In step S7, the situation prediction module performs headcount, flow analysis, and aggregation analysis on the human behaviors collected in step S1;

[0118] Use a human detection algorithm to count the number of people in the 3D image collected in step S1. Assuming that the video frame of the 3D image is $F$ and the human detection algorithm is $D$, the number of people $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at coordinate $(x,y)$;

[0119] Use the human body detection algorithm to analyze the gait obtained in step S3, perform flow analysis on the movement direction and speed, and derive the overall flow trend. Assuming that the position sequence of the human body is ${(x_1,y_1),(x_2,y_2),…,(x_t,y_t)}$, the movement direction is $\vec{d}=(x_t-x_1,y_t-y_1)$, and the movement speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$;

[0120] Use the human detection algorithm to count the number of people in the 3D image collected in step S1, and use the crowd density calculation method to calculate the crowd density, and obtain the crowd-dense area for analysis. Assume that the video area in the 3D image is divided into $m\timesn$ grids, and the number of people in each grid is $p_{i,j}$, then the crowd density is:

[0121] $Density=\frac{\sum_{i=1}^{m}\sum_{j=1}^{n}p_{i,j}}{m\times n}$,

[0122] Among them, $m$ represents the number of grids into which the video area of the three-dimensional image is divided horizontally, $n$ represents the number of grids into which the video area of the three-dimensional image is divided vertically, $i$ represents the row index in the horizontally divided grid, and the value range is from 1 to $m$, $j$ represents the column index in the vertically divided grid, and the value range is from 1 to $n$, $p_{i,j}$ represents the number of people in the specific grid of the $i$th row and the $j$th column in the three-dimensional image video area,

[0123] $Density$ represents the crowd density of the entire video area, $\sum_{i=1}^{m}\sum_{j=1}^{n}p_{i,j}$ represents the sum of the number of people in all $m\times n$ grids, that is, the number of people in all grids in each row is first summed up (inner $\sum_{j=1}^{n}p_{i,j}$), and then the sum of all rows is summed up (outer $\sum_{i=1}^{m}$), and $m\times n$ represents the total number of grids after the video area is divided.

[0124] The situation prediction module leverages big data and machine learning technologies to perform real-time tracking and statistical analysis of human behavior in 3D image data. By deeply mining 3D image data, it can perform headcounts—identifying and counting each person in the video. This aids safety management, such as ensuring personnel numbers are within safe limits, and reflects project progress to facilitate adjustments. Flow analysis tracks human movement, analyzing direction and speed to derive overall flow trends, facilitating the planning of safe passages and evacuation routes, and providing a basis for optimizing construction layouts. Aggregation analysis detects dense areas in the video, analyzing the degree of crowd concentration and distribution, to mitigate the risk of accidents caused by excessive crowds and to rationally allocate resources.

[0125] Example 7

[0126] An embodiment of the present invention further provides a system for performing intelligent behavior analysis using the intelligent behavior analysis method based on the multi-visual recognition algorithm as described above, comprising a multi-vision acquisition module, a multi-target detection module, a data processing module, a behavior analysis module, a repeated behavior filtering module, an alarm and feedback module, and a situation prediction module;

[0127] The multi-viewing acquisition module is used to collect images from different perspectives, extract feature points, match the feature points of images from different perspectives, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image;

[0128] The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image that meets the target;

[0129] The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image;

[0130] The behavior analysis module is used for human posture judgment, gait analysis and face recognition in 3D images, and combines the above indicators to determine whether the human body has abnormal behavior;

[0131] The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal;

[0132] The alarm and feedback module is used to issue alarm information for abnormal behavior;

[0133] The situation prediction module is used to perform population counting, flow analysis and aggregation analysis on the collected three-dimensional images.

[0134] The application scope of the intelligent behavior analysis system based on the multi-visual recognition algorithm of the present invention is not limited to the management of construction sites such as water conservancy and hydropower, wind power, and solar power generation, but can also be widely used in communities, streets, and other application scenarios; whether in public areas of the community or on busy streets, it can play its powerful monitoring and early warning functions to protect the safety of residents and pedestrians; in the field of traffic management, the intelligent behavior analysis system based on the multi-visual recognition algorithm of the present invention can track the movement trajectories of vehicles and pedestrians in real time, timely discover problems such as traffic congestion and violations, provide solutions for traffic management departments, effectively alleviate traffic pressure, and ensure smooth roads; in the field of education, it can be used to monitor students' classroom performance and provide teachers with personalized teaching suggestions; in the medical field, it can assist doctors in remote diagnosis and treatment, and improve the quality and efficiency of medical services. Through its wide application in different scenarios, the intelligent behavior analysis system based on the multi-visual recognition algorithm of the present invention can bring a safer and more orderly environment to society; of course, further, it can provide deeper insights and decision support through its refined analysis capabilities. In short, this system can not only play an important role in traditional security management fields, but also has unique value and potential in retail, transportation, education, medical care and other fields.

[0135] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the scope of the present invention shall be included in the scope of protection of the present invention.

Claims

1. An intelligent behavior analysis method based on multi-visual recognition algorithm, characterized in that: The following steps are involved: S1. Using an image acquisition device to capture images from different perspectives, extracting image feature points from different perspectives, matching the image feature points from different perspectives, and calculating disparity and depth based on the positional differences of the successfully matched feature points in different images to obtain a three-dimensional image. S2. Normalize the three-dimensional image, extract features of the three-dimensional image, obtain a feature map of the three-dimensional image, and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body; S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image; S4, performing human posture judgment, gait analysis, and face recognition on the three-dimensional human body image obtained in step S3, and determining whether the human body has abnormal behavior; S5. Detect and record abnormal behaviors of the human body in the three-dimensional image, identify repeated abnormal behaviors that occur multiple times within a preset time period, filter out repeated behaviors, and determine abnormal behaviors as abnormal; S6. For the abnormal behavior determined to be abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded; In step S1, before collecting images from different perspectives, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and label the multi-visual data and record location information, behavior labels are set for the multi-visual data, and multi-visual data with behavior labels of abnormal human behavior are screened to build a human abnormal behavior database; In step S1, a scale space is constructed by a scale space feature extraction algorithm, and the scale space feature extraction algorithm is used to extract image feature points at different viewing angles; In step S1, the image feature points under different perspectives are matched by a brute force matching algorithm, and the Euclidean distance of the image feature points under different perspectives is calculated by the brute force matching algorithm to find the matching feature points, and the successfully matched feature points are obtained. The disparity and depth are calculated according to the position difference of the successfully matched feature points in different images, and the three-dimensional position of the target object corresponding to the image feature points is further determined by the triangulation formula, and the three-dimensional image of the image feature points under different perspectives is determined.

2. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: The scale space feature extraction algorithm is one of the SIFT algorithm, SURF algorithm, and ORB algorithm.

3. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S2, the 3D image obtained in step S1 is first resized and then normalized so that the pixel values of the 3D image are within a preset range. Assuming that the pixel values of the original 3D image are $I(x,y)$, the normalized pixel values are $I_{norm}(x,y)=\frac{I(x,y)}{255}$; A deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$.

4. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S2, the YOLO algorithm is used to transform the target detection problem into a regression problem. The feature map is divided into grids, and each grid predicts the confidence of multiple bounding boxes. The SSD algorithm is used to perform target detection on multiple feature maps. Each feature map corresponds to a different scale. The feature maps of different scales are combined to set the ground truth values of bounding boxes of different sizes for target detection to obtain a three-dimensional image of the human body.

5. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 4 is characterized in that: In step S2, the bounding boxes are sorted according to the confidence level, the bounding box with the highest confidence level is selected, and the overlap between the bounding box with the highest confidence level and other bounding boxes is calculated. If the overlap exceeds a threshold, the bounding box is removed.

6. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image of the human body obtained in step S2.

7. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 6 is characterized in that: In step S3, the three-dimensional human body image obtained in step S2 is denoised using a median filtering algorithm, and filtered using a mean filtering algorithm to smooth the image and highlight important features. The data value range of the denoised and filtered three-dimensional human body image is normalized.

8. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 7 is characterized in that: In step S3, the contrast of the 3D image is adjusted using a histogram equalization algorithm. For the adjustment of the brightness of the 3D image, assuming that the original brightness is $L$, the adjustment value is $\Delta L$, and the new brightness is $New\ Brightness=(L+\Delta L)\mod\ 256$. The color saturation of the 3D image is adjusted by changing the saturation component in the color space to perform image enhancement and correction.

9. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: Step S4 specifically includes: Convolutional neural network is used for feature extraction, and coordinate prediction is performed through the fully connected layer of the convolutional neural network to determine the position of human joints; A convolutional neural network is used to extract the motion characteristics of the human body, and the optical flow algorithm is used to calculate the motion speed and direction of pixels in the three-dimensional image to obtain the motion trajectory of the human body. A convolutional neural network is used to extract facial features, which are then compared with known faces in the database to determine the person's identity. The human body posture is judged based on the position of the human joints, gait analysis is performed based on the human body's movement trajectory, and face recognition is performed based on facial feature comparison. The human body posture, gait analysis and face recognition are combined to determine whether the human body has abnormal behavior.

10. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S5, abnormal human behavior is continuously monitored. Assuming that the set of abnormal behavior events is $E={e_1,e_2,…,e_n}$, abnormal behavior events that occur multiple times within a preset time are identified by comparing behavioral features. Assuming that an abnormal behavior event that occurs within the preset time is $e_i$ and another abnormal behavior event is $e_j$, if the feature similarity between the abnormal behavior event $e_i$ and the other abnormal behavior event $e_j$ exceeds a preset value, it is considered a repeated behavior. Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of abnormal behavior events, and obtain the set of behavior events that are actually judged to be abnormal. The set of behavior events that are actually judged to be abnormal is $E'=ER$.

11. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: Perform population counting, flow direction analysis, and aggregation analysis on the three-dimensional image collected in step S1.

12. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 11 is characterized in that , Use the human detection algorithm to count the number of people in the 3D image collected in step S1. Assume that the video frame of the 3D image is $F$, the human detection algorithm is $D$, and the number of people is $N$, then the number of people $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at the coordinate $(x,y)$. Use the human body detection algorithm to analyze the gait obtained in step S4, perform flow analysis on the movement direction and speed, and derive the overall flow trend. Assume that the position sequence of the human body is ${(x_1,y_1),(x_2,y_2),…,(x_t,y_t)}$, the movement direction is $\vec{d}=(x_t-x_1,y_t-y_1)$, and the movement speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$; The human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1, and the crowd density is calculated using the crowd density calculation method to obtain the crowd-dense area for aggregation analysis.

13. A system for performing intelligent behavior analysis using the intelligent behavior analysis method based on a multi-visual recognition algorithm as described in any one of claims 1 to 12, characterized in that: It includes multi-viewing acquisition module, multi-target detection module, data processing module, behavior analysis module, repeated behavior filtering module and alarm and feedback module; The multi-viewing acquisition module is used to collect images from different perspectives, extract feature points, match the feature points of images from different perspectives, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image; The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body; The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image of the human body; The behavior analysis module is used for human posture judgment, gait analysis and face recognition in three-dimensional human images, and to determine whether the human body has abnormal behavior; The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal; The alarm and feedback module is used to issue alarm information for abnormal behavior.

14. The system for performing intelligent behavior analysis based on the intelligent behavior analysis method of multiple visual recognition algorithms according to claim 13, characterized in that: The multi-viewing acquisition module includes a situation prediction module, which is used to perform headcount, flow analysis and aggregation analysis on the acquired three-dimensional images.

Images