Intelligent behavior analysis method and system based on multi-vision identification algorithm

Abstract

The invention belongs to the technical field of image information processing, and discloses an intelligent behavior analysis method and system based on a multi-vision recognition algorithm. The multi-view vision acquisition module is arranged to acquire the three-dimensional image, the multi-target detection module extracts the feature map of the three-dimensional image to obtain the human body three-dimensional image, and the data processing module carries out denoising, filtering and normalization processing on the human body three-dimensional image and carries out image enhancement and correction on the human body three-dimensional image. The behavior analysis module is used for judging whether abnormal behaviors occur in a human body or not, the repeated behavior filtering module is used for filtering repeated abnormal behaviors, the alarm and feedback module is used for sending alarm information for the abnormal behaviors, and people counting, flow direction analysis and aggregation degree analysis are carried out on collected three-dimensional images. The human body abnormal behavior can be quickly recognized, the early warning signal is sent out in time, and comprehensive support is provided for safety monitoring, behavior analysis and situation prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and in particular relates to an intelligent behavior analysis method and system based on a multi-vision recognition algorithm. Background Art

[0002] In modern society, the rapid advancement of science and technology and the continuous growth of intelligent demand have made the accurate analysis and understanding of human behavior increasingly critical in many fields. Traditional behavior analysis techniques often rely on a single visual data source or limited feature extraction methods, which limits the accuracy and comprehensiveness of the analysis results.

[0003] In the field of construction site monitoring such as water conservancy and hydropower, wind power, and solar power generation, previous monitoring systems can usually only rely on fixed-angle cameras to obtain limited scene information, and it is difficult to fully capture the behavior of people in complex environments. Monitoring of workers mostly relies on manual observation or basic image recognition technology, which makes it difficult to identify potential irregular behaviors and safety hazards in a timely manner. At the same time, when processing large amounts of visual data, existing technologies have low recognition efficiency, high misjudgment rate, and insufficient real-time performance, and cannot meet the needs of fast and accurate behavior analysis. Summary of the invention

[0004] In order to overcome the limitations of existing technologies and improve the accuracy, comprehensiveness and real-time performance of behavior analysis, it is particularly important to develop an intelligent behavior analysis method and system based on multi-visual recognition algorithm, which has significant practical significance and application value.

[0005] With the continuous advancement of computer vision technology and deep learning algorithms, multi-target detection algorithms and multi-viewing technologies have made significant progress. Multi-target detection algorithms such as YOLO and SSD directly predict the bounding box and category of the target through convolutional neural networks, which has the advantages of low computational complexity and good real-time performance. Multi-viewing technology uses multiple cameras or sensors to obtain multiple perspectives of the scene, and through image fusion and three-dimensional reconstruction, it can achieve all-round perception and understanding of the scene. Combining these two technologies can significantly improve the accuracy and reliability of behavior analysis.

[0006] The present invention aims to solve the technical problems existing in the prior art and provide an intelligent behavior analysis method and system based on multi-visual recognition algorithms that can quickly identify abnormal human behavior and issue early warning signals in a timely manner, providing comprehensive support for safety monitoring, behavior analysis and situation prediction.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] The present invention provides an intelligent behavior analysis method based on a multi-visual recognition algorithm, comprising the following steps:

[0009] S1. Use an image acquisition device to acquire images at different viewing angles, extract image feature points at different viewing angles, match the image feature points at different viewing angles, calculate the disparity and depth according to the position difference of the successfully matched feature points in different images, and acquire a three-dimensional image;

[0010] S2, normalizing the three-dimensional image, extracting features of the three-dimensional image, obtaining a feature map of the three-dimensional image, and performing target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0011] S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image;

[0012] S4, performing human posture judgment, gait analysis and face recognition on the three-dimensional human body image obtained in step S3, and judging whether the human body has abnormal behavior;

[0013] S5, detecting and recording abnormal behaviors of the human body in the three-dimensional image, identifying repeated abnormal behaviors that occur multiple times within a preset time period, filtering repeated behaviors, and obtaining abnormal behaviors that are determined to be abnormal;

[0014] S6. For the abnormal behavior determined as abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded.

[0015] Preferably, in step S1, before acquiring images from different perspectives, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and label the multi-visual data and record the location information, behavior labels are set for the multi-visual data, and the multi-visual data with behavior labels of abnormal human behavior are screened to construct an abnormal human behavior database.

[0016] Preferably, in step S1, a scale space is constructed by a scale space feature extraction algorithm, and the scale space feature extraction algorithm is used to extract image feature points at different viewing angles.

[0017] Preferably, the scale space feature extraction algorithm is one of the SIFT algorithm, the SURF algorithm and the ORB algorithm.

[0018] Preferably, in step S1, the image feature points at different viewing angles are matched by a brute force matching algorithm, the Euclidean distance of the image feature points at different viewing angles is calculated by the brute force matching algorithm to find the matching feature points, and the successfully matched feature points are obtained. The disparity and depth are calculated according to the position difference of the successfully matched feature points in different images, and the three-dimensional position of the target object corresponding to the image feature points is further determined by the triangulation formula, so as to determine the three-dimensional image of the image feature points at different viewing angles.

[0019] Preferably, in step S2, the three-dimensional image obtained in step S1 is first resized and then normalized so that the pixel values ​​of the three-dimensional image are within a preset range. Assuming that the pixel values ​​of the original three-dimensional image are $I(x,y)$, the normalized pixel values ​​are $I_{norm}(x,y)=\frac{I(x,y)}{255}$;

[0020] A deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$.

[0021] Preferably, in step S2, the target detection problem is converted into a regression problem using the YOLO algorithm, and a grid is divided on the feature map, and each grid predicts the confidence of multiple bounding boxes;

[0022] The SSD algorithm is used to perform target detection on multiple feature maps. Each feature map corresponds to a different scale. The feature maps of different scales are combined to set the true values ​​of bounding boxes of different sizes for target detection to obtain a three-dimensional image of the human body.

[0023] Preferably, in step S2, the bounding boxes are sorted according to the confidence, the bounding box with the highest confidence is selected, and the overlap between the bounding box with the highest confidence and other bounding boxes is calculated. If the overlap exceeds a threshold, the bounding box is removed.

[0024] Preferably, in step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image of the human body obtained in step S2.

[0025] Preferably, in step S3, the three-dimensional human body image obtained in step S2 is denoised using a median filtering algorithm, and filtered using a mean filtering algorithm to smooth the image and highlight important features, and the data value range of the denoised and filtered three-dimensional human body image is normalized.

[0026] Preferably, in step S3, the contrast of the three-dimensional image is adjusted using a histogram equalization algorithm; for the adjustment of the brightness of the three-dimensional image, assuming that the original brightness is $L$, the adjustment value is $\Delta L$, and the new brightness is $New\Brightness=(L+\Delta L)\mod\256$; the color saturation of the three-dimensional image is adjusted by changing the saturation component in the color space to perform image enhancement and correction.

[0027] Preferably, step S4 specifically includes:

[0028] Convolutional neural network is used for feature extraction, and coordinate prediction is performed through the fully connected layer of the convolutional neural network to determine the position of human joints;

[0029] The convolutional neural network is used to extract the motion characteristics of the human body, and the optical flow algorithm is used to calculate the motion speed and direction of the pixels in the three-dimensional image to obtain the motion trajectory of the human body;

[0030] Convolutional neural networks are used to extract facial features, which are then compared with known faces in the database to determine the identity of the person;

[0031] The human body posture is judged according to the position of the human joints, the gait analysis is performed according to the movement trajectory of the human body, and the face recognition is performed according to the comparison of facial features. The human body posture, gait analysis and face recognition are combined to determine whether the human body has abnormal behavior.

[0032] Preferably, in step S5, the abnormal behavior of the human body is continuously monitored, assuming that the set of abnormal behavior events is

[0033] $E={e_1,e_2,…,e_n}$, identify abnormal behavior events that occur multiple times within a preset time by comparing behavior features. Assume that the abnormal behavior event that occurs within the preset time is $e_i$, and another abnormal behavior event is $e_j$. If the feature similarity between the abnormal behavior event $e_i$ and the other abnormal behavior event $e_j$ exceeds a preset value, it is considered a repeated behavior.

[0034] Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of abnormal behavior events, and obtain the set of behavior events actually judged to be abnormal. The set of behavior events actually judged to be abnormal is $E'=ER$.

[0035] Preferably, the three-dimensional image collected in step S1 is subjected to population counting, flow direction analysis and aggregation analysis.

[0036] Preferably, a human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1. Assume that the video frame of the three-dimensional image is $F$, the human body detection algorithm is $D$, the number of people is $N$, $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at the coordinate $(x,y)$.

[0037] Use the human body detection algorithm to analyze the gait obtained in step S4, perform flow analysis of the moving direction and speed, and obtain the overall flow trend. Assume that the position sequence of the human body is ${(x_1, y_1), (x_2, y_2), …, (x_t, y_t)}$, the moving direction is $\vec{d}=(x_t-x_1, y_t-y_1)$, and the moving speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$;

[0038] The human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1, and the crowd density is calculated using the crowd density calculation method to obtain the crowd-dense area for aggregation analysis.

[0039] The present invention also provides a system for performing intelligent behavior analysis using the intelligent behavior analysis method based on the multi-vision recognition algorithm as described above, comprising a multi-vision acquisition module, a multi-target detection module, a data processing module, a behavior analysis module, a repeated behavior filtering module and an alarm and feedback module;

[0040] The multi-viewing acquisition module is used to acquire images from different viewing angles, extract feature points, match the feature points of images from different viewing angles, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image.

[0041] The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0042] The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image of the human body;

[0043] The behavior analysis module is used for human posture judgment, gait analysis and face recognition in human 3D images, and determines whether the human body has abnormal behavior;

[0044] The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal;

[0045] The alarm and feedback module is used to issue alarm information for abnormal behavior.

[0046] Preferably, the multi-eye vision acquisition module includes a situation prediction module, and the situation prediction module is used to perform headcount counting, flow analysis and aggregation analysis on the acquired three-dimensional images.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] (1) The present invention can quickly identify abnormal human behavior, avoid potential risk factors, issue early warning signals in the first place, and record on-site conditions, thereby achieving comprehensive monitoring and analysis of human behavior in images, thereby improving the efficiency and accuracy of safety monitoring, facilitating subsequent analysis and processing, significantly improving the efficiency of safety management, and significantly saving labor costs;

[0049] (2) The present invention realizes real-time monitoring and risk warning of construction sites such as water conservancy and hydropower, wind power, and solar power generation through an intelligent behavior analysis method based on a multi-vision recognition algorithm, thereby ensuring the safety and order of the construction site. In addition, the intelligent behavior analysis system based on a multi-vision recognition algorithm is not limited to construction site management, but can also be widely used in communities, streets, schools, hospitals, and many other application scenarios.

[0050] (3) The present invention integrates multi-eye vision technology and multi-target detection algorithms to achieve automatic recognition, tracking and analysis of human behavior in surveillance videos, thereby providing comprehensive support for safety monitoring, behavior analysis and situation prediction. It can automatically identify targets in three-dimensional images and predict their positions and bounding boxes to achieve accurate tracking of targets. In addition, pedestrians can be classified according to the attribute information of their helmets, for example, they can be classified into construction personnel, supervisors, owner personnel, etc., thereby achieving more detailed and accurate personnel classification management.

[0051] (4) The present invention removes noise and interference in the image by performing denoising, filtering and normalization on the image in the data preprocessing stage, thereby improving the clarity and accuracy of the data; the image enhancement and correction technology further improves the visual effect and accuracy, ensuring that the system can more accurately identify and analyze the targets and behaviors in the image, thereby improving the data quality and availability;

[0052] (5) The present invention uses a behavior analysis module to perform human posture judgment, gait analysis, and face recognition on three-dimensional images to determine whether the human body has abnormal behavior, thereby providing strong support for safety monitoring. In addition, the present invention uses a repeated behavior filtering module to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter repeated behaviors, effectively avoid data redundancy, maintain the latest data, and ensure that the system can reflect the latest behavior status in real time.

[0053] (6) The present invention performs situation prediction and crowd concentration analysis on the degree and distribution of crowds, thereby avoiding the risk of accidents caused by excessive crowds and rationally allocating resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of an intelligent behavior analysis method based on a multi-visual recognition algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example 1

[0057] Combination Figure 1 As shown, an embodiment of the present invention provides an intelligent behavior analysis method based on a multi-visual recognition algorithm, comprising the following steps:

[0058] S1. Use an image acquisition device to acquire images at different viewing angles, extract image feature points at different viewing angles, match the image feature points at different viewing angles, calculate the disparity and depth according to the position difference of the successfully matched feature points in different images, and acquire a three-dimensional image;

[0059] S2, normalizing the three-dimensional image, extracting features of the three-dimensional image, obtaining a feature map of the three-dimensional image, and performing target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body;

[0060] S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image;

[0061] S4, performing human posture judgment, gait analysis and face recognition on the three-dimensional human body image obtained in step S3, and judging whether the human body has abnormal behavior;

[0062] S5, detecting and recording abnormal behaviors of the human body in the three-dimensional image, identifying repeated abnormal behaviors that occur multiple times within a preset time period, filtering repeated behaviors, and obtaining abnormal behaviors that are determined to be abnormal;

[0063] S6. For the abnormal behavior determined as abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded.

[0064] Example 2

[0065] Different from Example 1, in this embodiment, in step S1, before collecting images from different viewing angles, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and annotate the multi-visual data and record the location information, a behavior label is set for the multi-visual data, and the multi-visual data with a behavior label of abnormal human behavior is screened to build a human abnormal behavior database;

[0066] In the process of building a human abnormal behavior database, it is necessary to scientifically and rationally divide the multi-visual data into training set, validation set and test set. Among them, the training set is mainly used for the initial training of the human abnormal behavior recognition model, the validation set is used to tune the model and adjust the parameters, and the test set is used to objectively evaluate the performance and effect of the model;

[0067] Use advanced distributed training frameworks (such as TensorFlow, PyTorch, etc.) to accelerate the model training process. During the training process, apply data enhancement technology to flip, rotate, and scale the data, thereby greatly increasing the diversity and richness of the data and effectively preventing the model from overfitting during training;

[0068] Perform hyperparameter adjustment: Use effective methods such as grid search and random search to comprehensively and deeply optimize many hyperparameters of the model, including but not limited to key parameters such as learning rate, convolution kernel size, and number of network layers; make full use of the validation set to accurately evaluate the performance of the model under different hyperparameter combinations, and then carefully select the most ideal hyperparameter combination to maximize the performance of the model;

[0069] Model compression and quantization can also be performed: cutting-edge technologies such as model pruning and quantization are used to reduce the number of model parameters and the amount of calculation; improve the model's reasoning speed and deployment efficiency in practical applications. Throughout the process, strict control is carried out to ensure that the model is lightweight within an acceptable range of accuracy loss, so that the model can be more efficiently and conveniently applied to various practical scenarios and hardware devices.

[0070] In step S1, the image acquisition device may be a camera or sensor installed at different locations within the monitoring area to ensure coverage of the entire area;

[0071] The SIFT algorithm (Scale-Invariant Feature Transform) is used to extract image feature points from different perspectives. Other algorithms that can be used to extract image feature points from different perspectives can also be used, such as the SURF algorithm and the ORB algorithm. The scale space is constructed by the SIFT algorithm. Let the image be $I(x,y)$, and the constructed scale space is expressed as $L(x,y,\sigma)=G(x,y,\sigma)\ast I(x,y)$, where $I(x,y)$ represents the input image, $x$ and $y$ are the horizontal and vertical coordinates on the image plane, respectively, representing the position of each pixel in the image; $L(x,y,\sigma)$ represents the constructed scale space, and similarly $x$ and $y$ are the horizontal and vertical coordinates on the image plane; $\sigma$ is a scale parameter used to control the smoothness of the Gaussian function, and different $\sigma$ values ​​correspond to different scale spaces; $G(x,y,\sigma)$ represents a scale-variable Gaussian function, which is a two-dimensional Gaussian function, and its parameter $\sigma$ determines the shape and width of the function, which is used to smooth the image to different degrees; $\ast$ represents a convolution operation, that is, the scale-variable Gaussian function; $G(x,y,\sigma)$ is convolved with the image $I(x,y)$ to obtain the scale space $L(x,y,\sigma)$;

[0072] The Brute-Force Matching algorithm is used to match the image feature points at different perspectives, and the corresponding feature points are found between images at different perspectives. Suppose the two feature point sets are $A={a_1,a_2,…,a_n}$ and $B={b_1,b_2,…,b_m}$, and the Euclidean distance between the two feature points is calculated. The calculation formula of the Euclidean distance is as follows:

[0073] $d(a_i,b_j)=\sqrt{(a_{i,x}-b_{j,x})^2+(a_{i,y}-b_{j,y})^2}$,

[0074] Among them, $A={a_1,a_2,\ldots,a_n}$ and $B={b_1,b_2,\ldots,b_m}$ represent two different feature point sets, $a_i$ represents the $i$th feature point in set $A$, where $i=1,2,\ldots,n$, $b_j$ represents the $j$th feature point in set $B$, where $j=1,2,\ldots,m$, $d(a_i,b_j)$ represents the Euclidean distance between feature point $a_i$ and feature point $b_j$, $a_{i,x}$ represents the feature The horizontal coordinate value of point $a_i$, $a_{i,y}$ represents the vertical coordinate value of feature point $a_i$, $b_{j,x}$ represents the horizontal coordinate value of feature point $b_j$, $b_{j,y}$: represents the vertical coordinate value of feature point $b_j$, i∈{1,...n}, j∈{1,...,m}, a_{i,x}-b_{j,x} represents the distance between the i-th feature point x in set A and the j-th feature point x in set B, a_{i,y}-b_{j,y} represents the distance between the i-th feature point y in set A and the j-th feature point y in set B;

[0075] The three-dimensional image of the image feature points at different viewing angles is further determined using the triangulation formula, including:

[0076] First, perform depth calculation. Assuming that the baseline distance between the optical centers of the two image acquisition devices is $b$, the focal length of the image acquisition devices is $f$, and the parallax of the corresponding feature points in the two images is $d$, the depth of the feature points is $Z=\frac{fb}{d}$;

[0077] The three-dimensional position of the target object corresponding to the image feature point is further determined by the triangulation formula, and the three-dimensional images of the image feature points under different viewing angles are determined. Assuming that the coordinates of the two image acquisition devices are $(x_1, y_1, z_1)$ and $(x_2, y_2, z_2)$, the coordinates of the feature points on the imaging planes of the two image acquisition devices are expressed as $(u_1, v_1)$ and $(u_2, v_2)$, and the three-dimensional coordinates of the feature points are expressed as $(X, Y, Z)$. $(X, Y, Z)$ is solved by the following formula:

[0078] $\begin{cases}\frac{u_1-x_1}{f}=\frac{X-x_1}{Z}\\frac{v_1-y_1}{f}=\frac{Y-y_1}{Z} \\frac{u_2-x_2}{f}=\frac{X-x_2}{Z}\\frac{v_2-y_2}{f}=\frac{Y-y_2}{Z}\end{cases}$.

[0079] Example 3

[0080] Different from Embodiments 1 and 2, in this embodiment, in step S2, the three-dimensional image obtained in step S1 is first resized, and a possible scaling operation is performed to adjust the image to an appropriate size, and then normalization is performed so that the pixel value of the three-dimensional image is within a preset range. Assuming that the pixel value of the original three-dimensional image is $I(x,y)$, the normalized pixel value is $I_{norm}(x,y)=\frac{I(x,y)}{255}$; a deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$;

[0081] The YOLO algorithm is used to transform the target detection problem into a regression problem. The feature map is divided into grids, and each grid predicts multiple bounding boxes and confidences. Assume that the prediction result of a bounding box is $(bx,by,bw,bh)$, and its true bounding box is $(tx,ty,tw,th)$. The relationship between it and the true bounding box $(tx,ty,tw,th)$ is expressed as confidence, which can be converted by the following formula:

[0082] $b_x=\sigma(t_x)+c_x$

[0083] $b_y=\sigma(t_y)+c_y$

[0084] $b_w=p_we^{t_w}$

[0085] $b_h=p_he^{t_h}$

[0086] Among them, for the formulas $b_x=\sigma(t_x)+c_x$ and $b_y=\sigma(t_y)+c_y$: $b_x$ represents the horizontal coordinate position of the center point of the predicted bounding box, $b_y$ represents the vertical coordinate position of the center point of the predicted bounding box, $t_x$ represents the parameter predicted by the model, representing the horizontal offset of the center point of the predicted bounding box relative to the upper left corner of the grid, $t_y$ represents the parameter predicted by the model, representing the vertical offset of the center point of the predicted bounding box relative to the upper left corner of the grid, $\sigma$ represents the Sigmoid function, which is used to compress the value range of $t_x$ and $t_y$ to $(0,1)$, $c_x$ represents the horizontal coordinate position of the upper left corner of the current grid, and $c_y$ represents the vertical coordinate position of the upper left corner of the current grid; for the formula

[0087] $b_w=p_we^{t_w}$and $b_h=p_he^{t_h}$: $b_w$represents the width of the predicted bounding box, $b_h$represents the height of the predicted bounding box, $t_w$represents the parameter predicted by the model, which is used to adjust the width of the predicted bounding box, $t_h$represents the parameter predicted by the model, which is used to adjust the height of the predicted bounding box, $p_w$represents the width of the prior box, $p_h$represents the height of the prior box, and $e$represents the base of the natural logarithm.

[0088] Use the SSD algorithm to detect targets on multiple feature maps. Each feature map corresponds to a different scale. Combine feature maps of different scales to set the ground truth of bounding boxes of different sizes for target detection.

[0089] Sort the bounding boxes according to the confidence level, select the bounding box with the highest confidence level, calculate the overlap between the bounding box with the highest confidence level and other bounding boxes, and remove them if they exceed the threshold. The overlap is $IoU$, which is calculated as:

[0090] $IoU=\frac{A\cap B}{A\cup B}$

[0091] Among them, $A$ and $B$ are the areas covered by the two bounding boxes respectively.

[0092] Example 4

[0093] Different from Embodiments 1, 2 and 3, in this embodiment, in step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image obtained in step S2;

[0094] In this embodiment, the median filtering algorithm is used to denoise the three-dimensional image obtained in step S2. Assume that the image is $I(x,y)$, the filter window size is $n\times n$, and the pixel value of the three-dimensional image after filtering is

[0095] $I_{filtered}(x,y)=median{I(x+i,y+j)\mid-k\leq i,j\leq k}$, where $k=\frac{n-1}{2}$;

[0096] Then use the mean filtering algorithm to filter and smooth the image to highlight important features. The pixel value of the three-dimensional image after using the mean filtering algorithm is $I_{filtered}(x,y)=\frac{1}{n^2}\sum_{i=-k}^{k}\sum_{j=-k}^{k}I(x+i,y+j)$, where $k=\frac{n-1}{2}$, $n\times n$ is the filter window size;

[0097] The data value range of the denoised and filtered three-dimensional image is normalized to normalize the pixel value to the interval [0,1]. The normalization formula is as follows:

[0098] $Normalized\Pixel\Value=\frac{Original\Pixel\Value}{Maximum\Pixel\Value}$;

[0099] The contrast of the three-dimensional image is adjusted using a histogram equalization algorithm. For the adjustment of the brightness of the three-dimensional image, assuming that the original brightness is $L$ and the adjustment value is $\Delta L$, the new brightness is $New\Brightness=(L+\Delta L)\mod\256$. The color saturation of the three-dimensional image is adjusted by changing the saturation component in the color space for image enhancement and correction.

[0100] Image correction eliminates or reduces distortion caused by factors such as lighting and shadows, ensuring that the objects and scenes in the image more realistically and accurately reflect the actual environment.

[0101] Example 5

[0102] Different from Embodiments 1, 2, 3 and 4, in this embodiment, step S4 specifically includes:

[0103] The three-dimensional image obtained in step S3 is normalized. Assuming that the pixel value of the three-dimensional image obtained in step S3 is $I(x,y)$, the normalized value is $I_{norm}(x,y)=\frac{I(x,y)}{255}$;

[0104] Convolutional neural networks are used for feature extraction. Specifically, other algorithms under deep learning algorithms, such as the OpenPose algorithm, can also be used for feature extraction; coordinate prediction is performed through the fully connected layer of the convolutional neural network to determine the positions of human joints, such as the head, shoulders, elbows, hands, hips, knees, and feet. Assuming that the input three-dimensional image is $I$, the prediction result of the joint point coordinates obtained by the fully connected layer of the convolutional neural network is $(x_1, y_1, x_2, y_2, …, x_n, y_n)$, where $(x_i, y_i)$ is the coordinate of the $i$th joint point;

[0105] Convolutional neural networks are used to extract the motion characteristics of the human body. Specifically, other algorithms under deep learning algorithms can also be used, such as step length, step frequency, gait cycle, etc. The optical flow algorithm is used to calculate the movement speed and direction of pixels in the three-dimensional image to obtain the movement trajectory of the human body. Assuming that the sequence of three-dimensional images is ${I_1,I_2,…,I_t}$, the optical flow calculation formula is:

[0106] $(\vec{v}(x,y))=(v_x,v_y)=(\frac{\partial I}{\partial t}) / (\frac{\partial I}{\partial x}+\frac{\partial I}{\partial y})$

[0107] Among them, $\vec{v}(x,y)$ is the optical flow vector at the pixel point $(x,y)$, $v_x$ and $v_y$ are the velocity components in the horizontal and vertical directions respectively;

[0108] A convolutional neural network is used to extract facial features. Specifically, a face recognition method based on feature faces or other algorithms under a deep learning algorithm can also be used, and then compared with known faces in a database to determine the identity of the person. Assume that the input face image is $I$, the feature vector extracted by the convolutional neural network is $\vec{f}$, and the feature vector in the database is ${\vec{f}_1,\vec{f}_2,…,\vec{f}m}$;

[0109] Compare $\vec{f}$ with ${\vec{f}_1,\vec{f}_2,…,\vec{f}m}$ and calculate the similarity using the Euclidean distance method. The Euclidean distance calculation formula is as follows:

[0110] $d(\vec{f},\vec{f}i)=\sqrt{\sum{j=1}^{n}(f_j-f{i,j})^2}$, where $n$ is the dimension of the feature vector;

[0111] Determine human posture based on the position of human joints, perform gait analysis based on the movement trajectory of the human body, perform face recognition based on facial feature comparison, and combine human posture, gait analysis and face recognition to determine whether the human body has abnormal behavior;

[0112] For example, human posture estimation can be used to determine the position and movement of a person's hands, and combined with the smoke features in the image (which can be detected by color, shape, etc.) to determine whether the person is smoking. For the behavior of using a mobile phone to make a call, human posture estimation can be used to determine the position of the person's hands and ears, as well as the features of the mobile phone (such as shape, color, etc.) to determine whether the person is making a call. For the behavior of not wearing a helmet in an environment where a helmet is required, human posture estimation can be used to determine the position of the person's head, and combined with the features of the helmet (such as color, shape, etc.) to make a judgment.

[0113] Example 6

[0114] Different from Embodiments 1, 2, 3, 4 and 5, in this embodiment, in step S5, the behavioral events of the human body are continuously monitored. It is assumed that the set of behavioral events is $E={e_1, e_2, …, e_n}$, where $e_i$ represents a behavioral event, and behavioral events that occur multiple times within a preset time are identified by comparing behavioral features. It is assumed that behavioral event $e_i$ and behavioral event $e_j$ occur within the preset time, and the feature similarity between behavioral event $e_i$ and behavioral event $e_j$ exceeds a preset value, then it is identified as a repeated behavior;

[0115] Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of behavior events, and obtain a new set of behavior events, which is $E'=ER$. The new set of behavior events is the abnormal behaviors that are determined to be abnormal.

[0116] When processing data, by identifying and filtering repeated behaviors and obtaining truly abnormal behaviors that are determined to be abnormal, the data can be ensured to be up-to-date and accurate, and data processing efficiency and quality can be improved;

[0117] In step S7, the situation prediction module performs headcount, flow analysis and aggregation analysis on the human behaviors collected in step S1;

[0118] Use a human body detection algorithm to count the number of people in the three-dimensional image collected in step S1. Assuming that the video frame of the three-dimensional image is $F$ and the human body detection algorithm is $D$, the number of people $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at the coordinate $(x,y)$;

[0119] Use the human body detection algorithm to analyze the gait obtained in step S3, perform flow analysis on the moving direction and speed, and obtain the overall flow trend. Assuming that the position sequence of the human body is ${(x_1, y_1), (x_2, y_2), …, (x_t, y_t)}$, the moving direction is $\vec{d}=(x_t-x_1, y_t-y_1)$, and the moving speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$;

[0120] Use the human body detection algorithm to count the number of people in the three-dimensional image collected in step S1, and use the crowd density calculation method to calculate the crowd density, and obtain the crowd-dense area for analysis. Assuming that the video area in the three-dimensional image is divided into $m\times n$ grids, and the number of people in each grid is $p_{i,j}$, the crowd density is:

[0121] $Density=\frac{\sum_{i=1}^{m}\sum_{j=1}^{n}p_{i,j}}{m\times n}$,

[0122] Among them, $m$ represents the number of grids into which the video area of ​​the three-dimensional image is divided horizontally, $n$ represents the number of grids into which the video area of ​​the three-dimensional image is divided vertically, $i$ represents the row index in the horizontally divided grid, and the value range is from 1 to $m$, $j$ represents the column index in the vertically divided grid, and the value range is from 1 to $n$, $p_{i,j}$ represents the number of people in the specific grid of the $i$th row and the $j$th column in the video area of ​​the three-dimensional image,

[0123] $Density$ represents the crowd density of the entire video area, $\sum_{i=1}^{m}\sum_{j=1}^{n}p_{i,j}$ represents the sum of the number of people in all $m\times n$ grids, that is, the number of people in all grids in each row is first summed up (internal $\sum_{j=1}^{n}p_{i,j}$), and then the sum of all rows is summed up (external $\sum_{i=1}^{m}$), and $m\times n$ represents the total number of grids after the video area is divided.

[0124] The situation prediction module uses big data and machine learning technology to track and statistically analyze human behavior in 3D image data in real time. Through deep mining of 3D image data, people counting can be performed, that is, identifying and counting each human body in the video, which is helpful for safety management, such as ensuring that the number of people is within a safe range, and reflecting the progress of the project to adjust the plan; flow analysis is to track the movement trajectory of the human body, analyze the movement direction and speed to obtain the overall flow trend, which is conducive to planning safe passages and evacuation routes, and provide a basis for optimizing the construction layout; aggregation analysis detects dense areas in the video, analyzes the degree of crowd aggregation and distribution, can avoid the risk of accidents caused by excessive crowds, and can reasonably allocate resources.

[0125] Example 7

[0126] The embodiment of the present invention also provides a system for performing intelligent behavior analysis using the intelligent behavior analysis method based on the multi-vision recognition algorithm as described above, comprising a multi-vision acquisition module, a multi-target detection module, a data processing module, a behavior analysis module, a repeated behavior filtering module, an alarm and feedback module, and a situation prediction module;

[0127] The multi-viewing acquisition module is used to acquire images from different viewing angles, extract feature points, match the feature points of images from different viewing angles, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image.

[0128] The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image that meets the target;

[0129] The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image;

[0130] The behavior analysis module is used for human posture judgment, gait analysis and face recognition in 3D images, and combines the above indicators to judge whether the human body has abnormal behavior;

[0131] The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal;

[0132] The alarm and feedback module is used to issue alarm information for abnormal behavior;

[0133] The situation prediction module is used to perform population counting, flow analysis and aggregation analysis on the collected three-dimensional images.

[0134] The application scope of the intelligent behavior analysis system based on multi-visual recognition algorithm of the present invention is not limited to the management of construction sites such as water conservancy and hydropower, wind power, and solar power generation, but can also be widely used in communities, streets, and other application scenarios; whether in the public areas of the community or on busy streets, it can play its powerful monitoring and early warning functions to protect the safety of residents and pedestrians; in the field of traffic management, the intelligent behavior analysis system based on multi-visual recognition algorithm of the present invention can track the movement trajectory of vehicles and pedestrians in real time, timely discover problems such as traffic congestion and violations, and provide solutions for traffic management departments, effectively alleviate traffic pressure, and ensure smooth roads; in the field of education, it can be used to monitor students' classroom performance and provide teachers with personalized teaching suggestions; in the medical field, it can assist doctors in remote diagnosis and treatment to improve the quality and efficiency of medical services. Through wide application in different scenarios, the intelligent behavior analysis system based on multi-visual recognition algorithm of the present invention can bring a safer and more orderly environment to society; of course, further, it can also provide deeper insights and decision support through its refined analysis capabilities. In short, this system can not only play an important role in the traditional security management field, but also has its unique value and potential in many fields such as retail, transportation, education, and medical care.

[0135] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the application scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent behavior analysis method based on multi-visual recognition algorithm, characterized in that: The following steps are involved: S1. Use an image acquisition device to acquire images at different viewing angles, extract image feature points at different viewing angles, match the image feature points at different viewing angles, calculate the disparity and depth according to the position difference of the successfully matched feature points in different images, and acquire a three-dimensional image; S2, normalizing the three-dimensional image, extracting features of the three-dimensional image, obtaining a feature map of the three-dimensional image, and performing target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body; S3, performing denoising, filtering and normalization processing on the three-dimensional human body image obtained in step S2, and performing image enhancement and correction on the three-dimensional human body image; S4, performing human posture judgment, gait analysis and face recognition on the three-dimensional human body image obtained in step S3, and judging whether the human body has abnormal behavior; S5, detecting and recording abnormal behaviors of the human body in the three-dimensional image, identifying repeated abnormal behaviors that occur multiple times within a preset time period, filtering repeated behaviors, and obtaining abnormal behaviors that are determined to be abnormal; S6. For the abnormal behavior determined as abnormal in step S5, an alarm message is issued and the abnormal behavior is recorded.

2. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S1, before collecting images from different perspectives, a distributed image acquisition system method is used to collect multi-visual data, a deep learning classification model is used to classify and label the multi-visual data and record the location information, behavior labels are set for the multi-visual data, and the multi-visual data with behavior labels of abnormal human behavior are screened to build an abnormal human behavior database.

3. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: In step S1, a scale space is constructed by a scale space feature extraction algorithm, and the scale space feature extraction algorithm is used to extract image feature points at different viewing angles.

4. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: The scale space feature extraction algorithm is one of the SIFT algorithm, SURF algorithm, and ORB algorithm.

5. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 3 is characterized in that: In step S1, the image feature points at different viewing angles are matched by a brute force matching algorithm, and the Euclidean distance of the image feature points at different viewing angles is calculated by the brute force matching algorithm to find the matching feature points, and the successfully matched feature points are obtained. The disparity and depth are calculated according to the position difference of the successfully matched feature points in different images, and the three-dimensional position of the target object corresponding to the image feature points is further determined by the triangulation formula, and the three-dimensional image of the image feature points at different viewing angles is determined.

6. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: In step S2, the three-dimensional image obtained in step S1 is first resized and then normalized so that the pixel value of the three-dimensional image is within a preset range. Assuming that the pixel value of the original three-dimensional image is $I(x,y)$, the pixel value after normalization is $I_{norm}(x,y)=\frac{I(x,y)}{255}$; A deep neural network is used to extract features from the normalized three-dimensional image. Assuming that the input three-dimensional image is $I$, the feature map obtained after feature extraction by the deep neural network is $F$.

7. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: In step S2, the target detection problem is converted into a regression problem using the YOLO algorithm, and a grid is divided on the feature map, and each grid predicts the confidence of multiple bounding boxes; The SSD algorithm is used to perform target detection on multiple feature maps. Each feature map corresponds to a different scale. The feature maps of different scales are combined to set the true values ​​of bounding boxes of different sizes for target detection to obtain a three-dimensional image of the human body.

8. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 7 is characterized in that: In step S2, the bounding boxes are sorted according to the confidence, the bounding box with the highest confidence is selected, and the overlap between the bounding box with the highest confidence and other bounding boxes is calculated. If the overlap exceeds a threshold, it is removed.

9. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: In step S3, a median filter algorithm or a Gaussian filter algorithm is used to denoise the three-dimensional image of the human body obtained in step S2.

10. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 9, characterized in that: In step S3, the three-dimensional human body image obtained in step S2 is denoised using a median filtering algorithm, and filtered using a mean filtering algorithm to smooth the image and highlight important features, and the data value range of the three-dimensional human body image after denoising and filtering is normalized.

11. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 10, characterized in that: In step S3, the contrast of the three-dimensional image is adjusted using a histogram equalization algorithm; for the adjustment of the brightness of the three-dimensional image, assuming that the original brightness is $L$, the adjustment value is $\Delta L$, and the new brightness is $New\Brightness=(L+\Delta L)\mod\256$; the color saturation of the three-dimensional image is adjusted by changing the saturation component in the color space to perform image enhancement and correction.

12. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: Step S4 specifically includes: Convolutional neural network is used for feature extraction, and coordinate prediction is performed through the fully connected layer of the convolutional neural network to determine the position of human joints; The convolutional neural network is used to extract the motion characteristics of the human body, and the optical flow algorithm is used to calculate the motion speed and direction of the pixels in the three-dimensional image to obtain the motion trajectory of the human body; Convolutional neural networks are used to extract facial features, which are then compared with known faces in the database to determine the identity of the person; The human body posture is judged according to the position of the human joints, the gait analysis is performed according to the movement trajectory of the human body, and the face recognition is performed according to the comparison of facial features. The human body posture, gait analysis and face recognition are combined to determine whether the human body has abnormal behavior.

13. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 2 is characterized in that: In step S5, the abnormal behavior of the human body is continuously monitored. Assuming that the set of abnormal behavior events is $E={e_1, e_2,…, e_n}$, the abnormal behavior events that occur multiple times within a preset time are identified by comparing the behavior features. Assuming that the abnormal behavior event that occurs within the preset time is $e_i$, and the other abnormal behavior event is $e_j$, if the feature similarity between the abnormal behavior event $e_i$ and the other abnormal behavior event $e_j$ exceeds a preset value, it is identified as repeated behavior; Assume that the set of repeated behaviors is $R={r_1,r_2,…,r_m}$, remove the set of repeated behaviors from the set of abnormal behavior events, and obtain the set of behavior events actually judged to be abnormal. The set of behavior events actually judged to be abnormal is $E'=ER$.

14. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 1 is characterized in that: Perform population counting, flow direction analysis and aggregation analysis on the three-dimensional image collected in step S1.

15. The intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 14 is characterized in that , Use a human body detection algorithm to count the number of people in the three-dimensional image collected in step S1. Assume that the video frame of the three-dimensional image is $F$, the human body detection algorithm is $D$, and the number of people is $N$, then the number of people $N=\sum_{x,y}D(F(x,y))$, where $D(F(x,y))$ indicates whether a human body is detected at the coordinate $(x,y)$; Use the human body detection algorithm to analyze the gait obtained in step S4, perform flow analysis of the moving direction and speed, and obtain the overall flow trend. Assume that the position sequence of the human body is ${(x_1, y_1), (x_2, y_2), …, (x_t, y_t)}$, the moving direction is $\vec{d}=(x_t-x_1, y_t-y_1)$, and the moving speed is $v=\sqrt{(x_t-x_1)^2+(y_t-y_1)^2} / t$; The human body detection algorithm is used to count the number of people in the three-dimensional image collected in step S1, and the crowd density is calculated using the crowd density calculation method to obtain the crowd-dense area for aggregation analysis.

16. A system for performing intelligent behavior analysis using the intelligent behavior analysis method based on multi-visual recognition algorithm as described in any one of claims 1 to 15, characterized in that: It includes multi-eye vision acquisition module, multi-target detection module, data processing module, behavior analysis module, repeated behavior filtering module and alarm and feedback module; The multi-viewing acquisition module is used to acquire images from different viewing angles, extract feature points, match the feature points of images from different viewing angles, calculate the disparity and depth based on the position differences of the successfully matched feature points in different images, and obtain a three-dimensional image. The multi-target detection module is used to extract the feature map of the three-dimensional image and perform target detection on the feature map of the three-dimensional image to obtain a three-dimensional image of the human body; The data preprocessing module is used to perform denoising, filtering and normalization on the three-dimensional image, and to perform image enhancement and correction on the three-dimensional image of the human body; The behavior analysis module is used for human posture judgment, gait analysis and face recognition in human 3D images, and determines whether the human body has abnormal behavior; The repeated behavior filtering module is used to identify repeated abnormal behaviors that occur multiple times within a preset time period, filter the repeated behaviors, and obtain abnormal behaviors that are determined to be abnormal; The alarm and feedback module is used to issue alarm information for abnormal behavior.

17. The system for performing intelligent behavior analysis based on the intelligent behavior analysis method based on multi-visual recognition algorithm according to claim 16, characterized in that: The multi-viewing acquisition module includes a situation prediction module, which is used to perform population counting, flow analysis and aggregation analysis on the acquired three-dimensional images.

Images