Crowd motion saliency detection method based on Langevin fluid driven model

Abstract

The present invention belongs to the technical field of video analysis and image processing, and discloses a crowd motion salient region detection algorithm in a large-scale scene based on a Langevin model and directional self-information. The Langevin model can be used to construct the interaction relationship between particles. The crowd particles are regarded as particle motion in a fluid, and the Langevin model is used to obtain the resultant external force on the particles, and the directional self-information of the particles is extracted based on a coherent vector field and two features are fused to construct the saliency value of the particles, and binary segmentation is performed according to the difference in saliency values ​​of each position, and finally, the salient region in the crowd can be detected after basic image processing. The present invention has good applicability and can be applied to large-scale crowd video scenes. At the same time, the algorithm is made more robust by constructing a crowd coherent flow field. In addition, the feature fusion mechanism enables the method to adapt to a wider range of crowd motion scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of video analysis and image processing, and in particular relates to a crowd motion saliency detection method based on a Langevin fluid driven model. Background Art

[0002] Crowd behavior analysis is a hot topic and a difficult topic in the field of computer vision. In the past, supervision of public places mostly relied on on-site security personnel. However, relying solely on staff to monitor these areas often cannot meet the needs. First of all, relying on manual supervision requires a lot of human resources; and people's attention will decrease over time, which is easy to produce fatigue supervision psychology, resulting in greatly reduced monitoring efficiency; in addition, the scope of human supervision area is relatively limited, especially in large-scale crowd scenes. When a crowd safety incident occurs in a certain area, the staff cannot communicate and interact in real time, which may eventually lead to adverse crowd safety incidents. Therefore, in order to make up for the shortcomings of manual monitoring and minimize the probability of crowd safety incidents, automatic real-time monitoring systems have become an urgent need for countries around the world. Thanks to the continuous development of computer technology, crowd safety has become a hot research issue in the field of computer vision. It aims to monitor and discover significant areas in the crowd in real time and alarm in time to achieve the purpose of reducing people's personal and property safety.

[0003] Due to the mutual occlusion of crowd movement in dynamic scenes and the uneven distribution of crowd density, crowd video surveillance itself is a difficult task in computer vision, and real-time detection of crowd safety incidents in crowd scenes will be an even greater challenge. Researchers have different definitions of significant crowd areas. Some regard situations with a lower probability of occurrence than mainstream behaviors in the scene as significant crowd areas, while others regard behaviors that deviate from the mainstream movement of the scene as significant areas. Despite the different perspectives, they all focus on areas that are prone to crowd safety incidents. Therefore, how to detect these significant areas in real time has become a key research task in video surveillance.

[0004] The fluid-driven model can reflect the mutual influence between particles. If we model the crowd as particles moving in a fluid, the interaction between particles in the crowd can be described with the help of the fluid-driven model. Therefore, fluid-driven can be used as a powerful tool for saliency detection in large-scale crowd scenes. Summary of the invention

[0005] The present invention aims to provide a crowd motion saliency detection method based on a Langevin fluid driven model to solve the technical problem of crowd safety that often occurs in crowd scenes.

[0006] In order to solve the above technical problems, the specific technical solution of the crowd motion saliency detection method based on the Langevin fluid driven model of the present invention is as follows:

[0007] A crowd motion saliency detection method based on a Langevin fluid driven model comprises the following steps:

[0008] Step 1: Use the Horn-Schunck optical flow algorithm to extract the crowd motion optical flow field;

[0009] Step 2: Regularize the optical flow field by referring to the physical "heat diffusion" idea;

[0010] Step 3: Solve particle characteristics;

[0011] Step 4: Feature thresholding and feature fusion;

[0012] Furthermore, the step 1 includes the following specific steps:

[0013] Step 1.1: Use the Horn-Schunck optical flow algorithm to process the frame sequence extracted from the selected video;

[0014] In the input video sequence, two consecutive frames are extracted, and the motion field of the particle is extracted through the position change of the same pixel point (a point with unchanged pixel value in the local neighborhood) between the two frames. Let p i is a point in two-dimensional motion, its coordinates are (x, y), and the optical flow size is (u, v), then p i The optical flow at the point is represented as (x, y, u, v), and finally the set of optical flows of all points is called the optical flow field, which can provide the motion information of the pixels in the video.

[0015] Step 1.2: From step 1.1, we can get the optical flow field set of particles in the large-scale crowd scene: The velocity direction set is n is the total number of particles in the crowd.

[0016] Furthermore, the step 2 includes the following specific steps:

[0017] Step 2.1: Based on the idea of ​​“heat diffusion” in physics, we construct formula (1) to obtain the coherent vector of the particle at position p receiving the diffusion from the particle at position q: The coherence vector is constructed as follows:

[0018]

[0019] in, represents the coherence vector of the particle at position p that diffuses from the particle at position q after a time interval l, represents the particle optical flow at p, λ = (x, y), dist(p, q) represents the pixel distance between particles, k p is the energy diffusion coefficient, k t is the optical flow diffusion coefficient, k t =0.2, Represent the optical flow direction vector at position p and q respectively, Indicates the similarity of the optical flow at positions p and q, where Q is a threshold;

[0020] Step 2.2: After obtaining the particle motion coherence vector, the particle coherence vector is averaged in the time and space domain to obtain the coherent flow field vector. The processing process is described by formula (2):

[0021]

[0022] Among them, ε 2 is the number of neighborhood particles, Represent the horizontal and vertical flow field vectors at time t respectively.

[0023] Furthermore, the step 3 includes the following specific steps:

[0024] Step 3.1: Consider the movement of crowd particles as an ideal fluid and use Stokes' law to describe the viscous resistance F of the fluid r (t), the Stokes viscous drag formula is:

[0025]

[0026] in, is the viscous force on the particle at time t, a is the volume of the particle, let a = 1, ζ is the viscosity of the fluid, and take is the velocity of the particle at time t;

[0027] Step 3.2: Model the crowd as fluid particles and use the Langevin model to simulate the crowd's motion behavior to obtain the interaction force between particles. The Langevin equation of particles in the fluid is as follows:

[0028]

[0029] Where m is the particle mass, is the particle velocity, is the viscosity of the fluid, is the net external force on the particle, is the random fluctuation force;

[0030] By comparing the motion characteristics of particles in the fluid and those in the crowd, we can find that the relationship between the external force and the random force in the crowd particles is satisfied. Combined with formula (4), we can get the resultant force on the particle: for:

[0031]

[0032] in, is the total external force on the particle, m is the mass of the particle, is the particle velocity, is the viscosity of the fluid;

[0033] Step 3.3: Through the coherent flow field, the flow field vector set in the crowd coherent vector field is n is the total number of all coherent vectors. In the crowd area obtained, the motion direction angle is divided into eight direction levels on [-π,π]. The direction angle of the particle is calculated by the direction set. The direction angle calculation formula is as follows:

[0034]

[0035] Where θ is the direction angle of the moving particle, are the particle horizontal and vertical flow field vectors respectively;

[0036] Step 3.4: Divide the crowd movement directions into eight, and count the probability P of the number of crowd particles in each direction to the total number of particles x , the self-information value in each direction is calculated by the self-information formula of information theory. The self-information formula is described as follows:

[0037] I x = -log2(P x ),(x=1,2…8) (7)

[0038] Among them, I x represents the self-information in the direction of level x, P x is the probability of a particle whose direction is at level x;

[0039] Step 3.5, obtain the self-information set I = (I1, I2…I8) in each direction through (7), and obtain the final particle self-information significance feature by traversing all particles.

[0040] Furthermore, step 4 includes the following specific steps:

[0041] Step 4.1: Extract the resultant force and self-information threshold T respectively through Otsu algorithm F , T I , and then threshold them respectively. The specific operations are as follows:

[0042]

[0043] Among them, F″ and I″ are the results of the thresholding of the resultant force and self-information, respectively, and F′ and I′ are the normalized representations of the resultant force and self-information. F , T I is the threshold of the resultant force and self-information;

[0044] Step 4.2: Fuse the thresholded features to obtain the significance value R of all particles. In the significant area of ​​crowd movement, the significance value of the particle is large, and in the mainstream crowd area, the significance value of the particle is small. The fusion process is described as follows:

[0045]

[0046] Among them, R is the result after feature fusion;

[0047] Step 4.3: Filter the saliency detection results using image mean filtering technology.

[0048] The crowd motion saliency detection method based on the Langevin fluid driven model of the present invention has the following advantages: the present invention makes the algorithm have good applicability by modeling the crowd as particle motion in the fluid, and can be applied to large-scale crowd video scenes. At the same time, the algorithm is more robust by constructing a crowd coherent flow field. In addition, the feature fusion mechanism enables the method to adapt to a wider range of crowd motion scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the algorithm flow chart of the present invention;

[0050] Figure 2 The optical flow graph and the coherent flow field graph of the present invention;

[0051] Figure 3 It is a motion direction division diagram of the present invention;

[0052] Figure 4 The experimental results of the present invention are shown in FIG. Figure 1 ;

[0053] Figure 5 The experimental results of the present invention are shown in FIG. Figure 2 ;

[0054] Figure 6 The experimental results of the present invention are shown in FIG. Figure 3 ;

[0055] Figure 7 The experimental results of the present invention are shown in FIG. Figure 4 ; DETAILED DESCRIPTION

[0056] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a crowd motion saliency detection method based on a Langevin fluid driven model of the present invention in conjunction with the accompanying drawings.

[0057] The present invention firstly uses the physical "heat diffusion" idea to regularize the crowd motion vector field to obtain the crowd coherent flow field, and regards the crowd particles as particle motion in the fluid, and extracts the resultant force on the particles by modeling the particle trajectory through the particle Langevin model, and then constructs the particle direction self-information through the self-information theory, and finally obtains the significance value of all particles. The significance value of particles in the significant area of ​​crowd motion is larger, and the significance value of particles in the normal area is smaller.

[0058] like Figure 1 As shown, the crowd motion significance detection method based on the Langevin fluid driven model of the present invention comprises the following steps:

[0059] Step 1, use the Horn-Schunck optical flow algorithm to extract the crowd motion optical flow field;

[0060] Step 1.1, using the Horn-Schunck optical flow algorithm to process the frame sequence extracted from the selected video;

[0061] In the input video sequence, we extract two consecutive frames and extract the motion field of the particle through the position change of the same pixel point (a point with unchanged pixel value in the local neighborhood) between the two frames. Let p i is a point in two-dimensional motion, its coordinates are (x, y), and the optical flow size is (u, v), then p i The optical flow at the point can be expressed as (x, y, u, v), and the set of optical flows of all points is called the optical flow field. The optical flow field can provide the motion information of the pixels in the video.

[0062] Step 1.2: From step 1.1, we can get the optical flow field set of particles in a large-scale crowd scene: The velocity direction set is n is the total number of particles in the crowd.

[0063] Step 2: In the crowd vector field obtained, the optical flow is regularized by referring to the physical "heat diffusion" idea to construct the crowd motion coherent flow field.

[0064] Step 2.1: From thermodynamics, we know that heat always diffuses from high temperature to low temperature, and eventually all regions of the system reach a stable equilibrium state. Similarly, when extracting the motion vector of crowd particles, the influence of external factors (such as rapid changes in the scene, camera shaking, the complexity of the scene itself, etc.) may cause noise in the extracted features, which in turn leads to errors in subsequent operations based on motion vectors. Therefore, in order to eliminate the influence of noise, we draw on the idea of ​​"heat diffusion" in physics and construct formula (1) to obtain the coherent vector diffused by the particle at position p from the particle at position q: The coherence vector is constructed as follows:

[0065]

[0066] in, represents the coherence vector of the particle at position p that diffuses from the particle at position q after a time interval l, represents the optical flow of the particle at p, λ = (x, y), dist(p, q) represents the Euclidean distance between positions p and q, and in this paper represents the pixel distance between particles, k p is the energy diffusion coefficient, and the present invention takes k p =0.8, k t is the optical flow diffusion coefficient, k t =0.2, Represent the optical flow direction vector at position p and q respectively, It indicates the similarity of the optical flow at positions p and q. Q is a threshold, and Q=0.7 is taken.

[0067] Step 2.2, after obtaining the particle motion coherence vector, we also perform mean processing on the particle coherence vector in the time and space domain to obtain the coherent flow field vector. The processing process is described by formula (2):

[0068]

[0069] Among them, ε 2 is the number of neighborhood particles, Represent the horizontal and vertical flow field vectors at time t respectively. The particle optical flow field and the coherent flow field are as follows Figure 2 shown.

[0070] Step 3: Model the motion of large-scale crowd particles as an ideal fluid, and use the Langevin model to model the motion trajectory of the particles to obtain the resultant force on the particles. At the same time, use information theory to construct the direction self-information of the particles.

[0071] Step 3.1: Consider the movement of crowd particles as an ideal fluid and use Stokes' law to describe the viscous resistance F of the fluid. r (t), the Stokes viscous drag formula is:

[0072]

[0073] in, is the viscous force on the particle at time t, a is the volume of the particle, in the present invention, a=1, ζ is the viscosity of the fluid, and is the velocity of the particle at time t.

[0074] Step 3.2, model the crowd as fluid particles, and use the Langevin model to simulate the crowd's motion behavior to obtain the interaction force between particles. The Langevin equation of particles in the fluid is as follows:

[0075]

[0076] Wherein, m is the particle mass, and in the present invention, m=1. is the particle velocity, is the viscosity of the fluid, is the net external force on the particle, is the random fluctuation force.

[0077] By comparing the motion characteristics of particles in the fluid and those in the crowd, we can get the relationship between the external force and the random force in the crowd particles. Combined with formula (4), we can get the resultant force on the particle: for:

[0078]

[0079] in, is the total external force on the particle, m is the mass of the particle, is the particle velocity, is the viscosity of the fluid.

[0080] Step 3.3, through the coherent flow field in the present invention, the flow field vector set in the crowd coherent vector field is n is the total number of all coherent vectors. In the obtained crowd area, the motion direction angle is divided into eight direction levels on [-π,π]. The level division strategy is as follows: Figure 3 As shown, the direction angle of the particle can be calculated through the direction set. The direction angle calculation formula is as follows:

[0081]

[0082] Where θ is the direction angle of the moving particle, are the particle horizontal and vertical flow field vectors respectively.

[0083] Step 3.4: Divide the crowd movement directions into eight, and count the probability P of the number of crowd particles in each direction to the total number of particles x , the self-information value in each direction can be calculated through the self-information formula of information theory. The self-information formula is described as follows:

[0084] I x = -log2(P x ),(x=1,2…8) (7)

[0085] Among them, I x Represents the self-information in the direction of level x. x is the probability of a particle whose direction is at level x.

[0086] Step 3.5, obtain the self-information set I = (I1, I2…I8) in each direction through (7). By traversing all particles, obtain the final particle self-information saliency feature.

[0087] Step 4: Threshold the extracted features through image processing technology, and fuse the processed results accordingly to construct the final particle significance value.

[0088] Step 4.1: Extract the resultant force and self-information threshold T respectively through Otsu algorithm F , T I , and then threshold them respectively. The specific operations are as follows:

[0089]

[0090] Among them, F″ and I″ are the results of the thresholding of the resultant force and self-information, respectively, and F′ and I′ are the normalized representations of the resultant force and self-information. F , T I is the threshold of the resultant force and self-information.

[0091] Step 4.2, the thresholded features are fused to obtain the significance value R of all particles. In the significant area of ​​crowd movement, the significance value of the particle is larger, and in the mainstream crowd area, the significance value of the particle is smaller. The fusion process is described as follows:

[0092]

[0093] Among them, R is the result after feature fusion.

[0094] Step 4.3: Filter the saliency detection results using image mean filtering technology.

[0095] like Figure 4-Figure 7The figure shows the experimental results of the method proposed in this invention. In different crowd scenes, the detected salient areas are displayed using heat maps. Figure 4-Figure 7 It can be seen that the method of the present invention can display significant areas in a crowd.

[0096] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A crowd motion saliency detection method based on Langevin fluid driven model, characterized in that: The steps include: Step 1: Use the Horn-Schunck optical flow algorithm to extract the crowd motion optical flow field; Step 1.1: Use the Horn-Schunck optical flow algorithm to process the frame sequence extracted from the selected video; In the input video sequence, two consecutive frames are extracted, and the motion field of the particle is extracted through the position change of the same pixel point (a point with unchanged pixel value in the local neighborhood) between the two frames. Let p i is a point in two-dimensional motion, its coordinates are (x, y), and the optical flow size is (u, v), then p i The optical flow at the point is represented as (x, y, u, v), and finally the set of optical flows of all points is called the optical flow field, which can provide the motion information of the pixels in the video. Step 1.2: From step 1.1, we can get the optical flow field set of particles in the large-scale crowd scene: The velocity direction set is n is the total number of particles in the crowd; Step 2: Regularize the optical flow field by referring to the physical "heat diffusion" idea; Step 2.1: Using the idea of ​​"heat diffusion" in physics, construct formula (1) to obtain the coherent vector of the particle at position p receiving the diffusion from the particle at position q: The coherence vector is constructed as follows: in, represents the coherence vector of the particle at position p that diffuses from the particle at position q after a time interval l, represents the particle optical flow at p, λ = (x, y), dist(p, q) represents the pixel distance between particles, kp is the energy diffusion coefficient, k t is the optical flow diffusion coefficient, k t =0.2, Represent the optical flow direction vector at position p and q respectively, Indicates the similarity of the optical flow at positions p and q, where Q is a threshold; Step 2.2: After obtaining the particle motion coherence vector, the particle coherence vector is averaged in the time and space domain to obtain the coherent flow field vector. The processing process is described by formula (2): Among them, ε 2 is the number of neighborhood particles, represent the horizontal and vertical flow field vectors at time t respectively; Step 3: Solve particle characteristics; Step 3.1: Consider the movement of crowd particles as an ideal fluid and use Stokes' law to describe the viscous resistance F of the fluid r (t), the Stokes viscous drag formula is: in, is the viscous force on the particle at time t, a is the volume of the particle, let a = 1, ζ is the viscosity of the fluid, and take is the velocity of the particle at time t; Step 3.2: Model the crowd as fluid particles and use the Langevin model to simulate the crowd's motion behavior to obtain the interaction force between particles. The Langevin equation of particles in the fluid is as follows: Where m is the particle mass, is the particle velocity, is the viscosity of the fluid, is the net external force on the particle, is the random fluctuation force; By comparing the motion characteristics of particles in the fluid and those in the crowd, we can find that the relationship between the external force and the random force in the crowd particles is satisfied. Combined with formula (4), we can get the resultant force on the particle: for: in, is the total external force on the particle, m is the mass of the particle, is the particle velocity, is the viscosity of the fluid; Step 3.3: Through the coherent flow field, the flow field vector set in the crowd coherent vector field is n is the total number of all coherent vectors. In the crowd area obtained, the motion direction angle is divided into eight direction levels on [-π,π]. The direction angle of the particle is calculated by the direction set. The direction angle calculation formula is as follows: Where θ is the direction angle of the moving particle, are the particle horizontal and vertical flow field vectors respectively; Step 3.4: Divide the crowd movement directions into eight, and count the probability P of the number of crowd particles in each direction to the total number of particles x , the self-information value in each direction is calculated by the self-information formula of information theory. The self-information formula is described as follows: I x =-log2(P x ),(x=1,2…8)(7) Among them, I x represents the self-information in the direction of level x, P x is the probability of a particle whose direction is at level x; Step 3.5, obtain the self-information set I = (I1, I2…I8) in each direction through (7), and obtain the final particle self-information significance feature by traversing all particles; Step 4: Feature thresholding and feature fusion.

2. The method for detecting crowd motion significance based on the Langevin fluid driven model according to claim 1, characterized in that: The step 4 comprises the following specific steps: Step 4.1: Extract the resultant force and self-information threshold T respectively through Otsu algorithm F , T I , and then threshold them respectively. The specific operations are as follows: Among them, F″ and I″ are the results of the thresholding of the resultant force and self-information, respectively, and F′ and I′ are the normalized representations of the resultant force and self-information. F , T I is the threshold of the resultant force and self-information; Step 4.2: Fuse the thresholded features to obtain the significance value R of all particles. In the significant area of ​​crowd movement, the significance value of the particle is large, and in the mainstream crowd area, the significance value of the particle is small. The fusion process is described as follows: Among them, R is the result after feature fusion; Step 4.3: Filter the saliency detection results using image mean filtering technology.

Images