Real-time people stream visualization system based on optical flow method and stereoscopic vision technology

Abstract

The invention discloses a real-time people stream visualization system based on an optical flow method and a stereoscopic vision technology. The system comprises the following functional units: a visual data acquisition unit for collecting visual data from an environment with dense people streams; the optical flow analysis and processing unit is used for carrying out motion state analysis and motion vector calculation on the key feature points of the human body; the stereoscopic vision processing unit is used for accurately predicting a target motion path; the data fusion and analysis unit is used for identifying advanced behaviors of individual movement; and the user interaction and visualization unit is used for displaying the comprehensive data obtained from the data fusion and analysis unit. The problems of insufficient positioning precision, poor real-time performance and limited data visualization capability in the existing people flow monitoring technology are solved. The system is designed to be used in crowded environments such as shopping malls, stations, public activity areas and the like, and aims to provide an accurate, real-time and multi-dimensional crowd monitoring and management tool for managers, operators and common customers.

Description

Technical Field

[0001] The invention belongs to the technical field of visual monitoring, and in particular relates to a real-time crowd flow visualization system based on optical flow method and stereo vision technology. Background Art

[0002] As commercial and public safety needs increase, the need for accurate monitoring and analysis of human traffic is also increasing. Traditional video surveillance systems usually rely only on two-dimensional images and lack the perception of spatial depth, which limits their accuracy and effectiveness in assessing human density and analyzing behavior. Therefore, it is particularly important to develop a system that can provide high-precision three-dimensional human traffic data.

[0003] Existing technologies include "a shopping mall passenger flow prediction system based on big data" and "a passenger flow monitoring method device electronic device and computer-readable storage medium". These technologies use big data and sensor fusion technology to predict and monitor passenger flow in public places such as shopping malls.

[0004] A shopping mall traffic prediction system based on big data: This patent describes a shopping mall traffic prediction system based on big data, which mainly uses historical data and real-time data combined with machine learning technology to predict traffic. The system analyzes past traffic data and predicts future traffic changes to help shopping malls optimize operations and management.

[0005] Crowd flow monitoring method, device, electronic device, and computer-readable storage medium: This patent involves a crowd flow monitoring method and device, which uses a variety of sensor data, such as video surveillance and Wi-Fi signals, to monitor crowd flow and analyze behavior through data fusion technology. The system emphasizes real-time data processing and visualization, providing detailed information on crowd density and behavior patterns.

[0006] Although the above technologies provide effective solutions for people flow monitoring, they still have some obvious limitations:

[0007] Limited adaptability to complex scenarios: Existing technologies, such as “A shopping mall traffic prediction system based on big data”, mainly rely on historical data and generalized models for prediction, which may not be suitable for rapid response to emergencies or non-standard behavior patterns. This limits the accuracy and efficiency of the system in dealing with sudden changes.

[0008] Lack of three-dimensional spatial information: Although the "Crowd Flow Monitoring Method, Device, Electronic Device, and Computer-readable Storage Medium" uses data fusion technology to improve the accuracy of monitoring, it mainly focuses on two-dimensional image data processing and lacks the capture and analysis of three-dimensional depth information of the scene. This may lead to inaccurate monitoring results in multi-layer or complex structure environments.

[0009] Insufficient real-time dynamic tracking capabilities: Existing systems may have limitations in continuously tracking individuals that move quickly or in a crowd. The combination of optical flow and stereo vision, as proposed in the present invention, can more effectively track the dynamic changes of individuals in complex environments and provide more consistent and accurate trajectory data. Summary of the invention

[0010] The present invention aims to provide a real-time crowd visualization system based on optical flow and stereo vision technology to solve the problems of insufficient positioning accuracy, poor real-time performance and limited data visualization capabilities in existing crowd monitoring technologies. The system is designed for crowded environments such as shopping malls, stations, and public activity areas, and is committed to providing managers, operators and ordinary customers with an accurate, real-time and multi-dimensional crowd monitoring and management tool.

[0011] A real-time crowd flow visualization system based on optical flow method and stereo vision technology includes the following functional units:

[0012] Visual data acquisition unit, used to collect high-quality visual data from crowded environments;

[0013] The optical flow analysis and processing unit combines the feature points related to the environment to make adaptive adjustments to the environment, and performs motion state analysis and motion vector calculation of key feature points of the human body;

[0014] The stereo vision processing unit uses the stereo image data captured by the dual camera system and combines it with the feature points related to the environment to pre-process the image and achieve accurate prediction of the target's motion path;

[0015] The data fusion and analysis unit combines the data from the optical flow analysis and processing unit and the stereo vision processing unit to identify the high-level behavior of individual movement;

[0016] The User Interaction and Visualization Unit creates an intuitive and interactive user interface to display the integrated data obtained from the Data Fusion and Analysis Unit.

[0017] The present invention not only solves the limitations of these existing technologies by combining optical flow and stereo vision technology, but also provides a more accurate and adaptable crowd monitoring system. This enables the system to better understand and analyze crowd dynamics in complex scenes, especially in multi-layered and frequently visually blocked environments, thereby greatly improving the safety and operation management capabilities of shopping malls and public areas.

[0018] The present invention has made significant improvements to the key defects in the existing crowd flow monitoring technology. By combining the optical flow method and stereo vision technology, it is expected to bring the following technical effects:

[0019] 1. The accuracy and real-time performance of crowd flow monitoring are significantly improved

[0020] By combining optical flow and stereo vision technology, this system can monitor the real-time dynamics of crowd flow in crowded areas with sub-meter accuracy. This method not only improves the spatial resolution of monitoring, but also accurately captures and analyzes the three-dimensional movement of the crowd, thus providing high-quality data support for crowd management and safety monitoring.

[0021] 2. Improvement of operational management efficiency

[0022] The 3D visualization interface and rich data analysis tools provided by the system enable managers to quickly and accurately understand the overall and local flow of people in crowded areas. For example, through the system's heat map and motion trajectory analysis functions, managers can reasonably allocate security and service resources to cope with peak traffic and improve the service quality and operational efficiency of the area.

[0023] 3. Improvement of public experience and safety

[0024] The system provides enhanced navigation services and congestion warnings for the public, helping people avoid crowded areas and choose the best routes to travel, thereby improving the efficiency and safety of public areas. In addition, real-time crowd flow data can also be used for emergency response, such as rapid evacuation, to improve the effectiveness of emergency management.

[0025] 4. Improvement of safety management capabilities

[0026] By real-time monitoring and analysis of crowd density and movement trends, the system can automatically trigger an alarm when the flow of people exceeds the safety threshold. This not only helps to timely guide the flow of people and prevent crowding accidents, but also effectively prevents potential safety risks.

[0027] 5. Data-driven decision support

[0028] The system's data analysis function provides managers with evidence-based decision support to optimize crowd management strategies and emergency response plans. By analyzing crowd flow data at different times and locations, managers can plan activities and resource allocation more effectively, improving overall management efficiency.

[0029] 6. Optimization of resource allocation

[0030] Accurate crowd monitoring data enables more reasonable resource allocation, such as security, medical support and traffic dispatch, thereby reducing resource waste and improving service efficiency. This is especially important for large-scale events or public transportation systems, which can significantly improve user satisfaction and system operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a structural block diagram of the real-time people flow visualization system of the present invention;

[0032] Figure 2 It is a flow chart of the implementation method of the real-time crowd flow visualization system of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0034] like Figure 1 As shown, a real-time crowd flow visualization system based on optical flow method and stereo vision technology includes the following functional units:

[0035] Visual data acquisition unit, used to collect high-quality visual data from crowded environments;

[0036] The optical flow analysis and processing unit combines the feature points related to the environment to make adaptive adjustments to the environment, and performs motion state analysis and motion vector calculation of key feature points of the human body;

[0037] The stereo vision processing unit uses the stereo image data captured by the dual camera system and combines it with the feature points related to the environment to pre-process the image and achieve accurate prediction of the target's motion path;

[0038] The data fusion and analysis unit combines the data from the optical flow analysis and processing unit and the stereo vision processing unit to identify the high-level behavior of individual movement;

[0039] The User Interaction and Visualization Unit creates an intuitive and interactive user interface to display the integrated data obtained from the Data Fusion and Analysis Unit.

[0040] 1. Visual data acquisition unit

[0041] (1) Functional description

[0042] This unit is responsible for collecting high-quality visual data from crowded environments such as shopping malls, stations, and public areas. A dual-camera system placed at strategic locations is used to capture stereo video data for analyzing crowd dynamics and behavior patterns.

[0043] (2) Technical details

[0044] A high-resolution and high-frame-rate camera is used, and the baseline distance between the two cameras is set to B.

[0045] (3) Feature point detection

[0046] Use deep learning models (such as Mask R-CNN) to automatically detect the key feature points of the human body: head a, left hand b, right foot c, and environment-related feature points: picture brightness level d, shadow e, occlusion f, and noise level g.

[0047] Feature points a, b, and c are selected based on their clarity and representativeness of motion in the image for subsequent optical flow analysis and motion state judgment. Feature point d is used to judge the current lighting environment so that the subsequent optical flow analysis and processing unit and stereoscopic vision processing unit can make adaptive adjustments according to the lighting conditions to ensure the accuracy of motion vector calculation and depth information extraction under different lighting conditions. Feature points e and f are used to identify shadow areas and possible occlusions in the image, respectively, to improve detection accuracy in complex scenes and reduce misjudgments. Feature point g is used to monitor the noise level to ensure that the necessary denoising processing is triggered when the image quality decreases, further optimizing the detection accuracy of feature points and subsequent motion analysis.

[0048] 2. Optical flow analysis and processing unit

[0049] (1) Functional description

[0050] This unit uses the image data provided by the visual data acquisition unit and combines it with the feature points d, e, f, g (light level, shadow, occlusion, noise information) to make adaptive adjustments to the environment. It then performs motion state analysis and motion vector calculation on the key feature points a, b, c (head, left hand, right foot) of the human body.

[0051] (2) Technical details

[0052] ①Calculation and application of illumination level at feature point d

[0053] The feature point d is selected based on the overall brightness distribution of the image, which is used to monitor the lighting conditions in real time and enhance the image when necessary.

[0054] The brightness value of feature point d can be calculated by the following formula:

[0055]

[0056] Among them, L d represents the average brightness of feature point d, N is the number of pixels in the image, and I i is the brightness value of the ith pixel. This value reflects the overall brightness of the current frame.

[0057] Based on the calculated brightness value, the system divides the lighting environment into different levels (such as low brightness, medium brightness, and high brightness) and sets the corresponding thresholds:

[0058] Low brightness: L d <T low

[0059] Medium brightness: T low ≤L d ≤T high

[0060] High brightness: L d >T high

[0061] Among them, T low and T high It is the preset brightness threshold used to judge the current lighting conditions.

[0062] Then, according to the brightness level of feature point d, the image processing method is dynamically adjusted. For example:

[0063] When the brightness level is low, the optical flow analysis and processing unit automatically starts image enhancement processing to improve contrast and remove noise to ensure that motion vectors can still be accurately calculated in poor lighting environments.

[0064] When the brightness level is high, the optical flow analysis and processing unit reduces the degree of contrast enhancement to prevent image overexposure from affecting the accuracy of the motion vector.

[0065] ②Calculation and application of feature point e shadow detection

[0066] The selection of feature points e is based on the brightness and contrast of the pixels in the image, and is used to identify the shadow area in the picture in real time. The presence of shadows may affect the detection of human feature points, so the system will adjust the subsequent image processing and feature point detection based on the information of the shadow area.

[0067] Feature point e identifies the shadow area by analyzing the brightness value changes of each pixel in the image. The method is as follows:

[0068] S e = {I i |I i <T shadow}

[0069] Among them, S e represents the detected shadow area, I i is the brightness value of the i-th pixel in the image, T shadow It is the preset shadow threshold. When the pixel brightness is lower than this threshold, the system determines that the pixel belongs to the shadow area.

[0070] Threshold settings for shadow detection:

[0071] Low shadow threshold: I e ≤T shadow , which is used to determine areas with weak lighting or obvious shadows.

[0072] High Shadow Threshold: I e >T shadow , used to determine the fuzzy shadow edge or transition area.

[0073] The system processes the image based on the detected shadow areas:

[0074] Shadow area marking: When a large area of ​​shadow is detected, the system will mark the area so that the weight of the area can be adjusted in subsequent feature point detection and motion analysis to avoid misjudgment caused by shadows.

[0075] Shadow area compensation: By combining the historical frame information of the image, feature points covered by shadows are compensated to ensure that shadows do not affect the calculation of motion vectors in dynamic scenes.

[0076] When feature point e detects a shadow, the system will work together with feature point d (brightness level) to determine the degree of influence of the shadow in the current lighting environment on feature points a, b, and c.

[0077] If the shadow area is too large and overlaps with the motion feature point, the system will combine the information of the occluded feature point f to optimize the motion analysis of the specific area.

[0078] ③Calculation and application of feature point f occlusion detection

[0079] The selection of feature points f is based on the occlusion of objects or people in the image. Especially in crowded scenes, some feature points (such as head, hands and feet) may be occluded. The purpose of occlusion detection is to identify the partially occluded area and compensate and optimize the feature points in the occluded area to ensure that the system can accurately track the moving target even in the case of occlusion.

[0080] The feature point f identifies the occluded area by analyzing the pixel changes in consecutive frames in the image. The method is as follows:

[0081] O f = {I i |I i Significant changes from historical frames and reduced movement speed}

[0082] Among them, O f Indicates the detected occlusion area, I i is the brightness or depth value of the i-th pixel in the image. By comparing the pixel changes of the previous and next frames, if it is found that the pixel changes in some areas suddenly decrease and the motion speed of the area decreases significantly, the system will determine that there may be occlusion in the area.

[0083] After the system detects an occluded area, it will take the following actions:

[0084] Occlusion marking: For areas where occlusion occurs, the system will mark the area so that appropriate weight adjustments can be made to the area in subsequent optical flow analysis and motion vector calculation to avoid misjudgment due to occlusion.

[0085] Historical frame compensation: The system compensates for obscured feature points by combining historical frame information. By interpolating the image information of the previous frames, the system can predict the possible positions of the obscured feature points to ensure that the target's motion trajectory remains consistent.

[0086] According to the degree of occlusion, the feature point f divides the occlusion into the following levels:

[0087] Partial occlusion: When only a portion of a feature point is occluded (such as a hand or foot), the system will perform motion prediction based on the unoccluded portion of information.

[0088] Complete occlusion: When key feature points (such as the head and all limbs) are completely blocked, the system will start the historical frame compensation mechanism to predict the position of the blocked feature points in the current frame by analyzing the motion trajectory of the historical frames.

[0089] If the system detects that feature points a (head), b (left hand), or c (right foot) are blocked, the system will prioritize motion compensation based on historical frame data, and use surrounding unobstructed feature points for auxiliary analysis.

[0090] Occlusion detection can also be combined with shadow detection to determine whether occlusion is caused by shadows. If occlusion and shadows exist at the same time, the system will prioritize the shadow impact to ensure that there is no misjudgment.

[0091] ④Calculation and application of noise level of feature point g

[0092] The feature point g is used to detect noise in the image, especially in low light environments or high ISO settings, where noise may interfere with feature point detection and motion vector calculation. The purpose of noise detection is to determine image quality and, if necessary, denoise the image to ensure that the system can still accurately analyze the motion and position of feature points in noisy scenes.

[0093] The feature point g identifies the noise level of the image by analyzing the random changes of pixels in the image. The method is as follows:

[0094]

[0095] Where Ng represents the noise level, N is the number of pixels in the image, Ii is the brightness value of the i-th pixel in the image, It is the smoothed value of the pixel (usually obtained by Gaussian filtering or mean filtering), which reflects the noise intensity in the current image.

[0096] According to the calculated noise value, the feature point g divides the noise level into different levels (such as low noise, medium noise and high noise) and sets the corresponding threshold:

[0097] Low noise: N g <T low , indicating that the image quality is high and no denoising is required.

[0098] Medium noise: T low ≤N g ≤T high , indicating that there is a certain degree of noise in the image and slight denoising can be performed.

[0099] High noise: N g >T high , indicating that the image quality is poor and stronger denoising processing is required.

[0100] When the system detects that the noise level exceeds the preset threshold, the denoising algorithm will be triggered to process the image. Low to medium noise levels are processed using Gaussian filtering, and high noise levels are processed using median filtering.

[0101] The noise level directly affects the detection accuracy of feature points a, b, and c. If a high noise level is detected, the system will prioritize denoising the image to ensure the clarity of the key feature points of the human body, thereby ensuring the accuracy of motion vector calculation.

[0102] The noise level is also closely related to the illumination feature point d. Low-light environments are often accompanied by higher noise. By combining the level information of the illumination feature point d, the system can dynamically adjust the exposure parameters to reduce the generation of noise and perform denoising at the same time.

[0103] ⑤Optical flow calculation of feature points a, b, c

[0104] Analyze the position change of feature point a between consecutive frames and calculate the motion vector:

[0105]

[0106] where v x Represents the velocity component of the object in the horizontal direction of the image (usually the x-axis direction). Positive values ​​usually indicate movement to the right, and negative values ​​indicate movement to the left. y Represents the velocity component of the object in the vertical direction of the image (usually the y-axis direction). In the image coordinate system, a positive value usually means moving downward (because the origin of the image coordinates is in the upper left corner and the y-axis increases downward), and a negative value means moving upward.

[0107] and

[0108]

[0109] Therefore, the motion vector of feature point a is:

[0110]

[0111] in, is the coordinate of feature point a at time t, is the coordinate of feature point a at time t+1, with a time interval of Δt. The calculation method of feature point b and feature point c is the same as feature point a.

[0112] The motion vector of feature point b is:

[0113]

[0114] The motion vector of feature point c is:

[0115]

[0116] Movement status definition:

[0117] Walking state (M): If the velocity of a feature point is below a certain threshold (e.g., 0.5 meters per second) and the motion vector shows a small direction change, it is defined as a walking state.

[0118] Running state (N): If the speed of a feature point exceeds a high threshold (e.g., 1 meter per second) and is accompanied by significant direction changes, it is defined as a running state.

[0119] Stationary state (S): If the velocity of a feature point is close to zero and there is no obvious change in direction, it is defined as a stationary state.

[0120] State transition definition: define motion state is the maximum absolute value of the motion state of the three feature points a, b, and c. When the value reaches a certain threshold, the motion state is switched.

[0121]

[0122] State M (walking):

[0123] State N (Running):

[0124] State S (prohibited):

[0125] (3) Subsequent analysis

[0126] Based on the motion state (M: walking, N: running, S: stationary) obtained by the optical flow calculation of feature points a, b, c, these state data will be used in the data fusion and analysis unit, combined with the lighting, shadow, occlusion and noise information provided by feature points d, e, f, g, to adaptively adjust the environment. Subsequently, by combining stereoscopic vision data, the system can further accurately predict future action paths or conduct complex crowd behavior analysis.

[0127] 3. Stereo Vision Processing Unit

[0128] (1) Functional description

[0129] The stereo vision processing unit uses the stereo image data captured by the dual camera system and pre-processes the image in combination with the feature points d (light level), e (shadow information), f (occlusion information) and g (noise information) to ensure the accuracy of depth information calculation in complex environments. The system calculates the three-dimensional spatial position of objects (such as human feature points) to accurately predict the target's motion path.

[0130] (2) Technical details

[0131] ① Adaptive lighting adjustment

[0132] The image captured by the dual cameras is preprocessed using the brightness information of the feature point d. If insufficient or excessive lighting is detected, the system will optimize image clarity by adjusting exposure parameters and contrast, thereby improving stereo matching accuracy.

[0133] ②Processing of shadow areas

[0134] Based on the detection results of feature points e, the system will identify shadow areas in the image. In these areas, stereo matching may be affected, so the system will mark these areas and reduce the reliance on these areas during the matching process to prevent errors caused by shadow areas.

[0135] ③Occlusion processing

[0136] Combined with the feature point f, the detected occlusion information will be used to adjust the stereo matching algorithm, especially in multi-layer or multi-crowd scenes, where occlusion may cause some feature points to fail to be detected normally. The system will make compensation predictions based on historical frame data to ensure that the tracking of feature points can be maintained even in the case of occlusion.

[0137] ④Noise processing

[0138] Using the noise information of the feature point g, the system can determine the noise level of the current image. If there is too much noise, the system will first perform denoising on the image (such as Gaussian filtering or median filtering) to ensure that the accuracy of the image during the stereo matching process is not affected by noise.

[0139] ⑤Stereo matching technology

[0140] The semi-global matching (SGM) algorithm is used to process the image pairs collected from the dual cameras and calculate the disparity map. The disparity map represents the pixel displacement of the same feature point between images and is a key indicator of depth information.

[0141] The calculation formula for parallax is:

[0142] d = x L -x R

[0143] Where d is the disparity, x L and x R They are the horizontal coordinates of the same feature point in the left and right cameras respectively.

[0144] ⑥ Depth calculation

[0145] The depth Z of the feature point is calculated based on the disparity value and the geometric parameters of the camera. The formula is:

[0146]

[0147] Where f is the focal length of the camera and B is the baseline distance between the two cameras.

[0148] ⑦ 3D coordinate calculation

[0149] Using the depth value and the pixel position in the image, the coordinates (X, Y, Z) of the feature points in three-dimensional space are calculated to ensure that the spatial position information of the feature points can be accurately extracted even in complex environments (such as low light, shadows, noise, and occlusion):

[0150]

[0151] Among them, x and y are the pixel coordinates of the feature point in the image, c x and c y are the pixel coordinates of the optical center of the camera.

[0152] (3) Adaptive adjustment and optimization

[0153] The system dynamically adjusts the parameters of the stereo vision processing unit based on the information of environmental feature points (d, e, f, g) to ensure high-precision three-dimensional reconstruction in different environments.

[0154] If extreme conditions (such as high noise, strong occlusion) are detected by environmental feature points, the system will trigger the historical frame compensation algorithm to reduce the negative impact of these factors on the calculation of depth information.

[0155] 4. Data fusion and analysis unit

[0156] (1) Functional description

[0157] Main function: Combines optical flow analysis and data from stereo vision processing units to achieve high-level behavior recognition of individual movements, such as chasing and escaping.

[0158] (2) Behavior Analysis Algorithm

[0159] Speed ​​calculation:

[0160] For each pair of individuals i and j, the motion vectors are and

[0161] The individual speed is and

[0162] The three-dimensional coordinates of individual i (X i , Y i , Z i ), the three-dimensional coordinates of individual j (X j , Y j , Z j )

[0163] The distance between individual i and individual j

[0164] Behavior Recognition:

[0165] Chasing: If the speed of an individual i exceeds the running threshold, and there is another individual j whose distance from i is gradually decreasing, and j also has a high speed, then i is considered to be chasing j.

[0166] Escape: If the speed of the pursued individual j also exceeds the running threshold and tries to increase the distance from the pursuer i, j is considered to be escaping.

[0167] (3) Other advanced behavior identification

[0168] Identify and respond to a variety of scenarios. By clearly describing the implementation of these algorithms, you can emphasize the originality and practicality of your technology, thereby enhancing the competitiveness of your patent application.

[0169] ① Detection algorithm of aggregation behavior

[0170] Core idea: Use density clustering algorithm to identify groups of individuals gathered in a smaller space.

[0171] Algorithm steps:

[0172] Input data: 3D coordinates of each individual (X i , Y i , Z i )、(X j , Yj , Z j )etc.

[0173] Density Clustering: Use the DBSCAN algorithm to cluster the location data.

[0174] Aggregation determination: Identify cluster areas with density greater than a preset threshold and mark them as aggregation behaviors.

[0175] ②Fall detection algorithm

[0176] Core idea: Analyze individual height changes and motion vector mutations to detect fall events.

[0177] Algorithm steps:

[0178] Input data: 3D coordinates and motion vectors of individuals.

[0179] Height Monitoring: Calculate the height change of an individual in consecutive frames.

[0180] Speed ​​change analysis: Detects sudden decreases in the speed of movement.

[0181] Fall determination: If the height drops rapidly and the movement speed decreases afterwards, it is determined to be a fall.

[0182] ③ Queuing behavior detection algorithm

[0183] Core idea: Analyze the time and spatial arrangement of people staying in a specific place.

[0184] Algorithm steps:

[0185] Input data: 3D coordinates and timestamp of the individual.

[0186] Trajectory Analysis: Use trajectory analysis algorithms, such as trajectory clustering, to identify dwell and movement patterns.

[0187] Queuing determination: If multiple individuals maintain a linear arrangement at a specific location and move at a low speed, it is determined to be queuing behavior.

[0188] ④Interaction behavior detection algorithm

[0189] Core idea: Analyze the relative positions and postures between individuals.

[0190] Algorithm steps:

[0191] Input data: 3D coordinates and posture information of each individual.

[0192] Position and orientation analysis: Calculate the relative distance and orientation angle between individuals.

[0193] Interaction determination: If two or more individuals are facing each other and close to each other, and the duration exceeds the threshold, it is determined to be an interactive behavior.

[0194] (4) Environmental Adaptation and Optimization

[0195] In complex environments (such as lighting changes, shadow occlusion, noise interference, etc.), the system dynamically adjusts the parameters of the data fusion and analysis unit by combining the lighting, shadow, occlusion and noise information of the feature points d, e, f, g to ensure the accuracy of behavior recognition and three-dimensional motion analysis.

[0196] ① Adaptive illumination processing

[0197] When feature point d detects a change in light level, the system will automatically adjust image processing parameters (such as exposure and contrast) according to the ambient lighting conditions to ensure that feature points can be accurately identified and motion vectors can be calculated in low-light or strong-light environments.

[0198] ②Shadow compensation processing

[0199] When feature points e detect shadow areas, the system will mark these areas and reduce the weight of these areas in optical flow and stereo vision analysis to avoid misjudgment caused by shadows. At the same time, motion compensation is performed on feature points covered by shadows in combination with historical frame information.

[0200] ③Occlusion processing

[0201] When occlusion is detected at a feature point f, the system uses a prediction mechanism based on historical frames to ensure that the motion information of the feature points in the occluded area can remain consistent through interpolation or prediction, thus avoiding interruption of behavior recognition due to occlusion.

[0202] ④Noise processing

[0203] When the feature point g detects that the image noise level is too high, the system will automatically enable the denoising algorithm (such as Gaussian filtering, median filtering, etc.) to improve the quality of the image and ensure that the accuracy of the optical flow and stereo vision analysis is not affected by noise.

[0204] 5. User interaction and visualization unit

[0205] (1) Functional description

[0206] Main function: This unit is responsible for creating an intuitive and interactive user interface to display the comprehensive data obtained from the data fusion and analysis unit. This includes crowd distribution maps, behavior pattern analysis and real-time surveillance video.

[0207] Application scenarios: Suitable for environments that require real-time monitoring and data analysis, such as security monitoring centers in shopping malls, stations and public areas, or urban management and operation centers.

[0208] (2) Technical details

[0209] Real-time data visualization:

[0210] Implement advanced data visualization techniques, such as dynamic heat maps, 3D modeling, and timeline views, to provide rich visual presentations and help users intuitively understand complex data.

[0211] Use WebGL or similar technologies to achieve high-performance 3D rendering and animation to support real-time processing and display of large-scale data.

[0212] User Interface Design:

[0213] Develop easy-to-use graphical user interfaces (GUIs), including dashboards, navigation bars, and toolbars, to support users’ different operational needs.

[0214] Design responsive interfaces to adapt to different devices and screen sizes to ensure that users have a good operating experience on a variety of devices.

[0215] (3) Data interaction and feedback

[0216] Interactive features:

[0217] Realize multi-touch and voice control functions to improve the convenience and interactivity of user operations.

[0218] Integrated advanced search and data filtering tools allow users to quickly find the information they need based on time, location or specific events.

[0219] Feedback and Adjustment:

[0220] Provide real-time feedback mechanisms, such as color changes, animations, or pop-ups to alert users to important events or status changes.

[0221] Allows users to customize interface and data display settings, including selecting different visual themes and layout options, to suit personal preferences and specific needs.

[0222] (4) Output and application

[0223] Visualization output:

[0224] Generate various forms of visual outputs, such as real-time video streams, statistical charts, and interactive maps, providing users with intuitive data understanding and analysis tools.

[0225] Output device compatibility, ensuring visualizations look good on desktop, tablet, and mobile devices.

[0226] Data output interface:

[0227] Providing an API or web service so that visualization results can be accessed and utilized by other systems or applications, such as emergency response systems or decision support systems.

[0228] Supports data export function, allowing users to export data reports or charts to PDF, Excel and other formats for further analysis or archiving. Control and regular optimization.

[0229] Figure 1 The core components of the present invention and their interconnections are shown. As the starting point of data input, the visual data acquisition unit is responsible for collecting raw visual data from crowded areas through a dual-camera system and sending them to the optical flow analysis and processing unit and the stereo vision processing unit respectively; then the optical flow analysis and processing unit executes the optical flow algorithm to analyze the movement of the object and outputs motion vector data; the stereo vision processing unit performs stereo matching and depth information processing and outputs three-dimensional space data; the data fusion and analysis unit is responsible for receiving the output data of the optical flow analysis and processing unit and the stereo vision processing unit, performing data fusion processing, integrating all relevant information, providing comprehensive data analysis results, and sending them to the user interaction and visualization unit.

[0230] The specific objectives of the present invention include:

[0231] 1. Venue managers (including shopping malls, stations and activity area management)

[0232] ◆ Understand the flow of people in key areas in real time, optimize space layout and improve operational efficiency.

[0233] ◆Quickly identify and manage congested areas or potential safety hazards through a real-time 3D visualization interface, and respond in a timely manner to ensure safety and efficiency.

[0234] 2. Shop and service point operators

[0235] ◆Use the real-time traffic data provided to understand the surrounding traffic density and changing trends, so as to optimize service layout and promotion strategies, and enhance customer appeal and sales performance.

[0236] ◆Accurate traffic data supports the formulation of effective marketing and customer service strategies, improving customer satisfaction and loyalty.

[0237] 3. Ordinary customers and passers-by

[0238] ◆Use real-time crowd flow information to optimize walking routes and time, avoid high-density areas, and improve the travel experience.

[0239] ◆Receive personalized recommendations, such as service points and rest areas with less traffic, to improve overall experience and satisfaction.

[0240] By combining the optical flow method and stereo vision technology, the present invention not only significantly improves the accuracy and real-time performance of crowd monitoring, but also expands the applicability of the system, enabling it to operate efficiently in various crowded environments and meet a wide range of monitoring and management needs.

[0241] like Figure 2 As shown, a method for implementing a real-time crowd flow visualization system based on optical flow method and stereo vision technology is provided.

[0242] Static relationship:

[0243] Start / End (oval): Indicates the start and end points of a process, and is the entry and exit of system operations. For example, "data collection".

[0244] Processing steps (rectangles): represent operations or tasks performed in the system, such as "preliminary processing" or "optical flow analysis".

[0245] Decision (diamond): Indicates that the system selects different subsequent operations according to different situations, such as "whether a valid motion vector is detected".

[0246] Dynamic Relationships:

[0247] Arrow (straight line): Indicates the direction of system flow, pointing from one step to the next step or decision point.

[0248] Effect:

[0249] 1. Visual data collection:

[0250] The visual data acquisition unit starts capturing video data. It continuously captures images and uses a high-precision camera system to collect visual information of crowded areas in real time.

[0251] 2. Preliminary processing:

[0252] Pre-processing of captured images, such as rectification, denoising, and enhancement, prepares the data for analysis algorithms.

[0253] 3. Parallel Processing

[0254] Optical flow analysis: collects processed images from the visual data acquisition unit, applies the optical flow algorithm to calculate motion vectors, determines the continuity and direction of motion, and outputs motion vector data.

[0255] Stereo vision analysis: collects processed images from the visual data acquisition unit, performs stereo matching, calculates depth information, generates three-dimensional space coordinates, and outputs depth maps and three-dimensional coordinate data.

[0256] 4. Data fusion analysis

[0257] Process the output data of the optical flow analysis and processing unit and the stereo vision processing unit, fuse the two data sources, conduct comprehensive analysis, and generate a comprehensive analysis report, such as crowd density, movement trend, etc.

[0258] 5. User Interaction and Visualization

[0259] Process comprehensive reports from the data fusion and analysis unit, customize the visualization interface according to user needs, and display dynamic heat maps, 3D views and real-time data. Users make decisions based on the visualization data, such as resource allocation, safety management, etc.

Claims

1. A real-time crowd visualization system based on optical flow and stereo vision technology, characterized by It includes the following functional units: Visual data acquisition unit, used to collect high-quality visual data from crowded environments; The optical flow analysis and processing unit combines the feature points related to the environment to make adaptive adjustments to the environment, and performs motion state analysis and motion vector calculation of key feature points of the human body; The stereo vision processing unit uses the stereo image data captured by the dual camera system and combines it with the feature points related to the environment to pre-process the image and achieve accurate prediction of the target's motion path; The data fusion and analysis unit combines the data from the optical flow analysis and processing unit and the stereo vision processing unit to identify the high-level behavior of individual movement; The User Interaction and Visualization Unit creates an intuitive and interactive user interface to display the integrated data obtained from the Data Fusion and Analysis Unit.

2. The real-time crowd flow visualization system based on optical flow method and stereo vision technology according to claim 1 is characterized in that The visual data acquisition unit uses a dual camera system to capture stereoscopic video data for analyzing crowd flow dynamics and behavior patterns; the baseline distance between the dual cameras is set to B; Use deep learning models (such as Mask R-CNN) to automatically detect key feature points of the human body: head a, left hand b, right foot c, and environment-related feature points: picture brightness level d, shadow e, occlusion f, noise level g; The selection of key feature points is based on their clarity and representativeness of motion in the image, and is used for subsequent optical flow analysis and motion state judgment; the image brightness level d is used to judge the current lighting environment, so that the subsequent optical flow analysis and processing unit and stereoscopic vision processing unit can make adaptive adjustments according to the lighting conditions to ensure the accuracy of motion vector calculation and depth information extraction under different lighting conditions. Shadow e and occlusion f are used to identify shadow areas and possible occlusion situations in the image, respectively, so as to improve the accuracy of detection in complex scenes and reduce misjudgment; The noise level g is used to monitor the noise level to ensure that the necessary denoising process is triggered when the image quality degrades, further optimizing the detection accuracy of feature points and subsequent motion analysis.

3. The real-time crowd flow visualization system based on optical flow method and stereo vision technology according to claim 2 is characterized in that The above optical flow analysis and processing unit performs adaptive adjustment of the environment in combination with the feature points related to the environment, and performs motion state analysis and motion vector calculation of the key feature points of the human body; the specific process is as follows: (2.1) Picture brightness level d Light level calculation: The selection of the picture brightness level d is based on the overall brightness distribution of the image, which is used to monitor the lighting conditions in real time and enhance the image; The brightness value of the picture brightness level d is calculated by the following formula: Among them, L d represents the average brightness of the picture brightness level d, N is the number of pixels in the image, I i is the brightness value of the i-th pixel, which reflects the overall brightness of the current frame; According to the calculated brightness value, the lighting environment is divided into different levels of low brightness, medium brightness and high brightness, and the corresponding thresholds are set: Low brightness: L d <T low ; Medium brightness: T low ≤L d ≤T high ; High brightness: L d >T high ; Among them, T low and T high is the preset brightness threshold used to determine the current lighting conditions; Then, the image processing method is dynamically adjusted according to the brightness level of the picture brightness level d; (2.2) Detection of shadow e: The selection of shadow e is based on the brightness and contrast of pixels in the image, and is used to identify the shadow area appearing in the picture in real time; Shadow e identifies the shadow area by analyzing the changes in the brightness value of each pixel in the image; the specific process is: S e ={I i |I i <T shadow } Among them, S e Indicates the detected shadow area, I i is the brightness value of the i-th pixel in the image, T shadow is the preset shadow threshold; when the pixel brightness is lower than this threshold, the system determines that the pixel belongs to the shadow area; Threshold settings for shadow detection: Low shadow threshold: I e ≤T shadow , used to determine areas with weak lighting or obvious shadows; High Shadow Threshold: I e >T shadow , used to determine the fuzzy edge or transition area of ​​the shadow; The system processes the image based on the detected shadow areas: Shadow area marking: When a large area of ​​shadow is detected, the system will mark the area so that the weight of the area can be adjusted in subsequent feature point detection and motion analysis to avoid misjudgment caused by shadows. Shadow area compensation: By combining the historical frame information of the image, feature points covered by shadows are compensated to ensure that shadows do not affect the calculation of motion vectors in dynamic scenes. When a shadow is detected, the system will work with the image brightness level d to determine the degree of impact of the shadow in the current lighting environment on the key feature points; if the shadow area is too large and overlaps with the motion feature points, the system will combine the information of the occlusion f to optimize the motion analysis of the specific area; (2.3) Detection of occlusion f: The selection of occlusion f is based on the occlusion of objects or people in the image, especially in crowded scenes; the purpose of occlusion detection is to identify the partially occluded area and compensate and optimize the feature points in the occluded area to ensure that the system can accurately track the moving target even in the case of occlusion; The occlusion f identifies the occlusion area by analyzing the pixel changes of consecutive frames in the image. The specific process is as follows: O f = {I i |I i Significant changes from historical frames and reduced movement speed} Among them, O f Indicates the detected occlusion area, I i is the brightness or depth value of the i-th pixel in the image. By comparing the pixel changes of the previous and next frames, if it is found that the pixel changes in some areas suddenly decrease and the motion speed of the area decreases significantly, the system will determine that there may be occlusion in the area; After the system detects an occluded area, it will take the following actions: Occlusion marking: For areas where occlusion occurs, the system will mark the area so that appropriate weight adjustment can be made to the area in subsequent optical flow analysis and motion vector calculation to avoid misjudgment caused by occlusion; Historical frame compensation: The system compensates for obscured feature points by combining historical frame information; by interpolating image information from previous frames, the system can predict the possible locations of obscured feature points to ensure that the target's motion trajectory remains consistent; According to the degree of occlusion, the feature point f divides the occlusion into the following levels: Partial occlusion: When only a part of the feature points are occluded, the system will perform motion prediction based on the unoccluded part of the information; Complete occlusion: When a key feature point is completely occluded, the system will start the historical frame compensation mechanism to predict the position of the occluded feature point in the current frame by analyzing the motion trajectory of the historical frame; If the system detects that a key feature point is blocked, it will first combine historical frame data for motion compensation and use surrounding unblocked feature points for auxiliary analysis; Occlusion detection is combined with shadow detection to determine whether occlusion is caused by shadows. If occlusion and shadows exist at the same time, the system will prioritize the shadow impact to ensure that there is no misjudgment. (2.4) Calculation of noise g level: Noise g is used to detect noise in the image, especially in low light environments or high ISO settings, where noise may interfere with feature point detection and motion vector calculation; Noise g identifies the noise level of an image by analyzing the random changes of pixels in the image. The specific process is as follows: Where Ng represents the noise level, N is the number of pixels in the image, Ii is the brightness value of the i-th pixel in the image, is the smoothing value of the pixel, which reflects the noise intensity in the current image; According to the calculated noise value, the feature point g divides the noise level into different levels of low noise, medium noise and high noise, and sets the corresponding threshold: Low noise: N g <T low , indicating that the image quality is high and no denoising is required; Medium noise: T low ≤N g ≤T high , indicating that there is a certain degree of noise in the image, and a slight denoising process is performed; High noise: N g >T high , indicating that the image quality is poor and requires stronger denoising; When the system detects that the noise level exceeds the preset threshold, it will trigger the denoising algorithm to process the image; Gaussian filtering is used for low to medium noise levels; median filtering is used for high noise levels; The noise level directly affects the detection accuracy of key feature points. If a high noise level is detected, the system will prioritize denoising the image to ensure the clarity of the key feature points of the human body, thereby ensuring the accuracy of motion vector calculation; Based on the information of noise level and picture brightness level, the exposure parameters are adjusted dynamically to reduce the noise generation and perform denoising at the same time.

4. The real-time crowd flow visualization system based on optical flow method and stereo vision technology according to claim 3 is characterized in that The calculation process of key feature points is: Analyze the position change of head a between consecutive frames and calculate the motion vector: where v x Represents the velocity component of the object in the horizontal direction of the image; positive values ​​usually indicate movement to the right, and negative values ​​indicate movement to the left; v y Represents the velocity component of the object in the vertical direction of the image; in the image coordinate system, a positive value usually indicates downward movement, and a negative value indicates upward movement; and Therefore, the motion vector of head a is: in, is the coordinate of head a at time t, is the coordinate of head a at time t+1, with a time interval of Δt; The motion vector of the left hand b is: The motion vector of the right foot c is: Movement status definition: Walking state (M): If the velocity of the key feature point is lower than the threshold and the motion vector shows a small direction change, it is defined as a walking state; Running state (N): If the speed of the key feature point exceeds the high threshold and is accompanied by a significant change in direction, it is defined as a running state; Stationary state (S): If the velocity of the key feature point is close to zero and there is no obvious change in direction, it is defined as a stationary state; State transition definition: define motion states It is the maximum absolute value of the motion state of the key feature points; when the value reaches a certain threshold, the motion state is switched. State M (walking): State N (Running): State S (prohibited):

5. The real-time crowd flow visualization system based on optical flow method and stereo vision technology according to claim 4 is characterized in that The stereo vision processing unit uses the stereo image data captured by the dual camera system and combines the picture brightness level d, shadow e, occlusion f and noise g to pre-process the image to achieve accurate prediction of the target motion path; the specific process is: (4.1) Adaptive lighting adjustment Use the brightness information of the picture brightness level d to pre-process the images captured by the dual cameras; if insufficient or excessive light is detected, adjust the exposure parameters and contrast to optimize the image clarity and improve the stereo matching accuracy; (4.2) Processing of shadow areas Based on the detection results of shadow e, the shadow area in the image is identified; the shadow area is marked to reduce the dependence on the shadow area during the matching process to prevent errors caused by the shadow area; (4.3) Occlusion Processing Combined with the occlusion f, the detected occlusion information will be used to adjust the stereo matching algorithm, especially in multi-layer or multi-crowd scenes; compensation prediction is performed through historical frame data to ensure that the tracking of feature points can be maintained even in the case of occlusion. (4.4) Noise processing Use the noise information of noise g to determine the noise level of the current image; If there is too much noise, the system will first denoise the image to ensure that the accuracy of the image is not affected by noise during the stereo matching process; (4.5) Stereo matching technology Use a semi-global matching algorithm to process the image pairs collected from the dual cameras and calculate the disparity map; The calculation formula for parallax is: d=x L -x R Where d is the disparity, x L and x R are the horizontal coordinates of the same feature point in the left and right cameras respectively; (4.6) Depth calculation The depth Z of the feature point is calculated based on the disparity value and the geometric parameters of the camera. The formula is: Where f is the focal length of the camera, and B is the baseline distance between the two cameras; (4.7) Three-dimensional coordinate calculation Using the depth value and the pixel position in the image, the coordinates (X, Y, Z) of the feature points in three-dimensional space are calculated to ensure that the spatial position information of the feature points can be accurately extracted even in complex environments: Among them, x and y are the pixel coordinates of the feature point in the image, c x and c y is the pixel coordinate of the optical center of the camera; According to the information of picture brightness level d, shadow e, occlusion f, and noise level g, the various parameters of the stereoscopic vision processing unit are dynamically adjusted to ensure high-precision three-dimensional reconstruction in different environments; if extreme conditions are detected in environmental feature points, the historical frame compensation algorithm will be triggered to reduce the negative impact on depth information calculation.

6. The real-time crowd flow visualization system based on optical flow method and stereo vision technology according to claim 5 is characterized in that The above data fusion and analysis unit, combined with the data from the optical flow analysis and processing unit and the stereoscopic vision processing unit, identifies the high-level behavior of individual movement; the specific process is as follows: Speed ​​calculation: For each pair of individuals i and j, the motion vectors are and The individual speed is and The three-dimensional coordinates of individual i (X i , Y i , Z i ), the three-dimensional coordinates of individual j (X j , Y j , Z j ) The distance between individual i and individual j Behavior Recognition: Chasing: If the speed of an individual i exceeds the running threshold, and there is another individual j whose distance from i is gradually decreasing, and j also has a high speed, then i is considered to be chasing j. Escape: If the speed of the pursued individual j also exceeds the running threshold and tries to increase the distance from the pursuer i, j is considered to be escaping.

Images