3D binocular person attribute recognition and counting method and system integrating AI and REID

Abstract

The present application relates to the technical field of data processing, and in particular to a 3D binocular personnel attribute recognition counting method and system integrating AI and REID, which comprises an image acquisition and personnel detection module, a feature extraction module, a feature enhancement module, a motion state matching module, a deduplication module and a correction module. The present application solves the problems of low recognition accuracy and easy misjudgment in the prior art under conditions such as similar personnel features and motion mutation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a 3D binocular person attribute recognition and counting method and system that integrates AI and REID. Background Technology

[0002] Passenger flow statistics serve as a crucial basis for business decision-making, operational management, and safety control. Their accuracy directly impacts the management efficiency and service quality of venues such as shopping malls, transportation hubs, scenic spots, and industrial parks.

[0003] In existing technologies, the counting methods for identifying and counting passengers mainly include the following aspects:

[0004] Statistical methods based on pedestrian re-identification: Pedestrian re-identification technology extracts appearance information such as clothing color, body shape, and posture features of pedestrians, and uses feature similarity matching to determine whether they are the same person. When multiple people wear the same clothing, such as school uniforms or work clothes, or have similar body shapes, their REID features are highly similar, making it difficult for traditional similarity matching methods to distinguish between different people and easily leading to misjudgments.

[0005] Tracking methods based on motion features: To compensate for the shortcomings of apparent features, some researchers have introduced three-dimensional spatial features such as walking speed, acceleration, and motion trajectory as auxiliary identification criteria. However, these methods assume that the person's motion state is relatively stable. Once the person's motion speed changes abruptly, such as from walking slowly to running, traditional matching algorithms based on speed, acceleration, or trajectory will produce serious misjudgments due to the failure of the prediction model, resulting in identity loss and duplicate counting.

[0006] Trajectory-based deduplication methods: Existing deduplication methods mostly rely on identity identifiers and crossing directions for turnaround judgments. For example, the pedestrian flow counting method disclosed in authorized patent CN112668525B uses area division and trajectory regular expressions for entry and exit judgments. However, this method is prone to counting errors due to identity switching, occlusion, etc. Relying solely on identity identifiers and direction judgments can easily misjudge different people's entry and exit as the same person's turnaround, leading to counting errors.

[0007] In summary, existing personnel identification technologies suffer from problems such as low accuracy and susceptibility to misjudgment when personnel have similar characteristics or undergo sudden changes in movement. Summary of the Invention

[0008] The purpose of this application is to provide a 3D binocular person attribute recognition and counting method and system that integrates AI and REID, which solves the problem of low recognition accuracy and easy misjudgment in the existing technology when people have similar characteristics or sudden changes in movement.

[0009] To solve the above-mentioned technical problems, the solution adopted in this application is as follows:

[0010] A 3D binocular person attribute recognition and counting method integrating AI and REID, characterized by the following steps:

[0011] S1: Acquire binocular images and construct a three-dimensional space based on the acquired binocular images. Detect personnel targets in the constructed three-dimensional space and obtain the three-dimensional spatial position and motion parameters of each personnel target.

[0012] S2: Construct a multi-task AI feature extraction network and use the multi-task AI feature extraction network to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image;

[0013] S3: Assign an orthogonal vector to the personnel target that is orthogonal to the historical orthogonal vector. Use the orthogonal vector to fuse and transform the basic REID feature vector to obtain the orthogonalized basic REID feature vector. Fuse the orthogonalized basic REID feature vector and the three-dimensional space feature vector to obtain the identity feature vector.

[0014] S4: Model the motion of personnel targets as a linear system of state vectors and control variables. Optimize the linear system by rolling to predict the predicted position of the personnel target in the next frame. Based on feature similarity, trajectory similarity, and position similarity, perform identity association matching between the predicted position and active trajectories in the historical trajectory database. If the predicted position matches an active trajectory in the historical trajectory database, count the matches and associate the active trajectory with its identity identifier. Simultaneously, detect and record the crossing events of each personnel target; otherwise, do not count the matches.

[0015] S5: Perform deduplication on the crossing events to obtain the deduplicated personnel target count result;

[0016] S6: Based on the deduplicated personnel target count results, the monitoring area is gridded, and its density field is estimated using the depth map of the monitoring area. The velocity field is solved by the density field of continuous frames. Based on the density field and velocity field, the macro passenger flow and micro passenger flow are calculated, and a filter is established to correct the micro passenger flow micro counting deviation, so as to obtain the final personnel target count results.

[0017] Preferably, the specific implementation of S1 includes the following steps:

[0018] Simultaneously acquire left and right eye images, and perform distortion correction and epipolar correction on the acquired left and right eye images to align the rows of the left and right eye images;

[0019] Stereo matching is performed on the corrected left and right images, and a semi-global algorithm is used to calculate the disparity value of each pixel in each image to obtain a disparity map.

[0020] By using the principle of triangulation, the disparity map is converted into a depth map, and the three-dimensional spatial coordinates of each pixel in the monitoring area are obtained.

[0021] A virtual detection line is defined in three-dimensional space, and the virtual detection line is located at the entrance and / or exit of the monitoring area;

[0022] A lightweight object detection network is used to detect people in the left eye image, and a two-dimensional bounding box corresponding to each person target is obtained. The minimum three-dimensional bounding box of each person target is calculated based on the two-dimensional bounding box.

[0023] Traverse the 3D coordinates of all pixels within the smallest 3D bounding box, and calculate the height, width, and thickness of each person target based on the 3D coordinates;

[0024] Based on the coordinates of the three-dimensional center point of several consecutive depth maps, the motion parameters of the human target are calculated, including motion velocity and acceleration.

[0025] Preferably, the specific implementation method of S2 includes the following steps:

[0026] Construct a multi-task AI feature extraction network that includes a backbone network and a 3D spatial feature extraction branch;

[0027] The left eye image is input into the backbone network to extract the basic REID feature vector;

[0028] The height, width, thickness, and movement speed of the target person are input into the three-dimensional spatial feature extraction branch to extract the three-dimensional spatial feature vector, thus obtaining the three-dimensional spatial feature vector.

[0029] Preferably, the specific implementation method of S3 includes the following steps:

[0030] Assign an orthogonal vector to each newly appearing person target in the detection area;

[0031] Map the base REID feature vectors to the same dimension as the orthogonalized vectors;

[0032] The orthogonalized vectors of the same dimension and the basic REID feature vectors are multiplied element-wise to obtain the orthogonalized REID feature vectors;

[0033] The orthogonalized REID feature vector is concatenated with the 3D spatial feature vector to obtain the final identity feature vector.

[0034] Preferably, the step of modeling the motion of the person target as a linear system of state vectors and control variables, and predicting the predicted position of the person target in the next frame by optimizing the linear system through rolling, includes the following steps:

[0035] The motion state of each person target is modeled as a state vector. With control variables A linear system is used to deduce the predicted motion position of the personnel target in the depth map of the next frame by solving a rolling optimization problem in the depth map of each frame.

[0036] Where the state vector Control variables Indicates acceleration The rate of change; the rolling optimization problem is:

[0037] ;

[0038] control variables , indicating jerk;

[0039] Where N=10 is the prediction time domain, This is the reference trajectory obtained from the historical trajectory of the target person. The balance coefficient has a value of 0.1; the state vector. middle The three-dimensional spatial location of the personnel target. For the velocity component of the personnel target, by The first derivative with respect to time is calculated as follows: The acceleration component of the personnel target is... The first derivative with respect to time is calculated; To extrapolate the predicted motion position of the target personnel in the next frame's depth map;

[0040] Calculate the Euclidean distance between the predicted movement position and the actual observed position. If the Euclidean distance is greater than a first distance threshold, determine that the movement of the personnel target has abruptly changed, and reduce the balance coefficient. Reduce the weight of the control variable constraint terms.

[0041] Preferably, the method for performing identity association matching and determination based on feature similarity, trajectory similarity, and location similarity between the predicted location and active trajectories in the historical trajectory database includes the following steps:

[0042] Based on each person target and each active trajectory in the current frame depth map, calculate the comprehensive matching score between the person target and the active trajectory. The comprehensive matching score is:

[0043] ;

[0044] in, For feature similarity; For trajectory similarity; For positional similarity; , , The weights are dynamically adjusted according to the motion state, satisfying ;

[0045] Feature similarity is: ;

[0046] in The range of values ​​is A larger value indicates more similar identity characteristics;

[0047] The trajectory similarity is: ;

[0048] For personnel goals Current trajectory segment With trajectory Historical fragments The normalized distance obtained by dynamic time warping calculation;

[0049] in The range of values ​​is A larger value indicates that the shape features are more similar;

[0050] Positional similarity is: ;

[0051] To obtain personnel targets 3D center point coordinates With trajectory Predicted location obtained through deduction European distance ; This is a spatial scale parameter, with a value of 0.5;

[0052] The movement state of the personnel target is classified into levels based on the movement speed of the personnel target obtained from the simulation, including stationary / slow walking, fast walking, and running;

[0053] Weight values ​​based on motion state level , , Make dynamic adjustments;

[0054] For each person target, an identity association judgment is made. If the overall matching score between the person target and the active trajectory is greater than the matching threshold, the person target and the active trajectory are determined to be the same person. The person target integrates the identity identifier of the active trajectory and the active trajectory is updated. Otherwise, it is determined to be a new person target, a new identity identifier is assigned to it, and its movement trajectory is added to the historical trajectory database.

[0055] For each person target that is successfully matched and whose trajectory is updated, determine whether it has crossed the virtual detection line based on its three-dimensional center point coordinates. If it has crossed the virtual detection line, record it as a crossing event.

[0056] Preferably, the specific implementation method of S5 includes the following steps:

[0057] Create a deduplicated status record for each person's objective, including: the timestamp of the most recent traversal. The direction of the most recent crossing ;

[0058] For each time travel event, perform the following judgment:

[0059] If the target ID of the person in the traversal event has no historical record, count it directly and create a deduplication status record. , ;

[0060] If the target ID of the person who traveled through the event has a historical record and and time interval If the result is a reversal, it is not counted, and the deduplication status record is updated. , ;

[0061] If the target ID of the person who traveled through the event has a historical record and and time interval If the traversal is repeated within a short period of time, it will not be counted, and the deduplication status record will not be updated.

[0062] for and time interval In the case of, or and time interval In the case of normal entry and exit, count and update the deduplication status record;

[0063] Output the cumulative number of people who entered and cumulative number of departures and net flow ;

[0064] in, For the turnaround time threshold, The threshold for repeated travel time.

[0065] Preferably, the specific implementation method of S6 includes the following steps:

[0066] Define micro-level customer flow This represents the net number of people crossing the route per unit time. ;

[0067] in and Time windows The number of new entries and exits;

[0068] The monitoring area is discretized into a discrete grid map composed of several grid cells in the xy plane. The number of depth points in each grid cell is counted to obtain the real-time density field. ;

[0069] Based on the density field of two consecutive depth maps, the velocity field is solved using the variational optical flow method. The optimization problem is as follows:

[0070] min + ;

[0071] Among them: the first term min For continuity equation constraints, Let be the partial derivative of density with respect to time. The divergence of density flux;

[0072] Second item For velocity field smoothing constraints, For the velocity space gradient, For smoothing coefficients;

[0073] Velocity field through virtual detection lines Points are used to obtain macro-level passenger flow. :

[0074] ;

[0075] in The normal vector of the detection line is positive when it points inward to the monitoring area, representing the macroscopic passenger flow. The number of people passing through the virtual detection line per unit time is a scalar, with a positive value representing the net number of people entering and a negative value representing the net number of people leaving.

[0076] Establish a filter for macroscopic passenger flow Make corrections to obtain the corrected passenger flow count.

[0077] Preferably, when establishing the filter, the state equation is a random walk of micro-counting bias, and the observation equation is the relationship between macroscopic flow rate and actual flow rate plus observation noise, specifically:

[0078] Equations of state: ;

[0079] Microscopic observation equation: ;

[0080] Macroscopic observation equation: ;

[0081] in, This represents the cumulative deviation of the microscopic count; This represents real traffic; , The noise follows a normal distribution. and ,Pick , ;

[0082] Based on the aforementioned state-space model, a Kalman filter is used to simultaneously estimate the actual flow rate and the cumulative deviation. When the cumulative deviation estimate is... At that time, the micro-count is corrected to obtain the corrected micro-count value:

[0083] ;

[0084] The revised cumulative number of entries and exits is obtained by integrating over a time window:

[0085] ;

[0086] ;

[0087] in, , and These are the positive and negative deviation components, respectively.

[0088] A 3D binocular personnel attribute recognition and counting system integrating AI and REID includes: an image acquisition and personnel detection module, a feature extraction module, a feature enhancement module, a motion state matching module, a deduplication module, and a correction module;

[0089] The image acquisition and personnel detection module is used to acquire binocular images and construct a three-dimensional space to detect personnel's eyes and their three-dimensional spatial position and motion parameters.

[0090] The feature extraction module, connected to the image acquisition and personnel detection module, is used to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image;

[0091] The feature enhancement module, connected to the feature extraction module, is used to assign orthogonal vectors that are orthogonal to historical orthogonal vectors to personnel targets, and fuse them to generate identity feature vectors;

[0092] The motion state matching module, connected to the image acquisition and personnel detection module and the feature enhancement module, is used to continuously optimize and predict the position of personnel targets, make identity association matching decisions, and detect and record the crossing events of each personnel target.

[0093] The deduplication module, connected to the motion state matching module, is used to deduplicatize crossing events and obtain the deduplicated personnel target count result.

[0094] The correction module, connected to the image acquisition and personnel detection module and the deduplication module, is used to calculate the macroscopic and microscopic passenger flow based on the density field and velocity field, and to establish a filter to correct the microscopic counting deviation of the microscopic passenger flow, so as to obtain the final target personnel counting result.

[0095] The technical solution of this application has at least the following advantages and beneficial effects:

[0096] This invention provides a 3D binocular person attribute recognition and counting method integrating AI and REID. The method first constructs a three-dimensional spatial coordinate system from the acquired binocular images, detects person targets in the three-dimensional space and extracts their motion parameters, and simultaneously constructs a multi-task AI feature extraction network to extract basic appearance features and three-dimensional spatial features of the person. Based on this, a unique orthogonal vector approximately orthogonal to the historical orthogonal vector is assigned to each person target. The appearance features and orthogonal vectors are fused and transformed, enabling the feature vectors of different people to be orthogonally separated in the feature space. This solves the problem of misidentification caused by the small angle between the feature vectors of seemingly similar people in traditional REID methods. Furthermore, the person's motion is modeled as a linear system of state vectors and control variables. The predicted position is optimized through rolling, and the model's response is adaptively adjusted based on the prediction deviation. To improve sensitivity, the system dynamically adjusts the weights of feature similarity, trajectory similarity, and position similarity in the identity association matching of the detected personnel target in the current frame with the existing trajectory in the historical trajectory database based on the inferred motion state. This determines whether a personnel target in the current frame is the same as a previously seen personnel target, enabling the matching to adapt to various motion modes from stationary to running. This solves the problems of identity loss and counting errors caused by the prediction failure of traditional tracking methods in scenarios with sudden speed changes. On this basis, the counting results of personnel targets are deduplicated, and based on the deduplicated counting results, the monitoring area is gridded. The macroscopic passenger flow is obtained by estimating the density field and velocity field, and a filter is established with the microscopic individual tracking results to estimate and correct the cumulative deviation of microscopic counting in real time, thereby correcting the counting results of personnel targets and making the counting results more accurate. Attached Figure Description

[0097] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0098] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0099] See Figure 1 This invention discloses a 3D binocular person attribute recognition and counting method integrating AI and REID, comprising the following steps:

[0100] S1: Acquire binocular images and construct a three-dimensional space based on the acquired binocular images. Detect personnel targets in the constructed three-dimensional space and obtain the three-dimensional spatial position and motion parameters of each personnel target.

[0101] S2: Construct a multi-task AI feature extraction network and use the multi-task AI feature extraction network to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image;

[0102] S3: Assign an orthogonal vector to the personnel target that is orthogonal to the historical orthogonal vector. Use the orthogonal vector to fuse and transform the basic REID feature vector to obtain the orthogonalized basic REID feature vector. Fuse the orthogonalized basic REID feature vector and the three-dimensional space feature vector to obtain the identity feature vector.

[0103] S4: Model the motion of personnel targets as a linear system of state vectors and control variables. Optimize the linear system by rolling to predict the predicted position of the personnel target in the next frame. Based on feature similarity, trajectory similarity, and position similarity, perform identity association matching between the predicted position and active trajectories in the historical trajectory database. If the predicted position matches an active trajectory in the historical trajectory database, count the matches and associate the active trajectory with its identity identifier. Simultaneously, detect and record the crossing events of each personnel target; otherwise, do not count the matches.

[0104] S5: Perform deduplication on the crossing events to obtain the deduplicated personnel target count result;

[0105] S6: Based on the deduplicated personnel target count results, the monitoring area is gridded, and its density field is estimated using the depth map of the monitoring area. The velocity field is solved by the density field of continuous frames. Based on the density field and velocity field, the macro passenger flow and micro passenger flow are calculated, and a filter is established to correct the micro passenger flow micro counting deviation, so as to obtain the final personnel target count results.

[0106] In this embodiment, the specific implementation method of S1 includes the following steps:

[0107] Simultaneously acquire left and right eye images, and perform distortion correction and epipolar correction on the acquired left and right eye images to align the rows of the left and right eye images;

[0108] Stereo matching is performed on the corrected left and right images, and a semi-global algorithm is used to calculate the disparity value of each pixel in each image to obtain a disparity map.

[0109] By using the principle of triangulation, the disparity map is converted into a depth map, and the three-dimensional spatial coordinates of each pixel in the monitoring area are obtained.

[0110] A virtual detection line is defined in three-dimensional space, and the virtual detection line is located at the entrance and / or exit of the monitoring area;

[0111] A lightweight object detection network is used to detect people in the left eye image, and a two-dimensional bounding box corresponding to each person target is obtained. The minimum three-dimensional bounding box of each person target is calculated based on the two-dimensional bounding box.

[0112] Traverse the 3D coordinates of all pixels within the smallest 3D bounding box, and calculate the height, width, and thickness of each person target based on the 3D coordinates;

[0113] Based on the coordinates of the three-dimensional center point of several consecutive depth maps, the motion parameters of the human target are calculated, including motion velocity and acceleration.

[0114] Specifically, calculate the minimum 3D bounding box for each person target:

[0115] Based on the position of the 2D bounding box in the left eye image and the corresponding depth region mapped to the depth map, the 3D spatial coordinates (x, y, z) of all pixels in the region are extracted. The difference between the maximum and minimum values ​​of x, y, and z is used to form the minimum 3D bounding box.

[0116] More specifically, the monitoring area is the image acquisition area of ​​the binocular camera. Distortion and epipolar correction are performed on the left and right eye images to align their rows. Based on this alignment, a semi-global algorithm is used to process the left and right eye images to obtain a disparity map, which is then converted into a depth map. This restores the two-dimensional planar image to its corresponding three-dimensional space, mapping information from the two-dimensional planar image into the three-dimensional space. Based on this restored three-dimensional space, virtual detection lines are defined. For example, a binocular camera is installed at the entrance of a shopping mall. The left and right eye images acquired by the camera simulate the mall entrance as a virtual three-dimensional space, and virtual detection lines are defined at the mall entrances and exits within this virtual three-dimensional space. The flow of people is counted by detecting whether individuals cross these virtual detection lines.

[0117] A lightweight object detection network is used to process the left-eye image to identify human targets in the left-eye image and define a two-dimensional bounding box for the human target. Based on the defined two-dimensional bounding box, the minimum three-dimensional bounding box corresponding to the two-dimensional bounding box in the depth map is calculated. Based on the three-dimensional coordinates within the three-dimensional bounding box, the height, width, and thickness of the human target are calculated. At the same time, the motion velocity and acceleration of the human target are calculated based on the coordinates of the three-dimensional center point of several consecutive depth maps.

[0118] Among them, the height of the personnel target is used Indicates body width using Indicates body thickness Indicates the speed of movement using Indicates acceleration is expressed as... express.

[0119] It should be noted that in this step, a lightweight object detection network is used to process the left eye image to identify human targets. For example, the YOLOv5 object detection network is used. This processing method is a common technique for object detection in the field of image processing and belongs to existing technology, so it will not be described in detail here.

[0120] In this embodiment, the specific implementation method of S2 includes the following steps:

[0121] Construct a multi-task AI feature extraction network that includes a backbone network and a 3D spatial feature extraction branch;

[0122] The left eye image is input into the backbone network to extract the basic REID feature vector; the basic REID feature vector is denoted as... , ;

[0123] The height, width, thickness, and movement speed of the target person are input into the 3D spatial feature extraction branch to extract 3D spatial feature vectors, resulting in 3D spatial feature vectors. , ;

[0124] Specifically, the backbone of the multi-task AI feature extraction network uses a ResNet-50 pre-trained on ImageNet, with its classification layer removed and connected to a global average pooling layer and a fully connected layer, outputting a 512-dimensional base REID feature vector. The size of the input left eye image is normalized to 256*128;

[0125] The 3D spatial feature extraction branch is a feature concatenation layer that does not contain trainable parameters. Its input consists of the height, width, and thickness of the person target, as well as the motion speed of the current frame. These four scalars are concatenated into a 4D vector. , , .

[0126] In this embodiment, the specific implementation method of S3 includes the following steps:

[0127] In this step, for each person target detected in the current frame, feature matching is first performed with the historical trajectory database. If the match is successful, the orthogonalized vector of that historical trajectory is directly inherited; if the match fails, it is determined to be a newly appeared person, and a new orthogonalized vector is assigned to it. When performing feature matching with the historical trajectory database, the cosine similarity is calculated using the basic REID features and the corresponding features stored in the historical trajectory database, and the matching judgment is made based on the cosine similarity.

[0128] Assign an orthogonal vector to each newly appearing person target in the detection area;

[0129] Map the base REID feature vectors to the same dimension as the orthogonalized vectors;

[0130] The orthogonalized vectors of the same dimension and the basic REID feature vectors are multiplied element-wise to obtain the orthogonalized REID feature vectors;

[0131] The orthogonalized REID feature vector is concatenated with the 3D spatial feature vector to obtain the final identity feature vector;

[0132] Specifically, the historical orthogonalization vector refers to the set of all orthogonalization vectors that have been assigned to other personnel targets before the current moment, and is stored in a global orthogonalization vector history database. When assigning orthogonalization vectors to newly appearing personnel targets, the new orthogonalization vector is made orthogonal to the existing orthogonalization vectors. Here, orthogonality is approximate, meaning their inner product is close to 0, thereby separating the orthogonalization vectors of different personnel targets from each other in the feature space and enhancing the distinguishability of the identity features of different personnel targets.

[0133] Let the orthogonalized vector assigned to the personnel target be denoted as , , The orthogonalized REID eigenvectors are denoted as... ;

[0134] It should be noted that, The dimension can be selected according to the actual situation. In actual processing, for personnel target trajectories that have not appeared for more than a set time, their orthogonal vectors can be recovered.

[0135] The orthogonal REID-enhanced feature vector is concatenated with the 3D spatial feature vector to form the final identity feature vector. ;

[0136] Specifically, the generation of orthogonalized vectors includes:

[0137] Randomly generate one dimensional vector ;

[0138] The vector was processed using the Gram-Schmidt procedure. Orthogonalize with all assigned orthogonalized vectors to obtain orthogonalized vectors. .

[0139] In this step, the feature discrimination of each person target is enhanced by orthogonalizing the basic REID feature vector, thereby reducing the mismatch rate of seemingly similar person targets.

[0140] In this embodiment, the specific implementation method of S4 includes the following steps:

[0141] The motion state of each person target is modeled as a state vector. With control variables A linear system is used to deduce the predicted motion position of the personnel target in the depth map of the next frame by solving a rolling optimization problem in the depth map of each frame.

[0142] Where the state vector Control variables Indicates acceleration The rate of change; the rolling optimization problem is:

[0143] ;

[0144] control variables , indicating jerk;

[0145] The state transition equation is ;in and Based on the uniformly accelerated motion model, this invention does not limit its specific values.

[0146] Where N=10 is the prediction time domain, This is the reference trajectory obtained from the historical trajectory of the target person. The balance coefficient has a value of 0.1; the state vector. middle The three-dimensional spatial location of the personnel target. For the velocity component of the personnel target, by The first derivative with respect to time is calculated as follows: The acceleration component of the personnel target is... The first derivative with respect to time is calculated; To extrapolate the predicted movement position of the target personnel in the next frame's depth map.

[0147] The rolling optimization problem includes the following constraints:

[0148] The movement speed of personnel is limited and can be taken to be no more than 5 m / s;

[0149] The rate of change of acceleration is bounded and can be taken to be no more than ;

[0150] The trajectory obtained from the simulation is continuous with the endpoints of the historical trajectory of the personnel target.

[0151] It should be noted that the first term in the rolling optimization problem is the state vector constraint term, used to constrain the positional deviation between the predicted trajectory and the reference trajectory of the personnel target, where the subscript "pos" indicates that only the sum of squared errors of the position components in the state vector are calculated; the second term is the control variable constraint term, used to constrain the rate of change of acceleration and the balance coefficient. Used to adjust the weights between control variable constraints and state vector constraints, reducing the balance coefficient when the motion of the detected personnel target changes abruptly. The value of allows the results of the rolling optimization problem to more flexibly follow changes in acceleration.

[0152] Furthermore, a reference trajectory is obtained from historical trajectories. Polynomial fitting can be used to extrapolate the historical trajectories, taking the position sequence of the person target from the most recent M-frame depth maps within the historical trajectory. A third-order polynomial is then used to fit the change in the target person's position over time, thus obtaining the reference trajectory of the target person. In addition, a reference trajectory can be obtained by extrapolating historical trajectories using a uniform velocity model, and this invention does not limit this method.

[0153] Calculate the Euclidean distance between the predicted movement position and the actual observed position. If the Euclidean distance is greater than a first distance threshold, determine that the movement of the personnel target has abruptly changed, and reduce the balance coefficient. Reduce the weight of control variable constraint terms;

[0154] The formula for calculating Euclidean distance is: ;

[0155] in This refers to the actual observation location. To predict the position of movement.

[0156] Specifically, the first distance threshold can be set to 0.5m;

[0157] Based on each person target and each active trajectory in the current frame depth map, calculate the comprehensive matching score between the person target and the active trajectory. The comprehensive matching score is:

[0158] ;

[0159] in, For feature similarity; For trajectory similarity; For positional similarity; , , The weights are dynamically adjusted according to the motion state, satisfying ;

[0160] Feature similarity is used to measure personnel targets With historical trajectory The degree of matching in terms of identity characteristics; targeting personnel. Identity feature vector With historical trajectory The latest identity feature vector The cosine similarity is calculated using the following formula:

[0161] ;

[0162] in The range of values ​​is A larger value indicates more similar identity characteristics;

[0163] trajectory similarity Used to measure personnel goals Current motion trajectory and trajectory The degree of shape matching of the historical movement trajectory, taking personnel targets Current trajectory segment With trajectory Historical fragments Perform dynamic time warping and calculate normalized distance. ,in Indicates recent Frame depth map, trajectory similarity is:

[0164] ;

[0165] in The range of values ​​is A larger value indicates that the shape features are more similar;

[0166] Location similarity Used to measure personnel goals Current location and trajectory Predict the proximity of locations and select personnel targets. 3D center point coordinates With trajectory Predicted location obtained through deduction European distance The Gaussian kernel function is used to transform it into positional similarity, specifically:

[0167] ;

[0168] in The value is 0.5. For spatial scale parameters; this Gaussian kernel function maps distance to The closer the distance, the higher the similarity.

[0169] The matching weights of feature similarity, trajectory similarity, and position similarity are dynamically adjusted based on the deduced motion state, specifically as follows:

[0170] The movement state of the personnel target is classified into levels based on the movement speed of the personnel target obtained from the simulation, including stationary / slow walking, fast walking, and running;

[0171] Weight values ​​based on motion state level , , Dynamic adjustments will be made, and the specific adjustment rules are as follows:

[0172] When the target is stationary / moving slowly, , , The three weights are evenly distributed;

[0173] When the target is moving at a fast pace, improve Weight value, decrease Weight value, The weight values ​​remain unchanged;

[0174] When the target is running, increase Weight value, decrease Weight value, The weight values ​​remain unchanged;

[0175] For example, when the target is stationary / walking slowly, , , ;

[0176] When the target is moving fast. , , ;

[0177] When the target is running , , .

[0178] For each target person, an identity association judgment is made. If the overall matching score between the target person and the active trajectory is greater than the matching threshold, the target person and the active trajectory are determined to be the same person. The target person integrates the identity identifier of the active trajectory and the active trajectory is updated. Otherwise, it is determined to be a new target person, a new identity identifier is assigned to it, and its movement trajectory is added to the historical trajectory database.

[0179] For each person target that is successfully matched and whose trajectory is updated, determine whether it has crossed the virtual detection line based on its three-dimensional center point coordinates. If it has crossed the virtual detection line, record it as a crossing event.

[0180] Specifically, whether or not the virtual detection line has been crossed is determined by the following steps:

[0181] If the coordinates of the 3D center point of the personnel target and the coordinates of the 3D center point of the depth map in the previous frame are on opposite sides of the virtual detection line, it is determined that a crossing event has occurred.

[0182] The direction of crossing is: if entering the monitoring area from outside the monitoring area, the direction is... If one leaves the monitoring area and goes outside the monitoring area, the direction... ;

[0183] Record each time travel event as ,in Let be the orthogonalization vector of the target. This is the current frame time.

[0184] In this embodiment, the implementation of S5 includes the following steps:

[0185] In this step, the crossing events generated by S4 are deduplicated, eliminating backtracking behaviors of the same person target and repeated crossings within a short period of time, to obtain accurate personnel count results:

[0186] Create a deduplicated status record for each person's objective, including: the timestamp of the most recent traversal. The direction of the most recent crossing ;

[0187] If no crossing event occurs for a personnel target within 60 seconds, its deduplicated status record is deleted.

[0188] For each time travel event, perform the following judgment:

[0189] If the target ID of the person in the traversal event has no historical record, count it directly and create a deduplication status record. , ;

[0190] If the target ID of the person who traveled through the event has a historical record and and time interval If the result is a reversal, it is not counted, and the deduplication status record is updated. , ;

[0191] If the target ID of the person who traveled through the event has a historical record and and time interval If the traversal is repeated within a short period of time, it will not be counted, and the deduplication status record will not be updated.

[0192] for and time interval In the case of, or and time interval In the case of normal entry and exit, the entry and exit status is counted and the deduplication status record is updated.

[0193] Output the cumulative number of people who entered and cumulative number of departures and net flow .

[0194] Among them, the turnaround time threshold The threshold for repeated travel time is 5 seconds. The threshold is 2 seconds, and the above threshold can be adjusted according to the actual scenario.

[0195] In this embodiment, the specific implementation method of S6 includes the following steps:

[0196] Define micro-level customer flow This represents the net number of people crossing the route per unit time. ;

[0197] in and Time windows The number of new entries and exits; The settings can be adjusted according to the actual frame rate and scene. In this embodiment, we take... .

[0198] The monitoring area is discretized into a discrete grid map composed of several grid cells in the xy plane. The number of depth points in each grid cell is counted to obtain the real-time density field. ;

[0199] The grid size is 0.2m * 0.2m;

[0200] Based on the density field of two consecutive depth maps, the velocity field is solved using the variational optical flow method. The optimization problem is as follows:

[0201] min + ;

[0202] Among them: the first term min For continuity equation constraints, Let be the partial derivative of density with respect to time. The divergence of density flux;

[0203] Second item For velocity field smoothing constraints, For the velocity space gradient, For smoothing coefficients;

[0204] This optimization problem can be solved by solving the corresponding Euler-Lagrange equations or by using numerical iteration methods.

[0205] Velocity field through virtual detection lines Points are used to obtain macro-level passenger flow. :

[0206] ;

[0207] in The normal vector of the detection line is positive when it points inward to the monitoring area, representing the macroscopic passenger flow. The number of people passing through the virtual detection line per unit time is a scalar, with a positive value representing the net number of people entering and a negative value representing the net number of people leaving.

[0208] Establish a filter for macroscopic passenger flow Make corrections to obtain the corrected passenger flow count results;

[0209] When establishing the filter, the state equation is a random walk with micro-counting bias, and the observation equation is:

[0210] Equations of state: ;

[0211] Microscopic observation equation: ;

[0212] Macroscopic observation equation: ;

[0213] in, This represents the cumulative deviation of the microscopic count; This represents real traffic; , The noise follows a normal distribution. and ,Pick , ;

[0214] Based on the aforementioned state-space model, a Kalman filter is used to simultaneously estimate the actual flow rate and the cumulative deviation. When the cumulative deviation estimate is... At that time, the micro-count is corrected to obtain the corrected micro-count value:

[0215] ;

[0216] The revised cumulative number of entries and exits is obtained by integrating over a time window:

[0217] ;

[0218] ;

[0219] in, , and These are the positive and negative deviation components, respectively, which can be derived from... It is obtained by decomposing positive and negative values.

[0220] The specific recursive formula for Kalman filtering is common knowledge in this field and will not be elaborated here.

[0221] In another aspect, the present invention provides a 3D binocular personnel attribute recognition and counting system integrating AI and REID, comprising: an image acquisition and personnel detection module, a feature extraction module, a feature enhancement module, a motion state matching module, a deduplication module, and a correction module;

[0222] The image acquisition and personnel detection module is used to acquire binocular images and construct a three-dimensional space based on the acquired binocular images. Personnel targets are detected in the constructed three-dimensional space, and the three-dimensional spatial position and motion parameters of each personnel target are obtained.

[0223] The feature extraction module, connected to the image acquisition and personnel detection module, is used to construct a multi-task AI feature extraction network. The multi-task AI feature extraction network is used to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image.

[0224] The feature enhancement module, connected to the feature extraction module, is used to assign an orthogonal vector orthogonal to the historical orthogonal vector to the personnel target. The orthogonal vector is used to fuse and transform the basic REID feature vector to obtain the orthogonal basic REID feature vector. The orthogonal basic REID feature vector and the three-dimensional space feature vector are fused to obtain the identity feature vector.

[0225] The motion state matching module, connected to the image acquisition and personnel detection module and the feature enhancement module, is used to model the motion of personnel targets as a linear system of state vectors and control variables. By scrolling and optimizing the linear system, it predicts the predicted position of the personnel target in the next frame. Based on feature similarity, trajectory similarity, and position similarity, it performs identity association matching judgment between the predicted position and active trajectories in the historical trajectory database. If the predicted position matches an active trajectory in the historical trajectory, it counts and associates the identity of the active trajectory. At the same time, it detects and records the crossing events of each personnel target.

[0226] The deduplication module, connected to the motion state matching module, is used to deduplicatize crossing events and obtain the deduplicated personnel target count result.

[0227] The correction module, connected to the image acquisition and personnel detection module and the deduplication module, is used to grid the monitoring area based on the deduplicated personnel target count results, estimate its density field using the depth map of the monitoring area, solve the velocity field through the density field of consecutive frames, calculate the macro passenger flow and micro passenger flow based on the density field and velocity field, and establish a filter to correct the micro counting deviation of the micro passenger flow, thus obtaining the final personnel target count result.

[0228] The various embodiments of the present invention have now been described in detail. To avoid obscuring the concept of the invention, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions of this invention based on the above description, and the scope of the invention is defined by the appended claims.

Claims

1. A 3D binocular person attribute recognition and counting method integrating AI and REID, characterized in that, Includes the following steps: S1: Acquire binocular images and construct a three-dimensional space based on the acquired binocular images. Detect personnel targets in the constructed three-dimensional space and obtain the three-dimensional spatial position and motion parameters of each personnel target. S2: Construct a multi-task AI feature extraction network and use the multi-task AI feature extraction network to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image; S3: Assign an orthogonal vector to the personnel target that is orthogonal to the historical orthogonal vector. Use the orthogonal vector to transform the basic REID feature vector to obtain the orthogonalized basic REID feature vector. Then fuse it with the three-dimensional space feature vector to obtain the identity feature vector. S4: Model the motion of the personnel target as a linear system of state vectors and control variables. Optimize the linear system by rolling to predict the predicted position of the personnel target in the next frame. Based on feature similarity, trajectory similarity, and position similarity, perform identity association matching between the predicted position and active trajectories in the historical trajectory database. If the match is successful, count the events and associate them with identity identifiers, while detecting and recording crossing events; otherwise, do not count the events. S5: Perform deduplication on the crossing events to obtain the deduplicated personnel target count result; S6: Based on the deduplicated personnel target count results, the monitoring area is gridded, and its density field is estimated using the depth map of the monitoring area. The velocity field is solved by the density field of continuous frames. Based on the density field and velocity field, the macro passenger flow and micro passenger flow are calculated, and a filter is established to correct the micro passenger flow micro counting deviation, so as to obtain the final personnel target count results.

2. The 3D binocular person attribute recognition and counting method integrating AI and REID as described in claim 1, characterized in that, The specific implementation of S1 includes the following steps: Simultaneously acquire left and right eye images, and perform distortion correction and epipolar correction on the acquired left and right eye images to align the rows of the left and right eye images; Stereo matching is performed on the corrected left and right images, and a semi-global algorithm is used to calculate the disparity value of each pixel in each image to obtain a disparity map. By using the principle of triangulation, the disparity map is converted into a depth map, and the three-dimensional spatial coordinates of each pixel in the monitoring area are obtained. A virtual detection line is defined in three-dimensional space, and the virtual detection line is located at the entrance and / or exit of the monitoring area; A lightweight object detection network is used to detect people in the left eye image, and a two-dimensional bounding box corresponding to each person target is obtained. The minimum three-dimensional bounding box of each person target is calculated based on the two-dimensional bounding box. Traverse the 3D coordinates of all pixels within the smallest 3D bounding box, and calculate the height, width, and thickness of each person target based on the 3D coordinates; Based on the coordinates of the three-dimensional center point of several consecutive depth maps, the motion parameters of the human target are calculated, including motion velocity and acceleration.

3. The 3D binocular person attribute recognition and counting method integrating AI and REID as described in claim 1, characterized in that, The specific implementation method of S2 includes the following steps: Construct a multi-task AI feature extraction network that includes a backbone network and a 3D spatial feature extraction branch; The left eye image is input into the backbone network to extract the basic REID feature vector; The height, width, thickness, and movement speed of the target person are input into the three-dimensional spatial feature extraction branch to extract the three-dimensional spatial feature vector, thus obtaining the three-dimensional spatial feature vector.

4. The 3D binocular person attribute recognition and counting method integrating AI and REID as described in claim 1, characterized in that, The specific implementation method of S3 includes the following steps: Assign an orthogonal vector to each newly appearing person target in the detection area; Map the base REID feature vectors to the same dimension as the orthogonalized vectors; The orthogonalized vectors of the same dimension and the basic REID feature vectors are multiplied element-wise to obtain the orthogonalized REID feature vectors; The orthogonalized REID feature vector is concatenated with the 3D spatial feature vector to obtain the final identity feature vector.

5. The 3D binocular person attribute recognition and counting method integrating AI and REID as described in claim 1, characterized in that, The method of modeling the motion of the person target as a linear system of state vectors and control variables, and predicting the predicted position of the person target in the next frame by optimizing the linear system through rolling, includes the following steps: The motion state of each person target is modeled as a state vector. With control variables A linear system is used to deduce the predicted motion position of the personnel target in the depth map of the next frame by solving a rolling optimization problem in the depth map of each frame. Where the state vector Control variables Indicates acceleration The rate of change; The rolling optimization problem is: ; control variables , indicating jerk; Where N is the prediction time domain, This is the reference trajectory obtained from the historical trajectory of the target person. Balance coefficients; state vector middle The three-dimensional spatial location of the personnel target. For the velocity component of the personnel target, The acceleration component of the personnel target; To predict the motion position of the target person in the next frame depth map; Calculate the Euclidean distance between the predicted movement position and the actual observed position. If the Euclidean distance is greater than a first distance threshold, determine that the movement of the personnel target has abruptly changed, and reduce the balance coefficient. .

6. The 3D binocular person attribute recognition and counting method integrating AI and REID according to claim 1, characterized in that, The method for performing identity association matching and determination based on feature similarity, trajectory similarity, and location similarity between the predicted location and active trajectories in the historical trajectory database includes the following steps: Based on each person target and each active trajectory in the current frame depth map, calculate the comprehensive matching score between the person target and the active trajectory. The comprehensive matching score is: ； in, For feature similarity; For trajectory similarity; For positional similarity; , , The weights are dynamically adjusted according to the motion state, satisfying ; Feature similarity is: ;in Indicates the first Individual targets, Indicates the first A historical trajectory, Indicate personnel objectives Identity feature vector, Representing historical trajectory The latest identity feature vector; in The range of values ​​is A larger value indicates more similar identity characteristics; The trajectory similarity is: ; For personnel goals Current trajectory segment With trajectory Historical fragments The normalized distance obtained by dynamic time warping calculation; in The range of values ​​is A larger value indicates that the shape features are more similar; Positional similarity is: ; To obtain personnel targets 3D center point coordinates With trajectory Predicted location obtained through deduction European distance ; This is a spatial scale parameter, with a value of 0.5; The movement state of the personnel target is classified into levels based on the movement speed of the personnel target obtained from the simulation, including stationary / slow walking, fast walking, and running; Weight values ​​based on motion state level , , Make dynamic adjustments; For each person target, an identity association judgment is made. If the overall matching score between the person target and the active trajectory is greater than the matching threshold, the person target and the active trajectory are determined to be the same person. The person target integrates the identity identifier of the active trajectory and the active trajectory is updated. Otherwise, it is determined to be a new person target, a new identity identifier is assigned to it, and its movement trajectory is added to the historical trajectory database. For each person target that is successfully matched and whose trajectory is updated, determine whether it has crossed the virtual detection line based on its three-dimensional center point coordinates. If it has crossed the virtual detection line, record it as a crossing event.

7. The 3D binocular person attribute recognition and counting method integrating AI and REID according to claim 1, characterized in that, The specific implementation method of S5 includes the following steps: Create a deduplicated status record for each person's objective, including: the timestamp of the most recent traversal. The direction of the most recent crossing ; For each time travel event, perform the following judgment: If the target ID of the person in the traversal event has no historical record, count it directly and create a deduplication status record. , ; If the target ID of the person who traveled through the event has a historical record and and time interval If the result is a reversal, it is not counted, and the deduplication status record is updated. , ; If the target ID of the person who traveled through the event has a historical record and and time interval If the traversal is repeated within a short period of time, it will not be counted, and the deduplication status record will not be updated. for and time interval In the case of, or and time interval In the case of normal entry and exit, count and update the deduplication status record; Output the cumulative number of people who entered and cumulative number of departures and net flow ; in, The turnaround time threshold, The threshold for repeated travel time.

8. The 3D binocular person attribute recognition and counting method integrating AI and REID according to claim 1, characterized in that, The specific implementation method of S6 includes the following steps: Define micro-level customer flow This represents the net number of people crossing the route per unit time. ; in and Time windows The number of new entries and exits; The monitoring area is discretized into a discrete grid map composed of several grid cells in the xy plane. The number of depth points in each grid cell is counted to obtain the real-time density field. ; Based on the density field of two consecutive depth maps, the velocity field is solved using the variational optical flow method. The optimization problem is as follows: ； Among them: the first term min For continuity equation constraints, Let be the partial derivative of density with respect to time. The divergence of density flux; Second item For velocity field smoothing constraints, For the velocity space gradient, For smoothing coefficients; Velocity field through virtual detection lines Points are used to obtain macro-level passenger flow. : ； in The normal vector of the detection line is positive when it points inward to the monitoring area, representing the macroscopic passenger flow. The number of people passing through the virtual detection line per unit time is a scalar, with a positive value representing the net number of people entering and a negative value representing the net number of people leaving. Establish a filter for macroscopic passenger flow Make corrections to obtain the corrected passenger flow count.

9. A 3D binocular person attribute recognition and counting method integrating AI and REID according to claim 8, characterized in that, When establishing the filter, the state equation is a random walk of microscopic counting bias, and the observation equation is the relationship between macroscopic flow rate and actual flow rate plus observation noise, specifically: Equations of state: ; Microscopic observation equation: ; Macroscopic observation equation: ; in, This represents the cumulative deviation of the microscopic count; This represents real traffic; , The noise follows a normal distribution. and ,Pick , ; Based on the aforementioned state-space model, a Kalman filter is used to simultaneously estimate the actual flow rate and the cumulative deviation. When the cumulative deviation estimate is... At that time, the micro-count is corrected to obtain the corrected micro-count value: ； The revised cumulative number of entries and exits is obtained by integrating over a time window: ； ； in, , and These are the positive and negative deviation components, respectively.

10. A 3D binocular people attribute recognition and counting system integrating AI and REID, used to perform the method according to any one of claims 1-9, characterized in that, include: Image acquisition and personnel detection module, feature extraction module, feature enhancement module, motion state matching module, deduplication module, and correction module; The image acquisition and personnel detection module is used to acquire binocular images and construct a three-dimensional space to detect personnel's eyes and their three-dimensional spatial position and motion parameters. The feature extraction module, connected to the image acquisition and personnel detection module, is used to extract the basic REID feature vectors and three-dimensional spatial feature vectors of each person target in the binocular image; The feature enhancement module, connected to the feature extraction module, is used to assign orthogonal vectors that are orthogonal to historical orthogonal vectors to personnel targets, and fuse them to generate identity feature vectors; The motion state matching module, connected to the image acquisition and personnel detection module and the feature enhancement module, is used to continuously optimize and predict the position of personnel targets, make identity association matching decisions, and detect and record the crossing events of each personnel target. The deduplication module, connected to the motion state matching module, is used to deduplicatize crossing events and obtain the deduplicated personnel target count result. The correction module, connected to the image acquisition and personnel detection module and the deduplication module, is used to calculate the macroscopic and microscopic passenger flow based on the density field and velocity field, and to establish a filter to correct the microscopic counting deviation of the microscopic passenger flow, so as to obtain the final target personnel counting result.

Images