A multi-camera behavior trajectory tracking method

Abstract

The present application relates to a kind of multi-camera behavior trajectory tracking method, belong to computer vision technical field, including the following steps: S1: specify public service hall multiple cameras, multiple cameras under service area coordinate information is collected respectively, coordinate information is bound with camera;S2: in combination with target detection model and pedestrian re-identification model, customer track is tracked, and customer activity trajectory library of multiple cameras is established;S3: the IoU of customer target detection frame and service area under multiple cameras and depth map confirm whether customer reaches service area;S4: the behavior of customer entering service area to customer leaving service area is identified;S5: the customer behavior trajectory library under multiple cameras is constructed, and behavior is filtered by time information alignment, and the behavior trajectory of customer is obtained.The patent is tracked by multiple cameras under customer, and more effective customer behavior trajectory is obtained, which can be used for assisting public service hall business process formulation and division.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a method for tracking the behavior trajectory of multiple cameras. Background Technology

[0002] Public service halls receive a large number of customers daily, and some service procedures are quite complex. Currently, there is no good method to identify customer behavior patterns. By identifying customer behavior patterns, staff can understand the customer's transaction process and dwell time, thereby optimizing business process design and the placement of equipment in the service hall. Combining customer transaction information from multiple cameras can help identify the overall customer transaction process.

[0003] Currently, research methods for behavioral trajectory recognition can be divided into two types: one is based on computer vision, and the other is based on establishing dynamic models using sensors.

[0004] Sensor-based methods require customers to wear corresponding sensors, but public service halls see a large number of customers daily, making sensor-based methods costly. This approach has certain limitations due to cost constraints.

[0005] Computer vision-based methods model and simulate real-world scenarios to obtain relatively accurate recognition results. These methods do not require customer cooperation, saving time and money. Furthermore, they cleverly combine customer behavior trajectories from multiple cameras, avoiding recognition errors and occlusions, and can obtain effective behavior trajectories based on customer service priorities. This can be used to assist in the formulation and division of business processes in public service halls. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a multi-camera behavior trajectory tracking method. By tracking the behavior trajectory of customers in the service hall, the method can understand the customer's business process and dwell time, thereby optimizing the business process and the placement of service hall equipment. The method determines the customer's movement trajectory in the service hall by using the location information of the customer and key service areas. In key service areas, the customer's behavior is recorded. Behavioral trajectory information from different angles captured by multiple cameras is aligned and integrated using time information to ultimately form the customer's behavior trajectory in the service hall.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for tracking the behavior trajectory of multiple cameras includes the following steps:

[0009] S1: Calibrate multiple cameras and collect the coordinate information of the service area under each camera, and bind the coordinate information to the camera;

[0010] S2: Combine the target detection model and the pedestrian re-identification model to track customer trajectories and establish a customer activity trajectory database from multiple cameras;

[0011] S3: Calculate the intersection-union ratio (IoU) between the service area and the customer target detection box under multiple cameras. When the IoU is greater than a certain threshold, use the depth map detection model to confirm whether the customer has arrived at the service area.

[0012] S4: Use a behavior recognition model to identify the behavior of customers from the time they enter the service area to the time they leave the service area;

[0013] S5: Build a customer behavior trajectory database under multiple cameras, align time information, delete duplicate behaviors, select the final behavior based on priority when different behaviors appear under different cameras at the same time, and thus obtain the customer's behavior trajectory.

[0014] Furthermore, step S1 specifically includes the following steps:

[0015] S11: Obtain the video capture information from multiple cameras in the public service hall, specify the corresponding camera and mark it as {C1,…,C1,…} k ,…,C cam}, k∈{1,…,cam}, where C k Let represent the k-th camera, and there are a total of cam different cameras.

[0016] S12: Obtain the coordinate information of the public service hall service area from multiple cameras using an object detection model: After labeling and organizing the collected image information, it is fed into the model for training to obtain an object detection model for recognizing the service area. This is done according to different cameras {C1,…,C…}. k ,…,C cam The obtained image information from different perspectives is sequentially processed by a trained object detection model to calculate the coordinate information of the service area under multiple cameras. The coordinate information includes the indices of the upper left and lower right corners of the object detection boxes, denoted as follows: Where s i Let represent the i-th service region, i∈{1,2,3,…,sen}, where there are a total of sen service regions. 1 represents the coordinate of the top-left corner, and 2 represents the coordinate of the bottom-right corner. In C... k The coordinate information obtained from the camera is represented as C. k :

[0017] Furthermore, step S2 specifically includes the following steps:

[0018] S21: Locating customers under multiple cameras using an object detection model: Based on a certain frame rate, extract frames from the service hall camera footage. Use the trained object detection model to sequentially identify customers from multiple camera feeds, sampling at a specified step size to obtain C. k C Frame images captured by the camera and their corresponding time information k :{[T1,F1],[T2,F2],…,[T d ,F d ]}, where T records the time information of the sampled frame F (this time represents Beijing time, accurate to the second), and d represents the total number of frames. Then, the object detection model is used to sequentially sample the video frames {F1, F2, ..., F}. d Customers were detected in the data, and their cus was obtained. j Object detection bounding box coordinate information po j , Among them, cus j Let represent the j-th customer, where 1 represents the top-left corner coordinate and 2 represents the bottom-right corner coordinate. Since multiple customers may exist in a single frame, the time interval is set to T. d frame F d The set of location information of all customers detected in the process is represented as follows: in This represents the coordinate information of the j-th customer in the d-th frame, meaning there are a total of custd customers with coordinate information in this frame. After obtaining the customer coordinate information, it is bound to the frame and time information, represented as C. k :{[T1,Pos1,F1],[T2,Pos2,F2],…,[T d Pos d ,F d ]}.

[0019] S22: Establish a customer ID set, and save the frame sequence of the same customer based on the customer ID set to form an activity trajectory set: Crop the customer location information Pos saved in each frame image F in S21 and input it into the pedestrian re-identification model PCB. Identify the ID value corresponding to the customer at each location and store it in the customer ID set. If the customer does not exist in the customer ID set and does not belong to the staff ID, it indicates that the customer is a newly appeared customer, and assign it a new ID and save it to the customer ID set; if the customer ID set contains the customer's information, extract its ID as the identifier for trajectory tracking. Frame images containing the same customer ID are arranged according to the time order saved in S21 {T1, T2, ..., T...}. d Save as customer's cus j The activity trajectory of C is represented as C. k :{cus j [F′ m ,…,F′n ]}, where m,n∈{1,2,3,…,d}, cus j F' represents the customer with id value j, and F' represents the customer with cus. j The frame contains coordinate and time information. Let cus j activity trajectory Traversing C k The camera captures different activity trajectories of all customers, which are saved as a customer activity trajectory set, represented as... Where j = 1, ..., allcust, Indicates customer cus j The activity trajectory shows a total of allcust customers.

[0020] Furthermore, step S3 specifically includes the following steps:

[0021] S31: Extract C from the activity trajectory set k Next customer cus j Activity trajectory sequence {F′ m ,…,F′ n}, calculate the customer and service area s in sequence. i The intersection and union (IoU) of the bounding boxes is calculated. If the customer's bounding box is A, and the service area's bounding box is B, then...

[0022]

[0023] Extract the coordinates of the service area and the customer from steps S12 and S21, respectively. and A area ∩B area Form a rectangle S, the length of S is... The width of S Then A∩B = width * length. Based on the values ​​of A, B, and A∩B, the IoU value can be calculated.

[0024] If the IoU is greater than a threshold I, it indicates that the customer's bounding box overlaps with the service area's bounding box. In this case, the depth map detection model is activated, and video frames whose bounding boxes overlap with the service area's bounding box are selected from the customer's activity trajectory. This video frame sequence is then saved as C. k :{cus j [F′ p ,…,F′ q ]},p,q∈{m,…,n}.

[0025] S32: Overlap the video frame sequence {F′} obtained in step S31 with the service area target detection box. p ,…,F′q The data is fed into the Megadepth depth map detection model to obtain the pixel value of each pixel in the customer and service areas. This pixel value represents the depth of that point from the camera. The set of depth pixels within the customer target detection box is... in Indicates customer cus j The pixel value of the a-th pixel; the set of depth pixels within the target detection bounding box of the service area is in Indicates service area s i The pixel value of the b-th pixel.

[0026] Since the target detection bounding box may contain other interfering pixels besides the currently tracked customer, such as other passersby or background pixels, the absolute median deviation (MAD) will be used to process the pixel values ​​in the customer target detection bounding box. First, the median value P of all pixels will be calculated. median , Next, calculate the absolute deviation of all elements from the median, bias = |Pi - Pmedian|, and obtain the median absolute deviation, MAD = bias. median Then, determine the parameter n and make the following adjustments to all data:

[0027]

[0028] Finally, the optimized set of customer pixel values ​​is obtained as follows: The set of pixel values ​​in the service area is

[0029] S33: Average the optimized customer and service area depth pixel values ​​from step S32, then subtract them to obtain the depth matching value mc. If the depth matching value mc is less than a certain threshold K, it means that the customer has entered the service area.

[0030] S34: In the activity trajectory map set, delete video frames with a depth matching value mc greater than K, indicating customer cus. j The frame did not remain in the service area. The optimized set of activity trajectory maps is represented as C. k : Where f, g ∈ {p, …, q}, s i This indicates the service area where the customer is currently located. When the customer's bounding box does not intersect with the service area and the depth matching value mc is greater than K, it means that the customer is not near this service area or has already left this service area.

[0031] S35: Repeat steps S31 to S34 to continue tracking the customer. When the customer enters the next service area, save the tracking information to the customer's activity trajectory set.

[0032] Furthermore, step S4 specifically includes the following steps:

[0033] S41: Training the behavior recognition model to detect customer behavior: Extract the coordinate information of the customer's activity trajectory sequence in each service area (S34), and label the customer's behavior frame by frame based on the coordinate information. During training, extract the last 63 frames of the labeled frames, and input these 64 frames as a video clip into the behavior recognition model for training. The same 64-frame activity trajectory sequence is cyclically input into two network branches using different sampling frequencies.

[0034] S42: Establish camera C k The behavior trajectory set is generated by using a trained behavior recognition model, Slowfast, to detect customer behavior in a certain service area and calculate the customer's dwell time in that service area. The obtained behavior data is then saved to the activity trajectory set, forming the behavior trajectory set. The behavior trajectory set contains the data of the cus-th... j A customer in a certain service area i Customer behavior, denoted as C. k :{cus j [(s1,act1),…,(s i ,act r )]}, where act1…,act r This represents all possible customer behaviors in the service hall, where r represents the total number of behavior categories. Obtain the specific service area s obtained in step S34. i Next frame sequence This frame sequence was acquired by sampling one frame out of 64 frames, so one frame represents two seconds. The total number of frames was calculated and multiplied by 2, and stored as the customer's stay time t in the service area. The customer cus with added time information... j In C k The behavioral trajectory under the camera is represented as C k :{cus j [(s1,act1,t1),…,(s i ,act r ,t i )]}, where s i Let act1, act2, act3 represent the i-th service area. r t represents all possible customer behaviors. i Indicates that the customer is in s i Stop time at service areas.

[0035] S43: Traverse all cameras {C1,…,C}k ,…,C cam The video data is used to repeat the above process to obtain multiple sets of customer behavior trajectories from multiple cameras, thus forming a customer behavior trajectory database from multiple cameras.

[0036] Furthermore, step S5, which involves constructing a customer behavior trajectory database based on multiple cameras, specifically includes the following steps:

[0037] S51: Establish a link between the behavioral patterns of a specific customer captured by multiple cameras: Let a specific customer... j In a single camera C k The behavioral trajectory is {(s1,act1,t1),…,(s i ,act r ,t i )} is C k_trackk C k_trackk ={(s1,act1,t1),…,(s i ,act r ,t i Based on the customer ID, the behavioral trajectory sets of the same customer under multiple cameras are saved as a behavioral trajectory library, i.e., cus. j : Where C k_trackk This represents the customer's behavioral trajectory under the k-th camera. k .

[0038] S52: Align customer behavior trajectories from multiple cameras using time information: After obtaining the behavior trajectory of the same customer from multiple cameras, query C in S22. k Next customer cus j Information C k :{cus j [F′ m ,…,F′ n ]}, obtain the time when the customer arrives at the service area, and add this time to the customer behavior trajectory to obtain {(T1,s1,act1,t1),…,(T i ,s i ,act r ,t i )}. Customer cus j behavioral trajectory By time sequence {T1,T2,…,T i The comparisons are performed sequentially, and duplicate entries are removed.

[0039] One challenge is selecting the appropriate behavior when different cameras detect inconsistent actions from the same customer at the same time. Since customer actions such as processing transactions and seeking information in public services have higher research value, this patent prioritizes customer behaviors. When different cameras detect different behaviors from the same customer at the same time, this patent selects the highest-priority behavior as the final behavior. The resulting customer behavior trajectory in the service hall is as follows: j :{(T1,s1,act1,t1),…,(T i ,s i ,act r ,t i )}, where (T i ,s i ,act r ,t i ) indicates customer cus j At time T i Arrived at s i The service area has made an action r The behavior, and the time spent there is t. i .

[0040] The beneficial effects of this invention are as follows:

[0041] (1) This patent proposes a multi-camera behavior trajectory tracking method. The method determines the customer's activity trajectory in the service hall by using the location information of the customer and key service areas. The customer's behavior in the key service areas is recorded to form a customer behavior trajectory. The customer's behavior trajectories across multiple cameras are aligned using time information. When different cameras identify different behaviors of the same customer at the same time, this patent considers selecting the behavior with higher priority as the final behavior to determine the customer's behavior trajectory in the service hall.

[0042] (2) This patent uses the Intersection over Union (IoU) ratio and the Megadepth monocular image depth estimation model to detect whether people have entered the service area in the public service hall. It can quickly identify customer activity trajectories in video information, assisting in the detection of customer behavior trajectories. Furthermore, it processes outlier pixels using Median Absolute Deviation (MAD) to avoid interference from background and other customers on the recognition results.

[0043] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart of a multi-camera behavior trajectory tracking method disclosed in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating the overall process of behavior trajectory recognition disclosed in an embodiment of the present invention. Detailed Implementation

[0047] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0048] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0049] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0050] like Figure 1-2 As shown, the present invention provides a multi-camera behavior trajectory tracking method, comprising the following steps:

[0051] S1: Calibrate multiple cameras and collect the coordinate information of the service area under each camera, and bind the coordinate information to the camera;

[0052] S11: Obtain footage from multiple cameras in the public service hall, selecting cameras that can cover the service areas as much as possible. These service areas include: temperature screening area, information desk, self-service machines, and counter service areas. When selecting cameras, choose those that can identify these service areas and meet the multi-view requirement; therefore, prioritize cameras in the four corners of the public service hall. Select the corresponding cameras and label them as {C1,…,C…}. k ,…,C cam}, k∈{1,…,cam}, where C k Let represent the k-th camera, and there are a total of cam different cameras.

[0053] S12: Use the optional object detection model YOLOv4 to obtain service area coordinate information from multiple cameras. After labeling and organizing the collected service hall image information, it is fed into the YOLOv4 model for training, resulting in an object detection model for recognizing the service area of ​​the public service hall. The training process is as follows: Select relatively clear video frames from different camera video feeds, and use ffmeg software to crop these frames. Select 1000 images from different cameras sequentially for training. Crop all camera image sets into 416*416 images and divide them into training and test sets. During training, perform operations such as horizontal flipping on the training set to enhance the data. Train for 100 rounds, with an initial learning rate of 1e-3, a batch size of 96, and a weight decay rate of 5e-4. After obtaining the trained object detection model through the above training process, it is used according to different cameras {C1,…,C…} k ,…,C cam The obtained image information from different perspectives is sequentially processed by a trained object detection model to calculate the coordinate information of the service area under multiple cameras. The coordinate information includes the indices of the upper left and lower right corners of the object detection boxes, denoted as follows: Where s i Let represent the i-th service region, i∈{1,2,3,…,sen}, where there are a total of sen service regions. 1 represents the coordinate of the top-left corner, and 2 represents the coordinate of the bottom-right corner. In C... k The coordinate information obtained from the camera is represented as C. k :

[0054] S2: Combine the target detection model and the pedestrian re-identification model to track customer trajectories and establish a customer activity trajectory database from multiple cameras;

[0055] S21: Locate customers from multiple cameras using an object detection model: Based on a frame rate of 32fps, extract frames from the service hall camera footage. Use the trained object detection model to sequentially identify customers from multiple camera surveillance videos, sampling at a specified step size of 64, to obtain C.k C Frame images captured by the camera and their corresponding time information k :{[T1,F1],[T2,F2],…,[T d ,F d ]}, where T records the time information of the sampled frame F (this time represents Beijing time, accurate to the second), and d represents the total number of frames. Then, the object detection model is used to sequentially sample the video frames {F1, F2, ..., F}. d Customers were detected in the data, and their cus was obtained. j Object detection bounding box coordinate information po j , Among them, cus j Let represent the j-th customer, where 1 represents the top-left corner coordinate and 2 represents the bottom-right corner coordinate. Since multiple customers may exist in a single frame, the time interval is set to T. d frame F d The set of location information of all customers detected in the process is represented as follows: in This represents the coordinate information of the j-th customer in the d-th frame, meaning there are a total of custd customers with coordinate information in this frame. After obtaining the customer coordinate information, it is bound to the frame and time information, represented as C. k :{[T1,Pos1,F1],[T2,Pos2,F2],…,[T d Pos d ,F d ]}.

[0056] S22: Establish a customer ID set, and save the frame sequence of the same customer based on the customer ID set to form an activity trajectory set: According to the customer location information Pos saved in each frame image F in S21, crop the image and input it into the pedestrian re-identification model PCB. Identify the ID value corresponding to each customer location and store it in the customer ID set. The specific implementation is as follows: The 64*128 customer image sequence obtained by cropping according to the customer location information is sequentially input into the ResNet50 feature extraction network to extract features, resulting in a feature tensor T with a spatial size of 24*8*2048. The PCB horizontally divides tensor T into p parts, and after average pooling, obtains p column vectors {g1, g2, ..., g...}. p Then, using a 1*1 convolution kernel, we obtain p lower-dimensional {h1,h2,…,h} groups, each with a dimension lower than g. p Finally, the p column vectors of type h {h1, h2, ..., h} are... pThe input is fed into a classifier consisting of a fully-connected layer (FC) and a softmax function to obtain the predicted customer ID. During training, the PCB uses the sum of the cross-entropies of p classifiers as the loss function and continuously trains and fine-tunes it. After training, the pedestrian re-identification model PCB is used to predict the customer ID. If the customer does not exist in the customer ID set and does not belong to the staff ID set, it means that the customer is a newly appeared customer, and a new ID is assigned to it and saved to the customer ID set; if the customer's information exists in the customer ID set, its ID is extracted as an identifier for trajectory tracking. Frames containing the same customer ID are arranged in the time order saved in S21 {T1, T2, ..., T...}. d Save as customer's cus j The activity trajectory of C is represented as C. k :{cus j [F′ m ,…,F′ n ]}, where m,n∈{1,2,3,…,d}, cus j F' represents the customer with id value j, and F' represents the customer with cus. j The frame contains coordinate and time information. Let cus j activity trajectory Traversing C k The camera captures different activity trajectories of all customers, which are saved as a customer activity trajectory set, represented as... Where j = 1, ..., allcust, Indicates customer cus j The activity trajectory shows a total of allcust customers.

[0057] S3: Calculate the intersection-union ratio (IoU) of the service area and the customer target detection box under multiple cameras. When the IoU is greater than a certain threshold, the depth map detection model is started. This step is used to confirm whether the customer has arrived at the service area.

[0058] S31: Extract C from the activity trajectory set k Next customer cus j Activity trajectory sequence {F′ m ,…,F′ n}, calculate the customer and service area s in sequence. i The intersection and union (IoU) of the bounding boxes is calculated. If the customer's bounding box is A, and the service area's bounding box is B, then...

[0059]

[0060] Extract the coordinates of the service area and the customer from steps S12 and S21, respectively. and A area ∩B area Form a rectangle S, the length of S is... The width of S Then A∩B = width * length. Based on the values ​​of A, B, and A∩B, the IoU value can be calculated.

[0061] If the IoU is greater than a threshold I, it indicates that the customer's bounding box overlaps with the service area's bounding box. In this case, the depth map detection model is activated, and video frames whose bounding boxes overlap with the service area's bounding box are selected from the customer's activity trajectory. This video frame sequence is then saved as C. k :{cus j [F′ p ,…,F′ q ]},p,q∈{m,…,n}.

[0062] S32: Overlap the video frame sequence {F′} obtained in step S31 with the service area target detection box. p ,…,F′ q The data is fed into the Megadepth depth map detection model to obtain the pixel value of each pixel in the customer and service areas. This pixel value represents the depth of that point from the camera. The set of depth pixels within the customer target detection box is... in Indicates customer cus j The pixel value of the a-th pixel; the set of depth pixels within the target detection bounding box of the service area is in Indicates service area s i The pixel value of the b-th pixel.

[0063] Since the target detection bounding box may contain other interfering pixels besides the currently tracked customer, such as other passersby or background pixels, the absolute median deviation (MAD) will be used to process the pixel values ​​in the customer target detection bounding box. First, the median value P of all pixels will be calculated. median , Next, calculate the absolute deviation of all elements from the median, bias = |Pi - Pmedian|, and obtain the median absolute deviation, MAD = bias. median Then, determine the parameter n and make the following adjustments to all data:

[0064]

[0065] Finally, the optimized set of customer pixel values ​​is obtained as follows: The set of pixel values ​​in the service area is

[0066] S33: Average the optimized customer and service area depth pixel values ​​from step S32, then subtract them to obtain the depth matching value mc. If the depth matching value mc is less than a certain threshold K, it means that the customer has entered the service area.

[0067] S34: In the activity trajectory map set, delete video frames with a depth matching value mc greater than K, indicating customer cus. j The frame did not remain in the service area. The optimized set of activity trajectory maps is represented as C. k : Where f, g ∈ {p, …, q}, s i This indicates the service area where the customer is currently located. When the customer's bounding box does not intersect with the service area and the depth matching value mc is greater than K, it means that the customer is not near this service area or has already left this service area.

[0068] S35: Repeat steps S31 to S34 to continue tracking the customer. When the customer enters the next service area, save the tracking information to the customer's activity trajectory set.

[0069] S4: Use a behavior recognition model to identify the behavior of customers from the time they enter the service area to the time they leave the service area;

[0070] S41: Training the behavior recognition model to detect customer behavior: Extract the coordinate information of the customer's activity trajectory sequence in each service area (S34), and label the customer's behavior frame by frame based on the coordinate information. During training, extract the last 63 labeled frames and input these 64 frames as a video clip into the behavior recognition model for training. The same 64-frame activity trajectory sequence is repeatedly input into two network branches using different sampling frequencies. Training is performed for 100 epochs, with an initial learning rate set to 1.25e-6. The optimization strategy is stochastic gradient descent (SGD), and the network model is optimized using the cross-entropy loss function.

[0071] S42: Establish camera C k The behavior trajectory set is generated by using a trained behavior recognition model, Slowfast, to detect customer behavior in a certain service area and calculate the customer's dwell time in that service area. The obtained behavior data is then saved to the activity trajectory set, forming the behavior trajectory set. The behavior trajectory set contains the data of the cus-th... j A customer in a certain service area i Customer behavior, denoted as C. k :{cus j [(s1,act1),…,(si ,act r )]}, where act1…,act r This represents all possible customer behaviors in the service hall, where r represents the total number of behavior categories. Obtain the specific service area s obtained in step S34. i Next frame sequence This frame sequence was acquired by sampling one frame out of 64 frames, so one frame represents two seconds. The total number of frames was calculated and multiplied by 2, and stored as the customer's stay time t in the service area. The customer cus with added time information... j In C k The behavioral trajectory under the camera is represented as C k :{cus j [(s1,act1,t1),…,(s i ,act r ,t i )]}, where s i Let act1, act2, act3 represent the i-th service area. r t represents all possible customer behaviors. i Indicates that the customer is in s i Stop time at service areas.

[0072] S43: Traverse all cameras {C1,…,C} k ,…,C cam The video data is used to repeat the above process to obtain multiple sets of customer behavior trajectories from multiple cameras, thus forming a customer behavior trajectory database from multiple cameras.

[0073] Furthermore, since this patent involves multiple cameras for behavior recognition, some camera angles may be obstructed, leading to inaccurate recognition. Therefore, this patent considers prioritizing customer behaviors, selecting the behavior with the highest priority, and finally obtaining the customer behavior trajectory fused from multiple cameras.

[0074] S5: Construct a customer behavior trajectory database based on multiple cameras. By aligning time information, duplicate behaviors are deleted, and different behaviors occurring at the same time are selected based on behavior priority to obtain the customer's behavior trajectory.

[0075] S51: Establish a link between the behavioral patterns of a specific customer captured by multiple cameras: Let a specific customer... j In a single camera C k The behavioral trajectory is {(s1,act1,t1),…,(s i ,act r ,t i )} is C k_trackk C k_trackk={(s1,act1,t1),…,(s i ,act r ,t i Based on the customer ID, the behavioral trajectory sets of the same customer under multiple cameras are saved as a behavioral trajectory library, i.e., cus. j : Where C k_trackk This represents the customer's behavioral trajectory under the k-th camera. k .

[0076] S52: Align customer behavior trajectories from multiple cameras using time information: After obtaining the behavior trajectory of the same customer from multiple cameras, query C in S22. k Next customer cus j Information C k :{cus j [F′ m ,…,F′ n ]}, obtain the time when the customer arrives at the service area, and add this time to the customer behavior trajectory to obtain {(T1,s1,act1,t1),…,(T i ,s i ,act r ,t i )}. Customer cus j behavioral trajectory By time sequence {T1,T2,…,T i The comparisons are performed sequentially, and duplicate entries are removed.

[0077] One challenge is selecting the appropriate behavior when different cameras detect inconsistent actions from the same customer at the same time. Since customer actions such as processing transactions and seeking information in public services have higher research value, this patent prioritizes customer behaviors. When different cameras detect different behaviors from the same customer at the same time, this patent selects the highest-priority behavior as the final behavior. The resulting customer behavior trajectory in the service hall is as follows: j :{(T1,s1,act1,t1),…,(T i ,s i ,act r ,t i )}, where (T i ,s i ,act r ,t i ) indicates customer cus j At time T i Arrived at s i The service area has made an action r The behavior, and the time spent there is t.i The specific implementation is as follows: When a customer cus j When the behavior trajectories in Table 1 are shown under cameras C1, C2, and C3:

[0078] Table 1

[0079]

[0080] By comparing the time information sequentially, it was found that at 8:15, the behaviors identified by the second and third cameras were inconsistent. Camera C3 identified a conversation, which differed from the self-service behavior identified by camera C2. Since we set behavior priorities, self-service has a higher priority than conversation, so we selected self-service as the final behavior. After handling the inconsistent behaviors, duplicate behaviors were deleted from the behavior trajectory database. For example, at 8:07, 8:10, and 8:12, the behaviors of the three cameras were consistent, and only one trajectory sequence needed to be retained. Thus, the customer's cus was obtained. j The final behavioral trajectory is {(8:07, temperature detector at the entrance, temperature measurement, 5 seconds), (8:10, information desk, consultation, 2 minutes), (8:12, counter, counter service, 5 minutes), (8:15, self-service machine, self-service service, 5 minutes)}.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for tracking the behavior trajectory of multiple cameras, characterized in that: Includes the following steps: S1: Calibrate multiple cameras and collect the coordinate information of the service area under each camera, and bind the coordinate information to the camera; S2: Combining the object detection model and the pedestrian re-identification model to track customer trajectories and establish a customer activity trajectory database from multiple cameras; step S2 specifically includes the following steps: S21: Locate customers under multiple cameras using an object detection model: Extract frames from the service hall camera footage according to a certain frame rate; use the trained object detection model to sequentially identify the surveillance videos from multiple cameras, sampling at a specified step size. Frames captured by the camera and their corresponding time information ,in Record sampling frame images Time information, This represents the total number of frames; then, the object detection model is used to sequentially detect each frame in the video. Customers were detected and obtained. Target detection bounding box coordinate information , ,in Indicates the first There are 1 customers, where 1 represents the top-left corner coordinate and 2 represents the bottom-right corner coordinate; the time is... frames The set of location information of all customers detected in the process is represented as follows: ,in Indicates the first The first frame The coordinate information of each customer, that is, there are a total of [number] customers in this frame. After obtaining the customer's coordinate information, it is bound to frame and time information, and represented as follows: ; S22: Establish a customer ID set, and save the frame sequence of the same customer based on the customer ID set to form an activity trajectory set: according to each frame image in S21 Customer location information stored in The data is cropped and input into the pedestrian re-identification model PCB. The ID value corresponding to each customer at each location is identified and stored in the customer ID set. If the customer does not exist in the customer ID set and does not belong to the staff ID, it means that the customer is a newly appeared customer. A new ID is assigned to the customer and saved to the customer ID set. If the customer ID set contains the customer's information, the ID is extracted as the identifier for trajectory tracking. Frames containing the same customer ID are arranged according to the time order saved in S21. Saved for this customer The activity trajectory is represented as ,in , This indicates that the id value is Customers, Indicates having customers The frame containing coordinate and time information; activity trajectory traversal The camera captures different activity trajectories of all customers, which are saved as a customer activity trajectory set, represented as... ,in , Indicates customer The activity trajectory, in total One customer; S3: Calculate the Intersection over Union (IoU) between the service area and the customer target detection box under multiple cameras. When the IoU is greater than a certain threshold, use the depth map detection model to confirm whether the customer has arrived at the service area. Step S3 specifically includes the following steps: S31: Extract from the activity trajectory set Customers Activity trajectory sequence Calculate the customer and service area in sequence. The object detection boxes are determined by the intersection and union ratio (IoU); customer object detection box A, service area object detection box B, then... Extract the coordinates of the service area and the customer from steps S12 and S21, respectively ( , ), ( , )and( , ), ( , ), =( - ) ( - ), =( - ) ( - ); ∩ Form a rectangle , length =max{ , }-min{ , }, width =max{ , }-min{ , },but = ,according to The value of IoU is then calculated. If IoU is greater than a threshold If the target bounding box of a customer overlaps with the target bounding box of a service area, the depth map detection model is activated. Video frames whose target bounding boxes overlap with the service area target bounding boxes are then selected from the customer's activity trajectory, and this video frame sequence is saved. ; S32: Overlap the video frame sequence obtained in step S31 with the service area target detection box. The data is fed into the Megadepth depth map detection model to obtain the pixel value of each pixel in the customer and service areas; this pixel value represents the depth of that point from the camera; the set of depth pixels within the customer target detection box is { , , …, },in Indicates customer The pixel value of the a-th pixel; the set of depth pixels within the target detection bounding box of the service area is { , , ,…, },in Indicates service area The pixel value of the b-th pixel; The data was processed using the absolute median deviation (MAD), resulting in the optimized set of customer pixel values ​​as { , , …, The set of pixel values ​​in the service area is { , , ,…, }; S33: Average the optimized customer and service area depth pixel values ​​from step S32, then subtract them to obtain the depth matching value. , If the depth matching value Less than a certain threshold This indicates that the customer has entered the service area; S34: Delete depth matching values ​​from the activity trajectory map set. Greater than The video frames indicate the customer The frame did not remain in the service area; the optimized activity trajectory map set is represented as follows. ,in , This indicates the service area where the customer is currently located. When the customer's target bounding box has no intersection with the service area and the depth matching value mc is greater than K, it means that the customer is not near this service area or has already left this service area. S35: Repeat steps S31-S34 to continue tracking the customer. When the customer enters the next service area, save the tracking information to the customer's activity trajectory set. S4: Use a behavior recognition model to identify the customer's behavior from entering the service area to leaving the service area; Step S4 specifically includes the following steps: S41: Train the behavior recognition model to detect customer behavior: Extract the coordinate information of the activity trajectory sequence of customers in each service area in S34, and label the customer behavior frame by frame according to the coordinate information; during training, extract the last 63 frames of the labeled frames, and input the 64 frames as a video set into the behavior recognition model for training; input the same 64-frame activity trajectory sequence in two network branches in a loop with different sampling frequencies. S42: Set up a camera The behavior trajectory set is generated by using a trained behavior recognition model, Slowfast, to detect customer behavior in a certain service area and calculate the customer's dwell time in that service area. The obtained behavior data is saved to the activity trajectory set, thus forming a behavior trajectory set. The behavior trajectory set contains the first... A customer in a certain service area Customer behavior , represented as ,in This indicates all possible customer behaviors in the service lobby. Indicates the total number of behavior categories; retrieves the specific service area obtained in step S34. next frame sequence { The frame sequence was obtained by sampling one frame out of 64 frames, so one frame represents two seconds. The total number of frames was calculated and multiplied by 2 to store the customer's stay time in the service area. Customers who add time information exist The behavior trajectory under the camera is represented as ,in Indicates the first One service area, This represents all possible customer behaviors. Indicates that the customer is Service area dwell time; S43: Traverse all cameras The video data is used to repeat steps S41-S43 to obtain multiple sets of customer behavior trajectories under multiple cameras, thereby forming a customer behavior trajectory library under multiple cameras. S5: Construct a customer behavior trajectory database under multiple cameras. By aligning time information, duplicate behaviors are deleted. Different behaviors appearing under different cameras at the same time are selected according to priority to obtain the customer's behavior trajectory. Step S5, constructing a customer behavior trajectory database under multiple cameras, specifically includes the following steps: S51: Establish a link between the behavioral patterns of a customer observed by multiple cameras: [This involves] setting a link between the behavioral patterns of a specific customer and [other individuals / cameras]. In a single camera The following behavioral trajectory for ,Right now = Based on the customer ID, the behavioral trajectory sets of the same customer under multiple cameras are saved as a behavioral trajectory database. : ,in This indicates that the customer was on the [date / time]. Behavioral trajectory under a camera ; S52: Align customer behavior trajectories from multiple cameras using time information: After obtaining the behavior trajectory of the same customer from multiple cameras, query the data in S22. Customers Information : The time when the customer arrives at the service area is obtained, and this time is added to the customer's behavior trajectory. ; customers behavioral trajectory By time sequence The comparisons are performed sequentially, and duplicate behaviors are removed. When a customer is identified by different cameras at the same time, the behavior with the highest priority is selected as the final behavior.

2. The multi-camera behavior trajectory tracking method according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Obtain footage from multiple cameras in the public service hall, specify the corresponding camera and mark it. , ,in Indicates the first There are a total of [number] cameras. Two different cameras; S12: Use an object detection model to obtain the coordinate information of the public service hall service area from multiple cameras: After labeling and organizing the collected image information, it is fed into the model for training to obtain an object detection model for recognizing the service area; according to different cameras... The obtained image information from different perspectives is sequentially processed by a trained object detection model to calculate the coordinate information of the service area under multiple cameras. The coordinate information includes the indices of the upper left and lower right corners of the object detection boxes, which are represented as ( , ), ( , ),in Indicates the first One service area, There are a total of Each service area is represented by a coordinate of 1 (top left) and 2 (bottom right). The coordinate information obtained from the camera is represented as :{ ( , ), ( , )}.

3. The multi-camera behavior trajectory tracking method according to claim 1, characterized in that: In step S32, the pixel values ​​in the customer target detection box are processed using the absolute median deviation (MAD). First, the median value of all pixels is calculated. , Then calculate the absolute deviation of all elements from the median. Obtain the median value of the absolute deviation. Then determine the parameters. Make the following adjustments to all data: Finally, the optimized set of customer pixel values ​​is obtained as follows: The set of pixel values ​​in the service area is .

4. The multi-camera behavior trajectory tracking method according to claim 1, characterized in that: In step S52, customer behaviors are prioritized. When different cameras detect different behaviors from the same customer at the same time, the behavior with the highest priority is selected as the final behavior. The final customer behavior trajectory in the service hall is as follows: : ,in Indicates customer In time Arrived The service area did The behavior, and the duration of stay. .

Images