Method and system for tracking people trajectories

Abstract

The application provides a personnel trajectory tracking method and system, and belongs to the technical field of personnel trajectory tracking. The method comprises the following steps: acquiring first data and second data; processing the first data to obtain a personnel bounding box set in the first data; processing the second data to generate a second coordinate system; and obtaining a personnel point cloud cluster set in the second coordinate system; determining a target personnel set based on the personnel bounding box set and the personnel point cloud cluster set through joint confidence calculation; calculating the current three-dimensional position of each target personnel in the second coordinate system as a current observation node; constructing a space-time association graph based on the current observation node and a historical trajectory node of the target personnel; and tracking the continuous motion trajectory of the target personnel based on the space-time association graph. The application can realize efficient alignment of personnel targets across modalities and continuous trajectory tracking.

Description

Technical Field

[0001] This application belongs to the field of personnel trajectory tracking technology, and more specifically, relates to a personnel trajectory tracking method and system. Background Technology

[0002] With the development of artificial intelligence and sensing technology, personnel trajectory tracking technology based on visual and radar sensors has shown broad application prospects in fields such as security monitoring, intelligent transportation, and human-computer interaction. By simultaneously collecting multiple types of data in the same scene, such as 2D images, depth images, and point clouds, the advantages of different modalities of data can be comprehensively utilized to improve the accuracy and robustness of environmental perception. However, in existing technologies, personnel detection is usually performed independently in image space and 3D space, and then associated through simple projection or matching strategies. This approach struggles to handle complex scenarios such as occlusion, changes in lighting, and dense crowds, leading to problems such as false detections, missed detections, and trajectory jumps.

[0003] Therefore, how to achieve efficient alignment and continuous trajectory tracking of personnel across modalities has become a key issue in improving the performance of intelligent monitoring systems. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for tracking personnel trajectories, so as to achieve efficient alignment of personnel targets across modalities and continuous trajectory tracking.

[0005] A first aspect of this application provides a method for tracking personnel trajectories, including:

[0006] Acquire first data and second data; the first data and second data are data synchronously acquired from the same scene being acquired; the first data is two-dimensional image data of the scene being acquired; the second data is three-dimensional point cloud data of the scene being acquired.

[0007] The first data is processed to obtain a set of personnel bounding boxes in the first data; the set of personnel bounding boxes contains at least one personnel bounding box.

[0008] The second data is processed to generate a second coordinate system; and a personnel point cloud cluster set is obtained in the second coordinate system; the personnel point cloud cluster set contains at least one personnel point cloud cluster.

[0009] Based on the set of personnel bounding boxes and the set of personnel point cloud clusters, the target personnel set is determined by joint confidence calculation; the target personnel set contains at least one target personnel.

[0010] Calculate the current three-dimensional position of each target person in the second coordinate system, and use it as the current observation node;

[0011] Based on the current observation nodes and the historical trajectory nodes of the target personnel, a spatiotemporal correlation graph is constructed;

[0012] Based on the spatiotemporal correlation graph, the continuous movement trajectory of the target personnel is tracked.

[0013] A second aspect of this application provides a personnel trajectory tracking system, comprising:

[0014] Data acquisition module: used to acquire first data and second data; the first data and second data are data synchronously acquired from the same scene being acquired; the first data is two-dimensional image data of the scene being acquired; the second data is three-dimensional point cloud data of the scene being acquired.

[0015] First data processing module: used to process the first data and obtain a set of personnel bounding boxes in the first data; the set of personnel bounding boxes contains at least one personnel bounding box;

[0016] The second data processing module is used to process the second data, generate a second coordinate system, and obtain a personnel point cloud cluster set in the second coordinate system; the personnel point cloud cluster set contains at least one personnel point cloud cluster.

[0017] Target Personnel Determination Module: This module determines the target personnel set based on the personnel bounding box set and the personnel point cloud cluster set, through joint confidence calculation; the target personnel set contains at least one target personnel.

[0018] Node determination module: used to calculate the current three-dimensional position of each target person in the second coordinate system, as the current observation node;

[0019] Spatiotemporal correlation graph determination module: used to construct a spatiotemporal correlation graph based on the current observation node and the historical trajectory nodes of the target person;

[0020] Trajectory tracking module: used to track the continuous movement trajectory of a target person based on a spatiotemporal correlation graph.

[0021] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the personnel trajectory tracking method described above.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the personnel trajectory tracking method described above.

[0023] A fifth aspect of this application provides a computer program product, including a computer program or computer-executable instructions, wherein when the computer program or computer-executable instructions are executed by a processor, the steps of the above-described personnel trajectory tracking method are implemented.

[0024] The beneficial effects of the personnel trajectory tracking method and system provided in this application are as follows:

[0025] This application embodiment provides rich texture, color, and appearance details through two-dimensional image data, facilitating high-precision personnel identification; three-dimensional point cloud data provides accurate distance, depth, and geometric shape information, unaffected by lighting changes, and effectively penetrates partial occlusion. Synchronous acquisition ensures that images and point cloud data from the same scene state at the same time, eliminating target position misalignment caused by asynchronous acquisition time, and providing a guarantee for subsequent accurate association. This application embodiment helps distinguish between people of different identities by identifying personnel in image space and determining their approximate two-dimensional position. Simultaneously, each point cloud cluster directly corresponds to a three-dimensional spatial entity of an object, providing absolute or relative three-dimensional position, volume, orientation, and other measurement information of the target. Furthermore, point cloud processing is insensitive to lighting changes, effectively separating potentially occluded or densely packed crowds on the image based on height and depth. This application embodiment performs decision-level fusion through joint confidence calculation. By comprehensively evaluating the reliability of two-dimensional detection results, the reliability of three-dimensional clustering results, and the degree of matching between the two, false positives of a single modality can be effectively eliminated, and false negatives of a single modality can be supplemented. Ultimately, a cleaner and more accurate set of target personnel is output, directly reducing the false positive and false negative rates of the entire system from the root cause. This embodiment abstracts the location of target personnel as observation nodes, facilitating subsequent unified spatiotemporal correlation analysis with historical trajectory nodes, laying a data foundation for continuous trajectory tracking. This embodiment also associates current observation nodes with historical nodes through a graph structure, considering both spatial positional continuity and temporal sequence correlation, effectively handling trajectory breakage problems in scenarios such as short-term occlusion and rapid movement. Utilizing the correlation relationships between nodes in the spatiotemporal correlation graph, the movement trajectory of the same target can be continuously correlated in complex scenes, reducing identity switching errors and ultimately achieving efficient cross-modal personnel target alignment and continuous trajectory tracking. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1A flowchart illustrating a personnel trajectory tracking method provided in an embodiment of this application;

[0028] Figure 2 This is a structural block diagram of a personnel trajectory tracking system provided in an embodiment of this application;

[0029] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0032] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a personnel trajectory tracking method provided in an embodiment of this application. The method can be executed by an electronic device, specifically a computer, server, or similar device. The method may include:

[0033] S101: Acquire first data and second data; the first data and second data are data synchronously acquired from the same scene being acquired; the first data is two-dimensional image data of the scene being acquired; the second data is three-dimensional point cloud data of the scene being acquired.

[0034] In this embodiment, the first data refers to the two-dimensional image data of the scene being collected. It captures the appearance features, planar positions, and visual texture information of people in the scene, forming the basis for subsequent identification of people's appearance and planar contours. The second data is the three-dimensional point cloud data of the scene being collected. By recording the three-dimensional coordinate information of a large number of points on the surface of objects in the scene, it can present the spatial structure of the scene, the three-dimensional position of people, and depth relationships. "Same collected scene" means that the physical spatial range targeted by both types of data is completely consistent, ensuring that the data reflects information from the same environment. "Synchronous acquisition" means that the processes of acquiring the first and second data are highly consistent in time, avoiding information misalignment caused by time differences. Two-dimensional image data is visual information stored in the form of a planar pixel array. Each pixel contains color or grayscale values, which can intuitively display the visual details of the scene. Three-dimensional point cloud data is a dataset composed of a large number of three-dimensional coordinate points. Each point contains spatial position information, which can construct a three-dimensional model of the scene.

[0035] In this embodiment, when performing the steps of acquiring the first and second data, it is necessary to first select an appropriate acquisition device according to the scale and requirements of the scene being collected. The device for acquiring two-dimensional image data can be an RGB camera, while the device for acquiring three-dimensional point cloud data can be a LiDAR or a depth camera. To achieve synchronous acquisition, the acquisition frequency and start-up time of the two devices can be bound by hardware triggering, or software timestamp alignment technology can be used to associate each frame of the acquired two-dimensional image with the corresponding three-dimensional point cloud data through a unified time stamp. Simultaneously, it is necessary to ensure that the acquisition fields of both devices cover the same scene being collected, avoiding data range mismatch caused by field-of-view deviations. Furthermore, the device position and posture must be kept stable during the acquisition process to prevent spatial consistency of the data from being affected by device shaking. The final acquired two-dimensional image data must have clear human outlines and textures, and the three-dimensional point cloud data must ensure uniform point cloud density and low noise points, providing a reliable foundation for subsequent data processing.

[0036] For example, taking the tracking of people's trajectories in the lobby of a large shopping mall as an example, the operation of acquiring first and second data is performed in this scenario. A 4K resolution RGB camera is installed at a key position at the top of the lobby, with the camera lens covering the entire lobby area to capture two-dimensional image data including customers, staff, and other personnel. At the same time, a 16-line LiDAR is installed at the same position, with the scanning range of the LiDAR completely overlapping the field of view of the RGB camera, to collect three-dimensional point cloud data of the lobby. Through a synchronous controller connecting the camera and LiDAR, the acquisition frequency of both is set to 30 frames / second. The controller sends a synchronous trigger signal to ensure that both devices start acquiring each frame of data at the same time. After the data acquisition is completed, it is sent to the processing terminal through the data transmission module. The terminal further confirms the time consistency between each frame of two-dimensional image and the corresponding three-dimensional point cloud data by parsing the timestamps on the devices. The finally acquired two-dimensional images can clearly show the external features and planar positions of people in the lobby, while the three-dimensional point cloud data accurately records the height of people, their distance from surrounding shelves, and their three-dimensional coordinates in the lobby, providing the required raw data for subsequent extraction of personnel bounding boxes and point cloud clustering.

[0037] S102: Process the first data to obtain a set of personnel bounding boxes in the first data; the set of personnel bounding boxes contains at least one personnel bounding box.

[0038] In this embodiment, a person bounding box refers to a rectangular region in a two-dimensional image used to define a single person target. This region determines the person's planar position and approximate range in the image. The set of person bounding boxes is a whole composed of all detected person bounding boxes in the image, with each bounding box corresponding to an independent person. The size of the set is adjusted according to the number of people in the image.

[0039] In this embodiment, when processing the first data to obtain a set of personnel bounding boxes, the two-dimensional image data can also be preprocessed. Gaussian filtering is used to remove image noise, and histogram equalization is used to enhance image contrast, providing a clearer image foundation for subsequent detection. The preprocessed image can be analyzed using object detection algorithms, such as the YOLO series and Faster R-CNN. This allows for the identification of personnel features in the image using deep learning models, automatically generating rectangular bounding boxes that can surround the personnel.

[0040] Furthermore, the generated bounding boxes can be post-processed by using a non-maximum suppression algorithm to remove redundant bounding boxes with high overlap, while filtering out suspected false detection bounding boxes with confidence scores below a preset threshold. Finally, all the filtered valid personnel bounding boxes are integrated to form a personnel bounding box set.

[0041] For example, taking the scenario of tracking people's movements in an office building lobby as an example, the collected two-dimensional image data of the lobby is processed. First, the images from the morning rush hour are preprocessed to remove noise points caused by glass reflections, making the outlines of people entering and exiting the lobby clearer. Next, the YOLOv8 object detection algorithm is used to scan the processed images. The algorithm generates rectangular bounding boxes for each office worker by recognizing features such as the head and torso, and outputs the coordinate information of each bounding box. Then, a non-maximum suppression program is run to delete duplicate bounding boxes caused by people turning sideways or slight overlap, and to remove bounding boxes with a confidence score below 0.6. Finally, a set of personnel bounding boxes containing 12 valid bounding boxes is obtained, each bounding box corresponding to a different person in the lobby who is waiting for the elevator, registering, or walking.

[0042] S103: Process the second data to generate a second coordinate system; and obtain the personnel point cloud cluster set in the second coordinate system; the personnel point cloud cluster set contains at least one personnel point cloud cluster.

[0043] In this embodiment, the second coordinate system is a unified spatial reference constructed for 3D point cloud data. It standardizes the spatial positioning of point cloud data, ensuring that point cloud information collected at different times and locations can be analyzed and correlated within the same framework. Personnel point cloud clustering refers to the aggregation of all spatial points belonging to the same person from 3D point cloud data into an independent group of point clouds, reflecting the person's three-dimensional outline and spatial position. The personnel point cloud cluster set is a whole composed of all detected personnel point cloud clusters in the scene, with each cluster corresponding to an independent personnel target.

[0044] In this embodiment, processing the second data may include preprocessing the 3D point cloud to reduce interference from irrelevant points in subsequent processing. In the second coordinate system generation stage, external parameters of acquisition devices such as LiDAR are first obtained through device calibration. Using fixed landmarks in the scene, such as corners or pillars, as the origin, coordinate axes are set along the length, width, and height of the scene to establish a global coordinate system. Then, the preprocessed point cloud data is mapped to this coordinate system through coordinate transformation. When obtaining the personnel point cloud cluster set, the density-based DBSCAN clustering algorithm can be used, setting a reasonable neighborhood radius and a minimum point threshold to perform cluster analysis on personnel points in non-ground areas. Furthermore, post-processing steps such as cluster size filtering and cluster shape verification can be used to ultimately integrate effective clusters to form a personnel point cloud cluster set.

[0045] For example, taking the scenario of tracking people's trajectories in a school cafeteria as an example, the collected 3D point cloud data of the cafeteria is processed. First, statistical filtering is used to remove noise points caused by air scattering and equipment vibration. The RANSAC algorithm is used to separate the point cloud of the cafeteria floor, retaining only non-ground points such as tables, chairs, and people. When generating the second coordinate system, the origin is set at the left corner of the cafeteria's main entrance, the x-axis is set along the length of the cafeteria, the y-axis is set along the width, and the z-axis is set perpendicular to the ground, establishing a global coordinate system for the cafeteria. The processed point cloud data is then transformed into this coordinate system. In the clustering stage, the DBSCAN algorithm is used to cluster the non-ground points, and then the non-personnel clusters are removed, retaining the clusters with a height between 1.2 meters and 2 meters that conform to human height characteristics. Finally, a set containing 8 personnel point cloud clusters is obtained, with each cluster corresponding to different students queuing for food or sitting down to eat in the cafeteria.

[0046] S104: Based on the set of personnel bounding boxes and the set of personnel point cloud clusters, determine the target personnel set through joint confidence calculation; the target personnel set contains at least one target personnel.

[0047] In this embodiment, joint confidence is a credibility index calculated by combining two-dimensional detection, three-dimensional clustering, and the degree of matching between the two. Its core function is to determine whether a person's bounding box and a person's point cloud cluster correspond to the same real person. The target person set is the set of people identified for specific tracking after being filtered by joint confidence. It includes all independent people who meet the tracking criteria and is the specific object of subsequent trajectory tracking. The target person set is a whole composed of all target people that need to be tracked, and only includes people who meet the preset tracking conditions.

[0048] In this embodiment, firstly, the bounding boxes in the set of all personnel bounding boxes are mapped to each cluster in the set of personnel point cloud clusters using a first coordinate system and a second coordinate system, generating all possible candidate pairs to ensure that no potential real personnel in the scene are missed. Next, for each candidate pair, three key parameters are calculated: 2D detection confidence, 3D clustering confidence, and matching score. These three parameters are then weighted and summed to obtain the joint confidence of each candidate pair. Based on a preset joint confidence threshold, candidate pairs with confidence scores higher than the threshold are selected. Each grid pair corresponds to one real person. Integrating these pairs forms the target personnel set.

[0049] For example, taking the scenario of tracking people's trajectories at the entrance and exit of a shopping mall as an example, first, a set of bounding boxes for people in this area is obtained, containing 6 rectangles corresponding to customers entering and exiting; a set of point cloud clusters for people is obtained, containing 5 3D point cloud groups corresponding to customers. First, each bounding box and each point cloud cluster are fused according to the coordinate system to map their corresponding positions. For one pair, the joint confidence of the pair is calculated to be 0.893. The joint confidence threshold is set to 0.85, and 3 valid pairs with confidence higher than the threshold are selected from the candidate pairs. The people corresponding to these 3 pairs are directly integrated to form a set containing 3 target people. Subsequent trajectory tracking is then carried out for these 3 target people.

[0050] S105: Calculate the current three-dimensional position of each target person in the second coordinate system as the current observation node.

[0051] In this embodiment, the current three-dimensional position refers to the specific spatial coordinates of the target person in the second coordinate system, which can accurately reflect their real-time three-dimensional orientation in the scene. The current observation node is an observation unit built with the target person's current three-dimensional position as the core. It is used to record the target person's spatial position information at that moment and is the basic unit for subsequently constructing the spatiotemporal correlation graph.

[0052] In this embodiment, the specific spatial coordinates of each target person in the second coordinate system are calculated to obtain the current three-dimensional position of the target person in the second coordinate system. This three-dimensional coordinate is defined as the current observation node of the target person, and the timestamp corresponding to the node is recorded synchronously to provide a dual temporal and spatial reference for subsequent association with historical trajectory nodes.

[0053] For example, taking the scenario of tracking people's trajectories in a hospital outpatient hall as an example, the current 3D position of two identified target individuals is calculated: a patient and a triage nurse. First, the point cloud clusters corresponding to the patient and the nurse are retrieved. Based on these two clusters, the current 3D position of the patient is (15.2, 8.7, 1.6), and the current 3D position of the nurse is (14.9, 9.1, 1.5). Then, these two 3D coordinates are used as the current observation nodes of the patient and the nurse, respectively, and the acquisition time is marked to prepare for the subsequent construction of a spatiotemporal correlation graph.

[0054] S106: Construct a spatiotemporal correlation graph based on the current observation node and the historical trajectory nodes of the target person.

[0055] In this embodiment, historical trajectory nodes refer to observation nodes that have been successfully tracked and verified as valid before the current moment for the target person. Each node contains the three-dimensional position, precise timestamp, movement speed, and unique identifier at that moment, and must undergo spatiotemporal consistency verification to exclude abnormal or false detection nodes, retaining only valid nodes that reflect the true movement state. Historical trajectory nodes can be nodes recorded in the previous frame. The spatiotemporal correlation graph is a graph structure with nodes and edges as its core, where nodes represent the current observation node and historical trajectory nodes of the target person, and edges are used to connect different nodes of the same target person, intuitively reflecting the temporal sequence and spatial location association between nodes.

[0056] In this embodiment, the previous valid observation nodes from the previous frame can be selected, and the specific number of frames can be adjusted according to the tracking accuracy requirements. The current observation node and historical trajectory nodes of the target person are used as nodes in the graph, and each node carries information in a uniform format. Then, edges are established between nodes with the same identifier, and the connection of the edges must satisfy temporal continuity and spatial rationality to ensure that the association relationship conforms to the actual movement pattern of the target person.

[0057] For example, a spatiotemporal relationship graph is constructed using a passenger with the subway platform identifier ID003. First, the passenger's historical trajectory nodes are filtered: valid nodes from the frame preceding the current time are selected. These two nodes are used as nodes, and an edge is established between them to form a spatiotemporal relationship graph containing continuous and valid nodes and associated edges for the passenger, without redundant or abnormal information.

[0058] S107: Based on the spatiotemporal correlation graph, track the continuous movement trajectory of the target personnel.

[0059] In this embodiment, a continuous motion trajectory refers to the complete movement path of a target person over a period of time, formed by multiple continuous and effective observation nodes connected in chronological order. It includes both continuous changes in spatial location and orderly connection of time sequence, and can fully reflect the movement process of the target person.

[0060] In this embodiment, when tracking continuous motion trajectories based on a spatiotemporal correlation graph, the first step is to retrieve the corresponding spatiotemporal correlation graph using the unique identifier of the target person as an index, confirming that the timestamps, 3D positions, and edge weights of the nodes in the graph are complete and valid. Then, all nodes in the correlation graph are sorted in ascending order of timestamps, ensuring that the node sequence strictly follows chronological order. Next, based on the edge connections in the correlation graph, the sorted nodes are sequentially connected. Combining this with the 3D structural information of the scene, the connection trajectory is verified to conform to physical space movement rules, such as avoiding unreasonable paths like passing through walls or obstacles. After confirmation, the node sequence and corresponding time information are integrated into a structured continuous motion trajectory, forming trajectory data that can be directly output or analyzed.

[0061] For example, taking visitor ID028 in the interactive exhibition hall of the science museum as an example, the spatiotemporal relationship graph used for tracking has been confirmed to be complete and valid, containing 6 nodes from t1 to t6 and 5 related edges. First, the relationship graph is retrieved, and the nodes are sorted by timestamp: t1 (10.3, 6.2, 1.6), t2 (10.8, 6.5, 1.6), t3 (11.2, 6.8, 1.6), t4 (11.5, 7.1, 1.6), t5 (11.2, 7.4, 1.6), t6 (10.9, 7.7, 1.6). Based on the established edge connection relationships of t1-t2, t2-t3, t3-t4, t4-t5, and t5-t6 in the graph, the nodes are directly connected in sequence. Verification combined with the exhibition hall layout shows that the trajectory moves continuously along the passage from the mechanics exhibition area to the acoustics exhibition area without spatial conflict. The final output of the continuous motion trajectory clearly presents the complete path of the visitor from stopping at the mechanics display case, to experiencing the acoustic interactive device, and then moving towards the exit. The trajectory data is generated entirely based on the nodes and relationships of the original spatiotemporal relationship graph.

[0062] As can be seen from the above, the embodiments of this application provide rich texture, color, and appearance details through two-dimensional image data, facilitating high-precision personnel identification; three-dimensional point cloud data provides accurate distance, depth, and geometric shape information, is unaffected by changes in illumination, and can effectively penetrate partial occlusion. Synchronous acquisition ensures that the image and point cloud data at the same time come from the same scene state, eliminating the target position misalignment problem caused by asynchronous acquisition time, and providing a guarantee for subsequent accurate association. The embodiments of this application help distinguish between personnel of different identities by identifying personnel in image space and determining their rough two-dimensional positions. At the same time, each point cloud cluster directly corresponds to a three-dimensional spatial entity of an object, providing measurement information such as the absolute or relative three-dimensional position, volume, and orientation of the target. Furthermore, point cloud processing is insensitive to changes in illumination, and it can effectively separate potentially occluded or densely packed crowds on the image based on height and depth. The embodiments of this application perform decision-level fusion through joint confidence calculation. By comprehensively evaluating the reliability of two-dimensional detection results, the reliability of three-dimensional clustering results, and the degree of matching between the two, false positives of a single modality can be effectively eliminated, and false negatives of a single modality can be supplemented. Ultimately, a cleaner and more accurate set of target personnel is output, directly reducing the false positive and false negative rates of the entire system from the root cause. This embodiment abstracts the location of target personnel as observation nodes, facilitating subsequent unified spatiotemporal correlation analysis with historical trajectory nodes, laying a data foundation for continuous trajectory tracking. This embodiment also associates current observation nodes with historical nodes through a graph structure, considering both spatial positional continuity and temporal sequence correlation, effectively handling trajectory breakage problems in scenarios such as short-term occlusion and rapid movement. Utilizing the correlation relationships between nodes in the spatiotemporal correlation graph, the movement trajectory of the same target can be continuously correlated in complex scenes, reducing identity switching errors and ultimately achieving efficient cross-modal personnel target alignment and continuous trajectory tracking.

[0063] In one embodiment of this application, a target set of people is determined by joint confidence calculation based on a set of people bounding boxes and a set of people point cloud clusters, including:

[0064] Pair each person's bounding box with each person's point cloud cluster to form candidate pairs;

[0065] For each candidate pair, calculate the two-dimensional detection confidence, three-dimensional clustering confidence, and matching score;

[0066] The joint confidence score is obtained by weighting the two-dimensional detection confidence score, the three-dimensional clustering confidence score, and the matching score.

[0067] The target personnel set is determined based on joint confidence levels.

[0068] In this embodiment, the 2D detection confidence score is a credibility index based on the appearance features of a person in a 2D image, such as clothing style, carried items, and hairstyle, to determine whether the person within the bounding box is a specific target person. The 3D clustering confidence score is a credibility index based on the 3D features of a person in 3D point cloud data, such as height, body shape, and the volume of carried items, to determine whether the person corresponding to the point cloud cluster is a specific target person. The matching score is a quantitative value that measures the degree of overlap between the bounding box and cluster pairings in spatial location, used to help verify whether the pairing corresponds to the same specific target person. The joint confidence score is a comprehensive credibility index obtained by weighting the 2D detection confidence score, 3D clustering confidence score, and matching score.

[0069] In this embodiment, the set of personnel bounding boxes and the set of personnel point cloud clusters are traversed, and each bounding box is mapped to each cluster to generate candidate pairs, ensuring that no pair that may correspond to a specific target personnel is missed. For each candidate pair, three-dimensional indicators are calculated: 2D detection confidence is obtained by comparing the appearance features of the personnel within the bounding box with preset target features; 3D clustering confidence is obtained by comparing the 3D features of the point cloud cluster with preset target features; and matching score is obtained by projecting the point cloud cluster onto a 2D image and calculating the intersection-union ratio (IUU) of the projected area and the bounding box. Weights are set according to the recognition priority of specific targets, and joint confidence is calculated by weighted summation. Specifically, one way to set the weights is as follows: if appearance features are more critical for target differentiation, the 2D detection confidence weight is set to 0.5, the 3D clustering confidence weight is set to 0.3, and the matching score weight is set to 0.2; if 3D features are more difficult to forge, the 3D clustering confidence weight is set to 0.5, the 2D detection confidence weight is set to 0.3, and the matching score weight is set to 0.2, ensuring that the sum of the weights is 1. Finally, by setting a joint confidence threshold, candidate pairs with confidence levels higher than the threshold are selected, and duplicate or low-confidence non-target pairs are eliminated. The final result is a target set that includes only specific target individuals.

[0070] Furthermore, the weight values ​​can be dynamically adjusted based on environmental or weather factors. For example, the confidence weight of 3D clustering can be increased in severely occluded scenarios, while the confidence weight of 2D detection can be increased in scenarios with stable lighting, ensuring that the comprehensive evaluation results are highly consistent with actual tracking needs and improving adaptability to complex scenarios.

[0071] For example, taking the tracking of a supermarket clerk wearing blue overalls and pushing a silver shopping cart as an example, the personnel bounding box set in this scenario contains 5 bounding boxes, and the personnel point cloud cluster set contains 4 clusters. First, the 5 bounding boxes and 4 clusters are mapped to each other using the positional correspondence between the first coordinate system and the second coordinate system, generating 5 candidate pairs. For one candidate pair: the 2D detection confidence score is obtained by comparing the blue overalls feature of the person within the bounding box and the absence of a shopping cart, resulting in a score of 0.75; the 3D cluster confidence score is obtained by comparing the point cloud features corresponding to the height of 1.72 meters and the absence of a shopping cart in the cluster, resulting in a score of 0.7; the matching score calculates the IoU between the cluster projection and the bounding box, which is 0.78, resulting in a score of 0.78; the third step, because... The 3D features of the shopping cart can better distinguish between stock clerks and ordinary customers. We set a 3D clustering confidence weight of 0.4, a 2D detection confidence weight of 0.3, and a matching score weight of 0.3, calculating the joint confidence score as 0.75×0.3+0.7×0.4+0.78×0.3=0.734. With a threshold of 0.8, we continued to filter other pairs, eventually finding one pair with a joint confidence score of 0.88. This pair was confirmed to correspond to the stock clerk to be tracked, forming a target personnel set containing only one target person.

[0072] In this embodiment, the number of personnel bounding box sets and personnel point cloud cluster sets are not necessarily equal. This characteristic is not a technical defect, but rather provides key support for reducing false matching and missed matching. The essence of the difference in the number of the two sets is due to the difference in modal characteristics between two-dimensional images and three-dimensional point clouds: two-dimensional images are easily affected by illumination, occlusion, and background interference, which may lead to duplicate detection or false detection, resulting in a larger number of bounding boxes; while three-dimensional point clouds can penetrate some occlusion and accurately reflect spatial entities, when the point cloud is sparse or the device resolution is insufficient, cluster merging or cluster loss may occur, resulting in a smaller number of clusters. This quantitative asymmetry creates a natural cross-validation relationship. When the number of bounding boxes exceeds the number of clusters, the uniqueness of entities in 3D clustering can be used to filter out 2D false detection boxes, and the 3D detection confidence and / or 3D detection confidence weight can be set to 0 accordingly. When the number of clusters exceeds the number of bounding boxes, the appearance specificity of 2D bounding boxes can be used to supplement 3D missed targets, and the 2D detection confidence and / or 2D detection confidence weight can be set to 0 accordingly, thereby reducing the risk of mismatches due to independent judgment of a single modality from the data level.

[0073] As can be seen from the above, this embodiment generates all candidate pairs by traversing the set of personnel bounding boxes and the set of personnel point cloud clusters. This can cover all potential correspondences when the number of the two sets is unequal, avoiding the omission of specific target personnel due to the omission of possible pairing combinations. This provides a complete data foundation for subsequent accurate screening, ensuring that all eligible target personnel can enter the evaluation scope. For each candidate pair, the two-dimensional detection confidence, three-dimensional clustering confidence, and matching score are calculated to quantitatively verify the authenticity of the pairing from three independent dimensions. This multi-dimensional evaluation breaks the limitations of single-modal judgment, avoiding misjudgments caused by illumination and occlusion in two-dimensional images, and compensating for feature loss caused by sparsity in three-dimensional point clouds, significantly improving the accuracy of identifying specific target personnel. This embodiment sets weights according to the recognition priority of specific targets, or dynamically adjusts weights in combination with environmental and weather factors, so that the joint confidence calculation can accurately adapt to different scenario requirements. Setting a joint confidence threshold and filtering high-confidence pairs can effectively eliminate low-confidence non-target pairs, ensuring that the target personnel set only includes personnel who meet specific tracking needs, thus laying a reliable foundation for subsequent trajectory tracking.

[0074] In one embodiment of this application, calculating the current three-dimensional position of each target person in the second coordinate system includes:

[0075] Obtain point cloud clusters of personnel corresponding to the target personnel;

[0076] The point cloud clusters of personnel are simplified to obtain simplified point cloud clusters.

[0077] Calculate the three-dimensional geometric centroid of the simplified point cloud cluster;

[0078] The coordinates of the three-dimensional geometric centroid are used as the current three-dimensional position of the target personnel in the second coordinate system.

[0079] In this embodiment, simplification refers to the simplification operation performed on the personnel point cloud cluster. By removing redundant points, noise points, and non-critical feature points, the data volume is reduced while retaining the core spatial features of the point cloud. The simplified point cloud cluster is the set of point clouds obtained after simplification, which retains the three-dimensional outline and spatial distribution features of the original personnel point cloud cluster while reducing data complexity. The three-dimensional geometric centroid refers to the arithmetic mean of the three-dimensional coordinates of all points in the simplified point cloud cluster. It can centrally reflect the overall spatial position of the point cloud cluster and can serve as a representative position of the personnel in three-dimensional space.

[0080] In this embodiment, a statistical filtering algorithm can be used to identify and remove noise points that deviate from the main distribution of the point cloud. Through voxel downsampling technology, representative points are retained at a preset voxel size while maintaining the overall structure of the point cloud, reducing the number of points to 30%-50% of the original number, forming a simplified point cloud cluster. The arithmetic mean of the x, y, and z coordinates of all points in the simplified point cloud cluster is calculated to obtain (x_avg, y_avg, z_avg), which is the centroid position. Finally, the coordinates of this three-dimensional geometric centroid are directly used as the current three-dimensional position of the target person in the second coordinate system, and the corresponding timestamp is recorded synchronously to provide data support for subsequent trajectory association.

[0081] For example, taking the tracking of ground staff wearing red tags in an airport terminal as an example, for a identified target person, the corresponding personnel point cloud cluster is first retrieved. Statistical filtering removes 15 isolated noise points caused by lidar reflection errors; voxel downsampling is used to retain 600 representative points, with voxel sizes of 4cm×4cm×4cm, forming a simplified point cloud cluster that still clearly retains the core three-dimensional features of the ground staff, such as height and shoulder width. Then, the average x-coordinate of the 600 points is calculated to obtain x_avg=38.52 meters, the average y-coordinate to obtain y_avg=12.36 meters, and the average z-coordinate to obtain z_avg=1.65 meters, yielding the centroid coordinates (38.52, 12.36, 1.65). Finally, these coordinates are determined as the current three-dimensional position of the target ground staff in the terminal's second coordinate system, accurately reflecting their real-time spatial orientation near the check-in counter in the departure hall.

[0082] As can be seen from the above, this embodiment of the application ensures that the point cloud data on which the subsequent location calculation is based belongs to the target person by accurately associating the target person with specific point cloud clusters, thus avoiding location deviations caused by data mismatch and laying the foundation for the accuracy of the calculation results. By removing noise points to improve the quality of point cloud data and reducing the number of points through downsampling, the computational power consumption of subsequent calculations is reduced while retaining core spatial features, improving the efficiency of location determination, which is particularly suitable for large-scale personnel tracking scenarios. Using the centroid coordinates directly as the location result ensures that the location is under a unified spatial reference in the second coordinate system, facilitating spatiotemporal association with historical trajectory nodes and providing standardized location data for the subsequent construction of continuous trajectories.

[0083] In one embodiment of this application, a spatiotemporal relationship graph is constructed, comprising: nodes and at least one edge;

[0084] Based on the current observation node and the target person's historical trajectory nodes, a spatiotemporal correlation graph is constructed, including:

[0085] For a target person who has been successfully tracked for at least one frame, the historical trajectory node of the target person consists of the target person's historical three-dimensional position, movement speed, appearance feature vector and identity identifier.

[0086] The current observation node and its corresponding historical trajectory node are used as nodes;

[0087] Establish an edge between the current observation node and the historical trajectory node;

[0088] The edge also includes a weight value, which is calculated based on the consistency between the current observation node and the historical trajectory node connected by the edge;

[0089] Construct a spatiotemporal relationship graph based on nodes and each edge.

[0090] In this embodiment, the spatiotemporal correlation graph is a graph structure composed of nodes and edges, used to associate the current observation node and historical trajectory nodes of a target person, intuitively reflecting the spatiotemporal relationship between nodes. Nodes here refer to the current observation node and the historical trajectory node, which are the basic units in the graph structure. Edges are relationship lines connecting the current observation node and the historical trajectory node, used to represent the association between the two. Identifiers are symbols or codes used to uniquely identify the target person, ensuring that nodes of the same target can be accurately associated. Weight values ​​are the numerical values ​​carried by the edges, used to measure the tightness of the association between the current observation node and the historical trajectory node it connects. Consistency refers to the degree of matching between the current observation node and the historical trajectory node in terms of spatiotemporal features.

[0091] In this embodiment, for a target person who has been successfully tracked for at least one frame, their historical trajectory nodes are extracted. These nodes must fully contain their historical 3D position, which could be their coordinates, movement speed, movement rate and direction, and appearance feature vectors (e.g., digital representations of clothing features and a unique identifier). Next, the target person's current observation node and the aforementioned historical trajectory nodes are used together as vertices in a graph structure, ensuring that each vertex carries corresponding spatiotemporal feature data. Then, an edge is established between the current observation node and the historical trajectory node to clarify their relationship. By comparing the 3D position distance, movement speed changes, and appearance feature vector matching degree between nodes, consistency is comprehensively evaluated and converted into weight values ​​between 0 and 1. All vertices and edges are integrated to form a complete spatiotemporal association graph, providing a structured association basis for subsequent trajectory tracking.

[0092] For example, taking a sprinter with tracking number B12 in a sports stadium as an example, this athlete has been successfully tracked for two frames. Their historical trajectory node includes: historical 3D position (10.2, 5.3, 1.7), speed (8.5 m / s, direction along the track), appearance feature vector (digital features of red sprinter's uniform and black running shoes), and the identifier B12. The current observation node includes their latest 3D position (12.3, 5.4, 1.7) and the corresponding motion state data. When constructing the spatiotemporal correlation graph, these two nodes are used as vertices, and an edge is established between them. When calculating the weight value, due to the close position distance, minimal speed change, and perfect matching of appearance features, a weight value of 0.92 is obtained based on a comprehensive consistency evaluation. The final spatiotemporal correlation graph contains these two vertices and an edge with a weight of 0.92, clearly reflecting the spatiotemporal correlation of the athlete in two consecutive frames.

[0093] In this embodiment, in response to the historical trajectory node not being matched with any current observation node in the current frame, it is also necessary to determine whether the identity identifier corresponding to the historical trajectory node is within a preset short-term holding window;

[0094] In response to being within a short-term hold window, the predicted 3D position of the historical trajectory node is calculated using a position model; the predicted historical trajectory node is then calculated based on the predicted 3D position.

[0095] If no current observation node is matched within the short-term hold window, the target person is officially marked as having left the field and their trajectory terminated by connecting to a virtual leave node via a leave edge.

[0096] In this embodiment, in response to the prediction of a historical trajectory node matching a new current observation node within a short-term hold window, the timing of the short-term hold window is stopped; and the spatiotemporal correlation graph of the target person is updated based on the new current observation node.

[0097] In this embodiment, the short-term holding window is a preset time range used to temporarily retain the identifiers corresponding to historical trajectory nodes that have not been matched with the current observation node, avoiding misjudgment of the target person leaving due to brief occlusion, detection delay, etc. The position model is a predictive model built based on the motion characteristics of historical trajectory nodes, used to calculate the possible position of the target person in the current frame. The predicted 3D position is calculated by the position model, representing the estimated 3D coordinates of the target person when not detected in the current frame. The predicted historical trajectory node is a node with the predicted 3D position as its core, containing the predicted motion state and corresponding identifier, used to temporarily maintain trajectory continuity when no matching with the current observation node is found. The departure edge is the edge connecting the historical trajectory node and the virtual departure node, used to identify the termination relationship of the target person's trajectory. The virtual departure node is a virtual vertex in the spatiotemporal graph used to mark that the target person has left, serving as a landmark node indicating the end of the trajectory.

[0098] Furthermore, when a historical trajectory node is not matched with any currently observed node in the current frame, the system's preset judgment logic is first invoked to check whether the identifier corresponding to the historical trajectory node is within a short-term holding window. The window duration can be dynamically set according to the scenario, such as 8-10 seconds for densely populated scenarios and 5-8 seconds for open scenarios. If it is within the window, the position model is activated: based on the historical 3D position, motion velocity, and acceleration of the historical trajectory node, combined with physical motion laws (such as uniform velocity or uniform acceleration models), its predicted 3D position is calculated; with this predicted position as the core, the predicted motion state and the original identifier are supplemented to generate a predicted historical trajectory node, which is temporarily included in the spatiotemporal correlation graph to maintain trajectory continuity. If no currently observed node is matched after the short-term holding window, the departure marking mechanism is triggered: a departure edge (with a fixed weight of 1.0, indicating deterministic departure) is established between the historical trajectory node and the virtual departure node, and the historical trajectory node and the virtual departure node are associated through this edge, officially marking that the target person has left the scene, and at the same time terminating its trajectory calculation.

[0099] If, within the short-term hold window, the predicted historical trajectory node successfully matches the newly emerging current observation node, the timing of the short-term hold window is immediately stopped, an edge connection is established between the new current observation node and the original historical trajectory node, and the spatiotemporal correlation graph of the target personnel is updated with the actual three-dimensional position, motion state, and other information of the new node, thus resuming normal trajectory tracking.

[0100] As can be seen from the above, this application embodiment, by integrating multi-dimensional features, ensures that historical nodes can comprehensively reflect the past status of the target personnel, providing rich evidence for consistency comparison with the current node and avoiding correlation deviations caused by single information. Abstracting discrete spatiotemporal information into unified vertex units facilitates standardized correlation analysis, lays the foundation for graph structure construction, and improves the clarity of correlation logic. Through the physical connection of edges, the spatiotemporal correlation between the two is intuitively established, breaking the information isolation between nodes and providing direct correlation clues for the continuous tracking of subsequent trajectories. Quantitative evaluation of correlation strength transforms the correlation from qualitative to quantitative, improving the accuracy of the correlation. Integrating all spatiotemporal correlation information through the graph structure not only preserves the detailed features of the nodes but also reflects the overall correlation logic, providing structured data support for subsequent graph-based trajectory tracking and improving the systematicness and efficiency of tracking. This application embodiment, by setting a time buffer, avoids erroneous trajectory termination due to non-exit factors such as brief occlusion and detection delay, preserving the possibility of trajectory continuation when the target personnel are detected again, and improving the anti-interference capability of the trajectory in complex scenarios. When no matching is found at the current observation node, trajectory continuity is maintained through prediction to avoid trajectory breaks, making trajectory data more complete and providing a reference benchmark for subsequent re-matching. The departure status of target personnel is clearly marked, invalid trajectories are terminated promptly, reducing the system's computational power consumption on departing targets, and ensuring that trajectory data includes a complete appearance-movement-departure cycle, improving data integrity. Normal tracking is quickly resumed when the target personnel are detected again, ensuring a smooth transition from the predicted state to the actual state, maintaining trajectory consistency and accuracy.

[0101] In one embodiment of this application, dynamic behavioral features of the target person are extracted using a deep learning model. These dynamic behavioral features include gait characteristics, limb movement frequency, and rate of change of movement direction.

[0102] Dynamic behavioral features are incorporated into the weight calculation of edges in the spatiotemporal correlation graph. When the matching degree of dynamic behavioral features between the current observation node and the historical trajectory node is higher than a preset threshold, the weight of the corresponding edge is increased.

[0103] In this embodiment, dynamic behavioral features refer to the set of dynamic movement attributes exhibited by the target person during movement, which are individualized. Specifically, these include gait characteristics, limb movement frequency, and rate of change of movement direction. Gait characteristics are the movement patterns of the target person's limbs when walking, reflecting the individual's unique walking posture. Limb movement frequency is the number of times the target person's limbs move per unit time, reflecting the rhythm of movement. Rate of change of movement direction is the magnitude of change in the target person's movement direction per unit time, reflecting the turning characteristics of the movement trajectory. Dynamic behavioral feature matching degree refers to the similarity between the dynamic behavioral features corresponding to the current observation node and the historical trajectory nodes, usually represented by a value between 0 and 1, with higher values ​​indicating greater feature similarity. The weight of the corresponding edge is increased through a reinforcement formula, as shown below:

[0104]

[0105] in, These are the initial weights, ranging from [0, 1]. The dynamic behavior feature matching degree is in the range [0, 1]; T is the preset matching degree threshold. Control the rate at which the weight increases after the matching degree exceeds a threshold. >0; a:min(1.0,...) is used to ensure that the final weight does not exceed 1.0.

[0106] In this embodiment, the dynamic behavioral features of the target person, including gait characteristics, limb movement frequency, and rate of change of movement direction, are first extracted using a preset deep learning model. The cosine similarity algorithm is then used to calculate the matching degree of the dynamic behavioral features between the current observation node and historical trajectory nodes. The dynamic behavioral features of the two types of nodes are converted into feature vectors, and the similarity is calculated to obtain a matching degree value. This matching degree is compared with a preset threshold. If the matching degree is higher than the threshold, it indicates that the dynamic behaviors of the two types of nodes are highly consistent, and the probability of them belonging to the same target person is higher. In this case, the weight of the edge between the two is increased by a preset amount. If the matching degree is lower than the threshold, the original weight is maintained or appropriately reduced to ensure that the edge weight accurately reflects the reliability of the node association.

[0107] For example, taking a Tai Chi practitioner with the identifier T08 practicing in a city park as an example, the dynamic behavioral characteristics of this practitioner—gait features, limb movement frequency, and rate of change of movement direction—are calculated using a deep learning model. When calculating the matching degree of the dynamic behavioral characteristics between the current observation node and historical trajectory nodes, a similarity of 0.86 is obtained, which is higher than the preset threshold of 0.8. Therefore, the weight of the edge between the two is increased from the initial calculation of 0.72 to 0.80. This adjustment ensures that when other practitioners are present in the park, even if their appearance or movement speed is similar, their unique dynamic behavioral characteristics can accurately associate them with T08's trajectory node, avoiding identity confusion.

[0108] As can be concluded from the above, the dynamic behavior features of this application embodiment have strong individual uniqueness. Compared with traditional position and speed features, they can more accurately distinguish target personnel with similar appearance or movement speed, providing more identifiable feature basis for subsequent node association and reducing the risk of misidentification. By quantifying the matching degree of dynamic behavior features to adjust the edge weights, the weights can not only reflect the spatiotemporal location correlation but also the consistency of dynamic behavior, further enhancing the reference value of the weights. Increasing the weight when the matching degree is higher than the threshold can strengthen the reliability of high-confidence associations, ensuring that nodes of the same target personnel can be accurately connected, especially in dense scenes or when targets have similar appearances, significantly improving the accuracy of trajectory tracking.

[0109] In one embodiment of this application, the dynamic behavioral features of the target person are extracted using a deep learning model, including:

[0110] A two-stream convolutional neural network is constructed, in which the first-stream network is used to extract temporal features of limb motion from consecutive two-dimensional image frames, and the second-stream network is used to extract spatial motion features from the dynamic changes of three-dimensional point clouds.

[0111] By fusing temporal and spatial motion features, the dynamic behavioral characteristics of the target personnel are obtained.

[0112] In this embodiment, the two-stream convolutional neural network (DNN) is a deep learning model comprising two parallel sub-networks: a first-stream network and a second-stream network. By processing dynamic data of different modalities separately and fusing features, it achieves more comprehensive feature extraction. The first-stream network is a sub-network in the DNN specifically designed to process continuous two-dimensional image frames, primarily used to capture the patterns of limb movement over time. The temporal characteristics of limb movement refer to the changes in the target person's limb movements over time, reflecting the sequence and rhythm of the movements. The second-stream network is a sub-network in the DNN specifically designed to process the dynamic changes of three-dimensional point clouds, primarily used to capture the morphological changes of limb movement in the spatial dimension. Spatial motion characteristics refer to the changes in the position, posture, and trajectory of the target person's limbs in three-dimensional space, reflecting the three-dimensional form and spatial distribution patterns of the movement.

[0113] In this embodiment, the first-stream network can be composed of a combination of 2D convolutions and LSTM. It takes a continuous sequence of two-dimensional image frames as input, extracts local appearance features of the limbs in each frame through 2D convolutional layers, and then captures the evolution of these features over time through LSTM layers, ultimately outputting a temporal feature vector of limb movement. The second-stream network can be composed of a combination of PointNet and convolutional layers. It first arranges the three-dimensional point clouds of consecutive frames in chronological order to form a point cloud sequence, extracts the spatial geometric features of the limbs in each frame through the PointNet network, and then learns the dynamic changes of these features over time through convolutional layers, outputting a spatial motion feature vector. An attention fusion mechanism is employed to calculate the correlation weights between temporal features and spatial motion features, giving greater attention to feature components with higher weights. The two are then concatenated and input into a fully connected layer for dimensional unification and feature integration, ultimately obtaining dynamic behavioral features of the target person including gait, limb movement frequency, and rate of change of movement direction.

[0114] Furthermore, attention weights are used to evaluate the importance of temporal and spatial features in the current scene, and their calculation formula is shown below:

[0115]

[0116]

[0117]

[0118] in, and It is a learnable weight matrix; and It is a learnable bias vector; and It maps the hidden layer to a scalar learnable weight vector; tanh is the activation function; For attention weights, the two scalars are combined using the softmax function. Convert to two attention weights that sum to 1 .

[0119] Furthermore, the calculated attention weights are used to perform a weighted summation of the original features to obtain the fused dynamic behavior feature vector.

[0120] For example, taking the museum guide with tracking number G06 as an example, when extracting his dynamic behavior features, the first-line network inputs continuous two-dimensional image frames of the guide, extracts local limb features such as hand gestures and turns through 2D convolutional layers, and then captures the evolution of these features over time through LSTM layers to obtain a temporal feature vector. The second-line network inputs the corresponding three-dimensional point cloud sequence, extracts the three-dimensional spatial coordinate features of the guide's torso and arms through PointNet, and then learns the spatial position change of the arms from being raised in front of the chest to the direction of the exhibit through convolutional layers to obtain a spatial motion feature vector. When using attention fusion, since the temporal pattern of hand gestures and the spatial features of the arm's spatial direction are most correlated, they are given higher weights. After being concatenated, they are integrated into a dynamic behavior feature vector through a fully connected layer. This vector accurately reflects the guide's unique rhythm of lecturing gestures and spatial movement patterns.

[0121] As can be seen from the above, the first-order network in this embodiment adapts to the temporal characteristics of two-dimensional image frames, effectively capturing the temporal evolution of limb movements and avoiding the limitation that a single frame image cannot reflect the dynamic process, thus providing temporal dimension support for dynamic behavioral features. The second-order network is designed for the spatial characteristics of three-dimensional point clouds, accurately capturing the three-dimensional shape and spatial position changes of limb movements, compensating for the lack of spatial features caused by the lack of depth information in two-dimensional images, and providing spatial dimension support for dynamic behavioral features. By integrating bimodal features through mechanisms such as attention fusion, the final dynamic behavioral features include both temporal motion patterns and spatial three-dimensional shapes, making them more individually identifiable than single-modal features. This allows for more accurate differentiation of target individuals with similar appearances or basic movements, providing high-quality feature basis for subsequent edge weight calculation in the spatiotemporal correlation graph, further improving the accuracy of trajectory tracking.

[0122] In one embodiment of this application, tracking the continuous movement trajectory of a target person based on a spatiotemporal correlation graph includes:

[0123] A physical motion sub-model is constructed based on the motion speed and acceleration of historical trajectory nodes. The physical motion sub-model is used to predict the linear motion trajectory of the target person.

[0124] A scene constraint sub-model is constructed by combining the three-dimensional structural information of the scene in the second data. The three-dimensional structural information includes the ground slope and the distribution of obstacles.

[0125] The prediction results of the physical motion sub-model are corrected based on the scene constraint sub-model. When the predicted position is in the obstacle area, it is adjusted to the nearest drivable area.

[0126] By fusing the outputs of the two sub-models, the predicted 3D position is obtained;

[0127] Based on spatiotemporal correlation maps and predicted 3D location, the continuous movement trajectory of target personnel is tracked.

[0128] In this embodiment, the physical motion sub-model is a prediction model built based on the historical motion parameters of the target person. These historical motion parameters can be velocity and acceleration, used to infer the linear motion trend of the target person from the perspective of physical motion laws. A linear motion trajectory refers to the path along a straight line or approximately a straight line when the target person is not disturbed by complex scenes. The scene constraint sub-model is a model that combines the three-dimensional structural information of the scene to constrain the motion prediction results, ensuring that the prediction results conform to the physical constraints of the actual scene. Three-dimensional structural information refers to three-dimensional data related to the spatial structure in the collected scene, specifically including ground slope and obstacle distribution. Ground slope refers to the degree of inclination of the scene's ground, reflecting the trend of elevation changes. Obstacle distribution refers to the spatial location and range of objects in the scene that hinder the movement of the person. A drivable area refers to the spatial area in the scene that allows normal movement of the person, excluding obstacles or impassable areas. The predicted three-dimensional position refers to the estimated three-dimensional coordinates of the target person at a future time, obtained by fusing the outputs of the physical motion sub-model and the scene constraint sub-model.

[0129] In this embodiment, when executing the trajectory tracking process, it is necessary to extract the target person's motion speed and acceleration from historical trajectory nodes. Based on the formula for uniformly accelerated linear motion, the linear motion trajectory in the near future is calculated to obtain a preliminary predicted position. The three-dimensional structural information of the scene is extracted from the second data. The ground slope is calculated using a plane fitting algorithm, and obstacles are identified and their spatial distribution is recorded through clustering and morphological processing. The preliminary predicted position of the physical motion sub-model is compared with the obstacle distribution in the scene constraint sub-model. If the preliminary predicted position falls within an obstacle area (e.g., the coordinate range occupied by a shelf), the nearest traversable area near that position is found through distance calculation, such as the passageway next to the shelf, and the predicted position is adjusted to that area. A weighted fusion strategy is used to calculate the prediction results of the physical motion sub-model and the corrected results of the scene constraint sub-model to obtain the final predicted three-dimensional position. Finally, combining the association relationship between the current observation node and historical trajectory nodes in the spatiotemporal correlation graph, the predicted three-dimensional position is included as a temporary node in the correlation graph to achieve continuous tracking of the target person's motion trajectory, especially maintaining trajectory continuity when detection is interrupted.

[0130] For example, taking the tracking of customer M pushing a shopping cart in a large supermarket as an example, when constructing the physical motion sub-model, based on the motion speed and acceleration of its historical trajectory nodes, the predicted endpoint position of its linear motion trajectory after 3 seconds is (18.5, 9.3, 1.6). When constructing the scene constraint sub-model, the 3D structural information of the area is extracted from the 3D point cloud data, the ground slope is 2°, and the distribution of shelf obstacles includes the coordinate range. When correcting the prediction result, it is found that the initial predicted position (18.5, 9.3, 1.6) is in the shelf area, and the nearest traversable area is calculated to be (18.5, 9.5, 1.6), and the predicted position is adjusted to this. When fusing the sub-model output, the weight of the physical model result is 0.6, and the weight of the corrected result is 0.4, and the final predicted 3D position is calculated to be (18.5, 9.42, 1.6). Combining the historical nodes of customer M in the spatiotemporal correlation graph with the current predicted node, this predicted position is included in the trajectory, and the continuous motion trajectory of customer M continuing to move around the shelves is successfully tracked.

[0131] As can be seen from the above, the embodiments of this application predict based on the laws of physical motion, which can capture the basic movement trend of the target person and provide a basic prediction result that conforms to the motion inertia for trajectory tracking, ensuring the rationality of the prediction. Incorporating the actual physical constraints of the scene into the prediction avoids predictions that deviate from the actual scene due to reliance solely on physical models, making the prediction results more consistent with the real environment. By adjusting the predicted position of obstacle areas to a drivable area, physically infeasible movement paths are directly excluded, ensuring the effectiveness of the predicted position and improving the realism of the trajectory. Combining the advantages of physical motion trends and scene constraints, the influence of motion inertia and environmental limitations is balanced, making the prediction results both consistent with the laws of motion and adaptable to the characteristics of the scene, improving prediction accuracy. By combining the predicted position with the spatiotemporal correlation graph, even when the target person is briefly occluded or detection is interrupted, the continuity of the trajectory can still be maintained through prediction, avoiding trajectory breakage and ensuring the integrity of tracking.

[0132] Corresponding to the personnel trajectory tracking method in the above embodiment, Figure 2 This is a structural block diagram of a personnel trajectory tracking system provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. Reference Figure 2 The personnel trajectory tracking system 20 includes: a data acquisition module 21, a first data processing module 22, a second data processing module 23, a target personnel determination module 24, a node determination module 25, a spatiotemporal correlation graph determination module 26, and a trajectory tracking module 27.

[0133] The data acquisition module 21 is used to acquire first data and second data; the first data and second data are data synchronously acquired from the same scene being acquired; the first data is two-dimensional image data of the scene being acquired; the second data is three-dimensional point cloud data of the scene being acquired.

[0134] First data processing module 22: used to process the first data and obtain a set of personnel bounding boxes in the first data; the set of personnel bounding boxes contains at least one personnel bounding box;

[0135] Second data processing module 23: used to process the second data, generate a second coordinate system; and obtain a personnel point cloud cluster set in the second coordinate system; the personnel point cloud cluster set contains at least one personnel point cloud cluster;

[0136] Target Personnel Determination Module 24: Used to determine the target personnel set based on the personnel bounding box set and the personnel point cloud cluster set through joint confidence calculation; the target personnel set contains at least one target personnel.

[0137] Node determination module 25: used to calculate the current three-dimensional position of each target person in the second coordinate system, as the current observation node;

[0138] Spatiotemporal correlation graph determination module 26: used to construct a spatiotemporal correlation graph based on the current observation node and the historical trajectory nodes of the target person;

[0139] Trajectory tracking module 27: Used to track the continuous movement trajectory of a target person based on a spatiotemporal correlation graph.

[0140] In one embodiment of this application, when the target personnel determination module 24 determines the target personnel set based on the personnel bounding box set and the personnel point cloud cluster set through joint confidence calculation, it is specifically used for:

[0141] Pair each person's bounding box with each person's point cloud cluster to form candidate pairs;

[0142] For each candidate pair, calculate the two-dimensional detection confidence, three-dimensional clustering confidence, and matching score;

[0143] The joint confidence score is obtained by weighting the two-dimensional detection confidence score, the three-dimensional clustering confidence score, and the matching score.

[0144] The target personnel set is determined based on joint confidence levels.

[0145] In one embodiment of this application, the node determination module 25, when calculating the current three-dimensional position of each target person in the second coordinate system, is specifically used for:

[0146] Obtain point cloud clusters of personnel corresponding to the target personnel;

[0147] The point cloud clusters of personnel are simplified to obtain simplified point cloud clusters.

[0148] Calculate the three-dimensional geometric centroid of the simplified point cloud cluster;

[0149] The coordinates of the three-dimensional geometric centroid are used as the current three-dimensional position of the target personnel in the second coordinate system.

[0150] In one embodiment of this application, the spatiotemporal correlation graph determination module 26, when constructing the spatiotemporal correlation graph, is specifically used for:

[0151] Based on the current observation node and the target person's historical trajectory nodes, a spatiotemporal correlation graph is constructed, including:

[0152] For a target person who has been successfully tracked for at least one frame, the historical trajectory node of the target person consists of the target person's historical three-dimensional position, movement speed, appearance feature vector and identity identifier.

[0153] The current observation node and its corresponding historical trajectory node are used as nodes;

[0154] Establish an edge between the current observation node and the historical trajectory node;

[0155] The edge also includes a weight value, which is calculated based on the consistency between the current observation node and the historical trajectory node connected by the edge;

[0156] Construct a spatiotemporal relationship graph based on nodes and each edge.

[0157] In one embodiment of this application, the spatiotemporal correlation graph determination module 26, when constructing the spatiotemporal correlation graph, is further configured to:

[0158] The dynamic behavioral features of the target personnel are extracted through deep learning models. These dynamic behavioral features include gait characteristics, limb movement frequency, and rate of change of movement direction.

[0159] Dynamic behavioral features are incorporated into the weight calculation of edges in the spatiotemporal correlation graph. When the matching degree of dynamic behavioral features between the current observation node and the historical trajectory node is higher than a preset threshold, the weight of the corresponding edge is increased.

[0160] In one embodiment of this application, the spatiotemporal correlation graph determination module 26, when extracting the dynamic behavioral features of the target personnel through a deep learning model, is specifically used for:

[0161] A two-stream convolutional neural network is constructed, in which the first-stream network is used to extract temporal features of limb motion from consecutive two-dimensional image frames, and the second-stream network is used to extract spatial motion features from the dynamic changes of three-dimensional point clouds.

[0162] By fusing temporal and spatial motion features, the dynamic behavioral characteristics of the target personnel are obtained.

[0163] In one embodiment of this application, when tracking the continuous movement trajectory of a target person based on a spatiotemporal correlation graph, the trajectory tracking module 27 is specifically used for:

[0164] A physical motion sub-model is constructed based on the motion speed and acceleration of historical trajectory nodes. The physical motion sub-model is used to predict the linear motion trajectory of the target person.

[0165] A scene constraint sub-model is constructed by combining the three-dimensional structural information of the scene in the second data. The three-dimensional structural information includes the ground slope and the distribution of obstacles.

[0166] The prediction results of the physical motion sub-model are corrected based on the scene constraint sub-model. When the predicted position is in the obstacle area, it is adjusted to the nearest drivable area.

[0167] By fusing the outputs of the two sub-models, the predicted 3D position is obtained;

[0168] Based on spatiotemporal correlation maps and predicted 3D location, the continuous movement trajectory of target personnel is tracked.

[0169] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the data acquisition module 21, the first data processing module 22, the second data processing module 23, the target personnel determination module 24, the node determination module 25, the spatiotemporal correlation diagram determination module 26, and the trajectory tracking module 27 are shown.

[0170] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0171] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0172] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0173] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the personnel trajectory tracking method provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0174] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0175] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0176] This application provides a computer program product, which includes computer-executable instructions or a computer program. The computer-executable instructions or computer program are stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the personnel trajectory tracking method described in this application embodiment.

[0177] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0180] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0181] Furthermore, the functional modules / units in the various embodiments of this application can be integrated into one processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated modules / units described above can be implemented in hardware or in the form of software functional modules / units.

[0182] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of tracking a trajectory of a person, characterized by, The method comprises the following steps: acquiring first data and second data; the first data and the second data are data synchronously acquired for the same collected scene; the first data is two-dimensional image data of the collected scene; the second data is three-dimensional point cloud data of the collected scene; processing the first data to acquire a personnel bounding box set in the first data; the personnel bounding box set comprises at least one personnel bounding box; processing the second data to generate a second coordinate system and acquire a personnel point cloud cluster set in the second coordinate system; the personnel point cloud cluster set comprises at least one personnel point cloud cluster; based on the personnel bounding box set and the personnel point cloud cluster set, determining a target personnel set through joint confidence calculation; the target personnel set comprises at least one target personnel; calculating a current three-dimensional position of each target personnel in the second coordinate system as a current observation node; based on the current observation node and a historical trajectory node of the target personnel, constructing a spatio-temporal association graph; based on the spatio-temporal association graph, tracking a continuous motion trajectory of the target personnel; the step of constructing the spatio-temporal association graph comprises a node and at least one edge; the step of constructing the spatio-temporal association graph based on the current observation node and the historical trajectory node of the target personnel comprises: for a target personnel that has been successfully tracked for at least one frame, the historical trajectory node of the target personnel is composed of a historical three-dimensional position, a motion speed, an appearance feature vector and an identity identifier of the target personnel; taking the current observation node and the corresponding historical trajectory node as the node; establishing an edge between the current observation node and the historical trajectory node; the edge further comprises a weight value, which is calculated based on the consistency between the current observation node and the historical trajectory node connected by the edge; constructing the spatio-temporal association graph based on the node and each edge; extracting dynamic behavior features of the target personnel through a deep learning model, the dynamic behavior features comprising gait features, limb motion frequency and motion direction change rate; integrating the dynamic behavior features into the weight calculation of the edge in the spatio-temporal association graph, and increasing the weight of the corresponding edge when the dynamic behavior feature matching degree of the current observation node and the historical trajectory node is higher than a preset threshold; wherein, increasing the weight of the corresponding edge is realized based on a reinforcement formula, the reinforcement formula being: , is an initial weight, ranging [0, 1]; is a dynamic behavior feature matching degree, ranging [0, 1]; T is a preset matching degree threshold value; controls the growth rate of the weight after the matching degree is higher than the threshold value, >

0.

2. The people trajectory tracking method according to claim 1, wherein, the step of determining the target personnel set based on the personnel bounding box set and the personnel point cloud cluster set through joint confidence calculation comprises: pairing each personnel bounding box with each personnel point cloud cluster to form a candidate pair; calculating a two-dimensional detection confidence, a three-dimensional clustering confidence and a matching score for each candidate pair; performing weighted calculation on the two-dimensional detection confidence, the three-dimensional clustering confidence and the matching score to obtain a joint confidence; determining the target personnel set based on the joint confidence.

3. The people trajectory tracking method of claim 1, wherein, the step of calculating the current three-dimensional position of each target personnel in the second coordinate system comprises: acquiring a personnel point cloud cluster corresponding to the target personnel; performing simplification processing on the personnel point cloud cluster to obtain a simplified point cloud cluster; calculating a three-dimensional geometric center of the simplified point cloud cluster; coordinates of the three-dimensional geometric centroid as a current three-dimensional position of the target person in a second coordinate system.

4. The people trajectory tracking method of claim 1, wherein, The dynamic behavior feature of the target person is extracted through the deep learning model, including: A double-flow convolutional neural network is constructed, wherein a first flow network is used to extract time sequence features of limb movement from continuous two-dimensional image frames, and a second flow network is used to extract spatial motion features from dynamic changes of three-dimensional point clouds; The time sequence features and the spatial motion features are fused to obtain the dynamic behavior feature of the target person.

5. The people trajectory tracking method of claim 1, wherein, The continuous motion trajectory of the target person is tracked based on the spatio-temporal association graph, including: A physical motion sub-model is constructed based on motion speed and acceleration of historical trajectory nodes, and the physical motion sub-model is used to predict a linear motion trajectory of the target person; A scene constraint sub-model is constructed in combination with three-dimensional structure information of a scene in the second data, and the three-dimensional structure information includes ground slope and obstacle distribution; The prediction result of the physical motion sub-model is corrected based on the scene constraint sub-model, and when the predicted position is in an obstacle area, the predicted position is adjusted to the nearest available area; The output results of the two sub-models are fused to obtain a predicted three-dimensional position; The continuous motion trajectory of the target person is tracked based on the spatio-temporal association graph and the predicted three-dimensional position.

6. A people trajectory tracking system characterized by, It includes: A data acquisition module is configured to acquire first data and second data; the first data and the second data are data synchronously acquired for the same collected scene; The first data is two-dimensional image data of the collected scene; and the second data is three-dimensional point cloud data of the collected scene; A first data processing module is configured to process the first data to obtain a personnel bounding box set in the first data; the personnel bounding box set includes at least one personnel bounding box; A second data processing module is configured to process the second data to generate a second coordinate system and obtain a personnel point cloud cluster set in the second coordinate system; the personnel point cloud cluster set includes at least one personnel point cloud cluster; A target person determination module is configured to determine a target person set based on the personnel bounding box set and the personnel point cloud cluster set through joint confidence calculation; the target person set includes at least one target person; A node determination module is configured to calculate a current three-dimensional position of each target person in a second coordinate system as a current observation node; A spatio-temporal association graph determination module is configured to construct a spatio-temporal association graph based on the current observation node and a historical trajectory node of the target person; A trajectory tracking module is configured to track a continuous motion trajectory of the target person based on the spatio-temporal association graph. The spatio-temporal association graph includes nodes and at least one edge; The spatio-temporal association graph determination module is specifically configured to, for a target person whose motion trajectory has been successfully tracked for at least one frame, a historical trajectory node of the target person is composed of a historical three-dimensional position, a motion speed, an appearance feature vector and an identity identifier of the target person; The current observation node and the corresponding historical trajectory node are taken as nodes; An edge is established between the current observation node and the historical trajectory node. The edge further comprises a weight value, which is calculated based on consistency between the current observation node and the historical trajectory node connected by the edge; constructing a spatio-temporal association graph based on the nodes and each edge; The spatio-temporal association graph determination module is further configured to extract dynamic behavior features of the target person through a deep learning model, and the dynamic behavior features include gait features, limb movement frequency, and movement direction change rate. The dynamic behavior features are integrated into weight calculation of the edge in the spatio-temporal association graph, and when the dynamic behavior feature matching degree between the current observation node and the historical trajectory node is higher than a preset threshold, the weight of the corresponding edge is improved. The weight of the corresponding edge is improved based on a reinforcement formula, and the reinforcement formula is as follows: , is an initial weight, ranging [0, 1]; is a dynamic behavior feature matching degree, ranging [0, 1]; T is a preset matching degree threshold; controls the growth rate of the weight after the matching degree is higher than the threshold, >

0.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to realize the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to realize the steps of the method according to any one of claims 1 to 5.

Images