Construction turnover material trajectory tracking method and system

Abstract

The invention relates to a construction turnover material trajectory tracking method and system, and the method comprises the steps: constructing a probability occupancy graph, building a corresponding relation between a shooting region of each camera and a grid in the probability occupancy graph, obtaining the probability occupancy graph of the shooting regions of the cameras based on the corresponding relation, and obtaining a thermodynamic diagram of a video image in a target recognition mode. Obtaining a target and apparent characteristics according to the probability occupancy graph and the thermodynamic diagram of the camera shooting area; obtaining a track and a track apparent feature set in each camera by adopting the target and apparent features of the video image in each camera; obtaining trajectory coupling based on the trajectories of all the cameras, obtaining a sub-trajectory apparent feature set of each trajectory in the co-existence time of the trajectories in the trajectory coupling, calculating the affinity of the trajectory coupling by using the sub-trajectory apparent feature set, and obtaining a space-time view hypergraph by taking the trajectory coupling as a hypergraph node, and obtaining the track of the same turnover material in different cameras through the space-time view hypergraph.

Description

Technical Field

[0001] The present invention relates to the field of trajectory tracking, and specifically to a method and system for tracking the trajectory of construction reusable materials. Background Art

[0002] Construction reusable materials can be recycled multiple times, and their value is mainly reflected through multiple cycles of use. However, due to factors such as the actual construction situation and actual construction conditions, there will be situations such as long-term backlogs or excessive procurement and leasing quantities, which will not only affect the construction site but also increase the construction cost. In the past, for the reusable materials used in construction, the method of manual statistics by the materials department was mostly adopted. However, manual statistics are prone to omissions and are time-consuming and laborious. The material tracking method based on RFID or two-dimensional code is easily affected by the environment in complex construction scenarios and cannot obtain spatial position information, etc. For construction safety, many cameras are generally deployed at the construction site. With the development of computer vision, the cameras deployed at the construction site can not only be used to monitor the construction site but also be used to identify and track the reusable materials, improving the real-time performance and tracking performance of the management of reusable materials without increasing costs.

[0003] However, construction reusable materials such as scaffolding and formwork are relatively similar in appearance and are difficult to distinguish through simple visual features, which is likely to cause confusion of target IDs. In addition, the construction site is relatively large, and it is difficult to cover all areas with a single camera. Tracking multiple targets across cameras or multiple cameras can cover the construction site. However, the existing cross-camera multi-target tracking includes two stages: single-camera target recognition and trajectory matching, and trajectory matching between multiple cameras. However, due to the influence of factors such as occlusion and lighting, the single camera has problems of inaccurate or missed recognition when identifying targets. In addition, unconfirmed trajectories are easily deleted, resulting in ID switching, which all affect the accuracy of the trajectory tracking of reusable materials. Summary of the Invention

[0004] In order to improve the accuracy of target recognition and the trajectory tracking of construction reusable materials and prevent ID switching from causing the inability to distinguish the same reusable material, the embodiments of the present invention provide a method and system for tracking the trajectory of construction reusable materials, which can avoid tracking errors of construction reusable materials under multiple cameras.

[0005] To achieve the above object, the embodiments of the present invention provide the following technical solutions:

[0006] In the first aspect of the present invention, a method for tracking the trajectory of construction reusable materials is provided. The method includes the following steps:

[0007] Construct a probabilistic occupancy map using video images captured by multiple cameras at the construction site, establish the correspondence between the shooting areas of each camera and the grids in the probabilistic occupancy map, obtain the probabilistic occupancy map of the camera shooting area based on the correspondence, obtain the heat map of the video image by means of object recognition, and obtain the target and appearance features according to the probabilistic occupancy map and the heat map of the camera shooting area;

[0008] Use the target and appearance features of the video image in each camera to obtain the trajectory and the set of trajectory appearance features in each camera;

[0009] Based on the trajectories of all the cameras, obtain trajectory coupling, obtain the set of sub-trajectory appearance features of each trajectory during the coexistence time of the trajectories in the trajectory coupling, calculate the affinity of the trajectory coupling using the set of sub-trajectory appearance features, take the trajectory coupling as a hypergraph node to obtain a spatio-temporal view hypergraph, and obtain the trajectories of the same turnover material in different cameras through the spatio-temporal view hypergraph.

[0010] Preferably, the obtaining of the target and appearance features according to the probabilistic occupancy map and the heat map of the camera shooting area is specifically as follows:

[0011] Convert the probabilistic occupancy map of the camera shooting area to the same size as the heat map;

[0012] Normalize the heat map and the probabilistic occupancy map respectively and add them bit by bit to obtain an updated heat map;

[0013] Use the updated heat map to determine the target and the appearance features of the target.

[0014] Preferably, the obtaining of the trajectory and the set of trajectory appearance features in each camera using the target and appearance features of the video image in each camera is specifically as follows:

[0015] Use a Kalman filter to obtain the prediction box of the trajectory;

[0016] Match the appearance features of the prediction box and the detection box. For the trajectories and targets that do not match successfully, further use the IOU matching method for matching;

[0017] If the appearance feature matching or the IOU matching is successful, add the appearance feature of the detection box to the set of trajectory appearance features; for the trajectories that do not match successfully, if the average value of the probabilistic occupancy map of the prediction box of the trajectory is greater than the average value of the probabilistic occupancy map of the prediction box of the successfully matched trajectory, and the trajectory is in an unconfirmed state, then convert the trajectory to a confirmed state.

[0018] Preferably, the calculating of the affinity of the trajectory coupling using the set of sub-trajectory appearance features is specifically as follows:

[0019] Convert the set of apparent features of the sub-trajectories of the trajectory into an apparent feature matrix of the trajectory;

[0020] Calculate the similarity of the apparent feature matrix to obtain a similarity matrix, the size of the similarity matrix being N×N, where N is the number of trajectories in the trajectory coupling;

[0021] Take the average value of the sum of all elements of the similarity matrix as the apparent affinity, and calculate the affinity of the trajectory coupling using the apparent affinity.

[0022] Preferably, the affinity further includes a trajectory smoothness affinity and / or a motion continuity affinity.

[0023] In a second aspect of the present invention, there is provided a trajectory tracking system for construction turnover materials, the system comprising the following modules:

[0024] A target recognition module, configured to construct a probability occupancy map using video images captured by multiple cameras at a construction site, establish a correspondence between the shooting area of each camera and the grid in the probability occupancy map, obtain the probability occupancy map of the shooting area of the camera based on the correspondence, obtain a heat map of the video image by using a target recognition method, and obtain a target and apparent features according to the probability occupancy map and the heat map of the shooting area of the camera;

[0025] A single-camera trajectory acquisition module, configured to obtain a trajectory and a set of trajectory apparent features in each camera by using the target and apparent features of the video image in each camera;

[0026] A cross-camera trajectory acquisition module, configured to obtain a trajectory coupling based on the trajectories of all the cameras, obtain a set of sub-trajectory apparent features of each trajectory during the co-existence time of the trajectories in the trajectory coupling, calculate the affinity of the trajectory coupling by using the set of sub-trajectory apparent features, take the trajectory coupling as a hypergraph node to obtain a spatio-temporal view hypergraph, and obtain the trajectories of the same turnover material in different cameras through the spatio-temporal view hypergraph.

[0027] Preferably, the obtaining of the target and apparent features according to the probability occupancy map and the heat map of the shooting area of the camera is specifically:

[0028] Convert the probability occupancy map of the shooting area of the camera to the same size as the heat map;

[0029] Normalize the heat map and the probability occupancy map respectively and add them bit by bit to obtain an updated heat map;

[0030] Use the updated heat map to determine the target and the apparent features of the target.

[0031] Preferably, the obtaining of the trajectory and the set of trajectory apparent features in each camera by using the target and apparent features of the video image in each camera is specifically:

[0032] The predicted bounding box of the trajectory is obtained by using a Kalman filter;

[0033] The predicted bounding box and the detected bounding box are subjected to appearance feature matching. For the trajectories and targets that do not match successfully, the IOU matching method is further used for matching;

[0034] If the appearance feature matching or the IOU matching is successful, the appearance features of the detected bounding box are added to the appearance feature set of the trajectory; for the trajectories that do not match successfully, if the average value of the probability occupancy map of the predicted bounding box of the trajectory is greater than the average value of the probability occupancy map of the predicted bounding box of the successfully matched trajectory, and the trajectory is in a non - confirmed state, then the trajectory is converted into a confirmed state.

[0035] Preferably, calculating the affinity of trajectory coupling by using the sub - trajectory appearance feature set specifically includes:

[0036] The sub - trajectory appearance feature set of the trajectory is converted into an appearance feature matrix of the trajectory;

[0037] The similarity of the appearance feature matrix is calculated to obtain a similarity matrix, and the size of the similarity matrix is N×N, where N is the number of trajectories in the trajectory coupling;

[0038] The average value of the sum of all elements of the similarity matrix is used as the appearance affinity, and the affinity of trajectory coupling is calculated by using the appearance affinity.

[0039] Preferably, the affinity further includes a trajectory smoothness affinity and / or a motion continuity affinity.

[0040] In the third aspect of the present invention, there is provided a computer program product containing instructions, which when running on a computer, enables the computer to execute any one of the methods provided in any aspect of the first aspect.

[0041] In the fourth aspect of the present invention, there is provided a computer - readable storage medium including instructions, which when running on a computer, enables the computer to execute any one of the methods provided in any aspect of the first aspect.

[0042] The present invention improves the heat map in target recognition by using the probability occupancy maps of multiple cameras, avoids the influence of occlusion and poor lighting on a single camera, and thus improves the accuracy of single - camera target recognition; in addition, calculating the affinity of trajectory coupling by using appearance features improves the accuracy of affinity calculation; moreover, converting the non - confirmed state to the confirmed state through the probability occupancy map avoids the ID switching and ensures the consistency of the trajectory tracking of construction turnover materials. Description of the Drawings

[0043] Figure 1 It is a flowchart of Embodiment 1;

[0044] Figure 2 is the structural diagram of the target recognition model;

[0045] Figure 3 is the trajectory state conversion process;

[0046] Figure 4 shows the trajectory coupling obtained from the trajectories in three cameras;

[0047] Figure 5 is the structural diagram of the second embodiment. Detailed implementation manners

[0048] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner for easy understanding.

[0049] It can be understood that the "embodiments" mentioned throughout the specification mean that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, the embodiments throughout the specification do not necessarily refer to the same embodiments. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It can be understood that in various embodiments of the present application, the magnitude of the serial numbers of the various processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0050] In the present invention, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of the present invention, as well as in each implementation manner / implementation method / realization method in each embodiment, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments, as well as between each implementation manner / implementation method / realization method in each embodiment, are consistent and can be referenced to each other. The technical features in different embodiments, as well as in each implementation manner / implementation method / realization method in each embodiment, can be combined in accordance with their internal logical relationships to form new embodiments, implementation manners, implementation methods, or realization methods. The implementation manners of the present application described below do not constitute a limitation to the protection scope of the present application.

[0051] Figure 1 shows the flowchart of the first embodiment of the present invention, Figure 1 The construction turnover material trajectory tracking method in includes the following steps:

[0052] S1. Construct a probabilistic occupancy map using the video images captured by multiple cameras at the construction site, establish the correspondence between the shooting areas of each camera and the grids in the probabilistic occupancy map, obtain the probabilistic occupancy map of the camera shooting area based on the correspondence, use the target recognition method to obtain the heat map of the video image, and obtain the target and apparent features according to the probabilistic occupancy map and heat map of the camera shooting area;

[0053] Multiple cameras will be arranged at the construction site, and the monitoring areas of each camera are different. The construction turnover materials will move at the construction site, for example, being transported from the storage or warehouse to a specific location for installation, etc. In one embodiment, the construction site includes a warehouse or storage; among them, the construction turnover materials are materials that will be reused, including but not limited to scaffolding, foundation pit baffles, etc.

[0054] After obtaining the videos of multiple cameras, for the video frame images of different cameras at the same time, calculate the Probabilistic Occupancy Map (POM). The value in the probabilistic occupancy map represents the probability of the object existing at that place. The probabilistic occupancy map divides the construction site into multiple grids, and each grid has a probability value. The larger the value, the greater the possibility of the object existing here. Since the shooting perspectives and ranges of the cameras are different, establish the correspondence between the shooting areas of each camera and the grids in the probabilistic occupancy map to fuse the data of different cameras into a unified probabilistic occupancy map. For example, if the shooting area of the camera is Area 1, then find the corresponding range in the probabilistic occupancy map. Through the above correspondence, the probabilistic occupancy map of each camera shooting area can be extracted from the global probabilistic occupancy map.

[0055] Use a target recognition model such as FairMOT and other models to perform multi-target recognition on the video frames captured by the camera. Specifically, after passing the video frame through the encoder-decoder, then obtain the heat map through convolution, activation, etc. The heat map represents the possibility of the target existing. The structure of the target recognition model is as Figure 2As shown in the figure. There will be some false detection results in the heat map. Moreover, due to factors such as light and occlusion, some targets are not detected by one camera, and the information of other cameras cannot be utilized. The present invention obtains the target and the appearance feature according to the probability occupancy map and the heat map of the camera shooting area, which not only helps to identify some non-existent targets, but also utilizes the information of other cameras, improving the accuracy of target recognition in video frames. In a specific embodiment, the POM and the heat map are corresponded according to the corresponding relationship, and both are transformed to the same size, and then the probability occupancy map and the heat map corresponding to the camera are fused. Preferably, the heat map and the probability occupancy map are normalized respectively and added bit by bit to obtain the updated heat map, and models such as FairMOT are used to continue target recognition. Since the offset in the target recognition model only represents the offset of the target in the video frame and does not utilize the information of other cameras, in a preferred embodiment, the offset information of the target recognition model is no longer used. The updated heat map is used to obtain the target and the appearance feature Re-ID of the target through the target recognition model.

[0056] S2. Obtain the trajectory and the trajectory appearance feature set in each camera by using the target and the appearance feature of the video image in each camera;

[0057] After obtaining the target in the video image captured by the camera, use the multi-target tracking model to match the target and the trajectory, so that the trajectory and the trajectory appearance feature set in each camera can be obtained.

[0058] The construction site environment is complex and occlusion and other situations are likely to occur. Traditional trajectory confirmation mainly relies on appearance feature matching and IOU matching. These methods may fail when the target is occluded, the light changes or the target moves rapidly, and it is easy to cause ID switching, resulting in multiple IDs being assigned to the same turnover material. The probability occupancy map (POM) can verify whether the trajectory prediction box actually occupies the physical space. If the average value of the POM corresponding to the prediction box is high, it indicates that there is a high probability of the target in this area, thus enhancing the reliability of trajectory confirmation. In an embodiment, the step of obtaining the trajectory and the trajectory appearance feature set in each camera by using the target and the appearance feature of the video image in each camera specifically includes:

[0059] Use the Kalman filter to obtain the prediction box of the trajectory;

[0060] Perform appearance feature matching between the prediction box and the detection box. For the trajectories and targets that do not match successfully, further use the IOU matching method for matching;

[0061] If the appearance features match or the IOU matching is successful, add the appearance features of the detection box to the appearance feature set of the track; for the tracks that do not match successfully, if the average probability occupancy map of the predicted box of the track is greater than the average probability occupancy map of the predicted boxes of the tracks that match successfully, and the track is in an unconfirmed state, then convert the track to a confirmed state.

[0062] The Kalman filter can predict the possible position of the target in the next frame based on the motion states of the target object in the past few frames, such as position and speed, and the prediction result is the predicted box. In an actual video image, through an object detection model such as FairMOT, a detection box (target box) can be obtained, and the detection box identifies the target object actually detected in the current frame.

[0063] There are multiple ways to match the track and the detection box. Preferably, first perform appearance feature matching, and then for the tracks and detection boxes that do not match successfully, further use IOU matching. For the detection boxes that still do not match successfully after the above two processes, a track will be initialized using the detection box. For the tracks that do not match successfully, calculate the average value of the probability occupancy map in the predicted box of the track. If the average value of the probability occupancy map of the predicted box of the track is greater than the average value of the probability occupancy maps of those predicted boxes that have successfully matched, the unmatched predicted box is very likely to be a real target. At this time, if the state of the track corresponding to this predicted box is in an unconfirmed state, then change the state of the track corresponding to this predicted box from uncertain to certain, that is, the confirmed state, as Figure 3 shown, to prevent this track from being deleted and thus avoid ID switching. In one embodiment, the conversion from the unconfirmed state to the confirmed state is only determined according to the average value of the probability occupancy map of the predicted box of the track; in another embodiment, the conversion from the unconfirmed state to the confirmed state is only determined according to the average value of the probability occupancy map of the predicted box of the track and the number of consecutive successful matches. For example, although the average value of the probability occupancy map of the predicted box of the track does not meet the conditions, but the number of consecutive successful matches of the unconfirmed state track exceeds a preset value such as 3 times, the unconfirmed state track will also be converted to the confirmed state.

[0064] S3. Obtain track coupling based on the tracks of all the cameras, acquire the sub-track appearance feature sets of each track during the coexistence time of the tracks in the track coupling, calculate the affinity of the track coupling using the sub-track appearance feature sets, take the track coupling as a hypergraph node to obtain a spatio-temporal view hypergraph, and obtain the tracks of the same turnover material in different cameras through the spatio-temporal view hypergraph.

[0065] After obtaining the trajectories of all cameras, obtain the trajectory couplings of all cameras. A trajectory coupling includes at least one trajectory, and any two trajectories in the trajectory coupling do not come from the same camera, and all trajectories in the trajectory coupling overlap in time. For example, the existence time of trajectory t1 in the trajectory coupling is 10 - 20s, the existence time of trajectory t2 is 16 - 27s, and trajectories t1 and t2 come from different cameras. Figure 4 Shows the trajectory couplings obtained from the trajectories in three cameras.

[0066] Calculate the co-existence time of the trajectories in the trajectory coupling, and obtain the subset of the trajectories and the corresponding apparent feature sets within this time from the trajectories of each camera, that is, the sub-trajectory apparent feature set. Calculate the similarity of the sub-trajectory apparent feature sets in the trajectory coupling, for example, using methods such as cosine similarity and Pearson correlation coefficient. Compared with traditional appearance affinity, since the sub-trajectory apparent features come from the target recognition model, the features they cover are more abundant. Take the trajectory coupling as the nodes of the hypergraph, and construct hyperedges to obtain the space-time-view hypergraph (STV hypergraph). Use the space-time-view hypergraph to match the trajectories of different cameras to complete the tracking of cross-camera target trajectories. Using the space-time-view hypergraph to achieve multi-camera multi-target tracking belongs to the prior art and will not be elaborated here.

[0067] The apparent feature is a vector, and there are multiple vectors in the apparent feature set of a time period, that is, the sub-trajectory apparent feature set. In one embodiment, the affinity of the trajectory coupling calculated using the sub-trajectory apparent feature set is specifically:

[0068] Convert the sub-trajectory apparent feature set of the trajectory into the apparent feature matrix of the trajectory;

[0069] Calculate the similarity of the apparent feature matrix to obtain the similarity matrix. The size of the similarity matrix is N×N, where N is the number of trajectories in the trajectory coupling;

[0070] Take the average value of the sum of all elements of the similarity matrix as the apparent affinity, and calculate the affinity of the trajectory coupling using the apparent affinity.

[0071] For each trajectory in the trajectory coupling, within the co-existing time period, a series of apparent features are extracted in chronological order. The extracted apparent features are arranged in chronological order to form a matrix. Each row of the matrix represents the apparent features at a time point, and each trajectory corresponds to an apparent feature matrix. For any two trajectories in the trajectory coupling, their apparent feature matrices are respectively taken out, and the similarity between the two apparent feature matrices is calculated. The similarity calculation methods include but are not limited to cosine similarity, Euclidean distance, dynamic time warping, Pearson correlation coefficient, etc. The similarity values between each pair of trajectories are stored in an N×N matrix to obtain a similarity matrix. The element in the i-th row and j-th column of the matrix represents the similarity between the i-th trajectory and the j-th trajectory. The sum of all elements in the similarity matrix is calculated and then divided by N×N to obtain the apparent affinity. In another embodiment, the affinity of the trajectory coupling further includes trajectory smoothness affinity and / or motion continuity affinity.

[0072] Figure 5 The second embodiment of the present invention is shown. Figure 5 The shown construction turnover material trajectory tracking system includes:

[0073] A target recognition module, which is used to construct a probability occupancy map by using video images captured by multiple cameras at the construction site, establish the correspondence between the shooting area of each camera and the grid in the probability occupancy map, obtain the probability occupancy map of the camera shooting area based on the correspondence, adopt a target recognition method to obtain the heat map of the video image, and obtain the target and apparent features according to the probability occupancy map and heat map of the camera shooting area;

[0074] A single-camera trajectory acquisition module, which is used to obtain the trajectory and trajectory apparent feature set in each camera by using the target and apparent features of the video image in each camera;

[0075] A cross-camera trajectory acquisition module, which is used to obtain trajectory coupling based on the trajectories of all the cameras, obtain the sub-trajectory apparent feature set of each trajectory within the co-existing time of the trajectories in the trajectory coupling, calculate the affinity of the trajectory coupling by using the sub-trajectory apparent feature set, take the trajectory coupling as a hypergraph node to obtain a spatio-temporal view hypergraph, and obtain the trajectories of the same turnover material in different cameras through the spatio-temporal view hypergraph.

[0076] Preferably, the obtaining of the target and apparent features according to the probability occupancy map and heat map of the camera shooting area is specifically:

[0077] Convert the probability occupancy map of the camera shooting area to the same size as the heat map;

[0078] Normalize the heat map and the probability occupancy map respectively and add them bit by bit to obtain an updated heat map;

[0079] Determine the target and the apparent features of the target using the updated heat map.

[0080] Preferably, obtaining the trajectory and the set of trajectory apparent features in each camera by using the target and the apparent features of the video image in each camera specifically includes:

[0081] Obtain the predicted bounding box of the trajectory by using a Kalman filter;

[0082] Perform apparent feature matching between the predicted bounding box and the detection bounding box. For the trajectories and targets that do not match successfully, further use the IOU matching method for matching;

[0083] If the apparent feature matching or the IOU matching is successful, add the apparent features of the detection bounding box to the set of trajectory apparent features; for the trajectories that do not match successfully, if the average value of the probability occupancy map of the predicted bounding box of the trajectory is greater than the average value of the probability occupancy map of the predicted bounding box of the successfully matched trajectory, and the trajectory is in a non - confirmed state, then convert the trajectory to a confirmed state.

[0084] Preferably, calculating the affinity of trajectory coupling by using the set of sub - trajectory apparent features specifically includes:

[0085] Convert the set of sub - trajectory apparent features of the trajectory into an apparent feature matrix of the trajectory;

[0086] Calculate the similarity of the apparent feature matrix to obtain a similarity matrix, the size of the similarity matrix is N×N, and N is the number of trajectories in the trajectory coupling;

[0087] Take the average value of the sum of all elements of the similarity matrix as the apparent affinity, and calculate the affinity of trajectory coupling by using the apparent affinity.

[0088] Preferably, the affinity further includes trajectory smoothness affinity and / or motion continuity affinity.

[0089] The above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated therein. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0090] The steps of the methods or algorithms described in the embodiments of the present application can be directly embedded in hardware, software units executed by a processor, or a combination of the two. The software units can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be provided in an ASIC.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 block or multiple blocks.

[0092] Although the present application has been described in connection with specific features and their embodiments, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined by the appended claims and are considered to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A construction turnover material trajectory tracking method, characterized in that: The method comprises the following steps: A probability occupancy map is constructed using video images captured by multiple cameras at the construction site, a correspondence between each camera shooting area and a grid in the probability occupancy map is established, a probability occupancy map of the camera shooting area is obtained based on the correspondence, a heat map of the video image is obtained using a target recognition method, and the target and appearance features are obtained according to the probability occupancy map and the heat map of the camera shooting area; The target and appearance features of the video image in each camera are used to obtain the trajectory and trajectory appearance feature set in each camera; The trajectory coupling is obtained based on the trajectories of all the cameras, and the sub-trajectory apparent feature set of each trajectory within the common existence time of the trajectories in the trajectory coupling is obtained. The affinity of the trajectory coupling is calculated using the sub-trajectory apparent feature set, and the trajectory coupling is used as a hypergraph node to obtain a spatiotemporal view hypergraph. The trajectories of the same turnover material in different cameras are obtained through the spatiotemporal view hypergraph.

2. The method according to claim 1, characterized in that The target and appearance features are obtained according to the probability occupancy map and heat map of the camera shooting area, specifically: Convert the probability occupancy map of the camera shooting area to the same size as the heat map; Normalize the heat map and probability occupancy map respectively and add them bit by bit to get the updated heat map; The updated heatmap is used to determine the apparent characteristics of the target and the object.

3. The method according to claim 1, characterized in that The target and appearance features of the video image in each camera are used to obtain the trajectory and trajectory appearance feature set in each camera, specifically: Use Kalman filter to get the prediction box of the trajectory; The prediction box and the detection box are matched for apparent features. For the unmatched trajectories and targets, IOU matching is used for further matching. If the appearance feature match or IOU match is successful, the appearance feature of the detection box is added to the appearance feature set of the trajectory; for the trajectory that is not successfully matched, if the average probability occupancy map of the predicted box of the trajectory is greater than the average probability occupancy map of the predicted box of the successfully matched trajectory, and the trajectory is in an unconfirmed state, the trajectory is converted to a confirmed state.

4. The method according to claim 1, characterized in that The affinity of trajectory coupling is calculated by using the sub-trajectory apparent feature set, specifically: Converting the sub-trajectory apparent feature set of the trajectory into an apparent feature matrix of the trajectory; Calculate the similarity of the apparent feature matrix to obtain a similarity matrix, wherein the size of the similarity matrix is ​​N×N, where N is the number of trajectories in the trajectory coupling; The average value of the sum of all elements of the similarity matrix is ​​taken as the apparent affinity, and the affinity of trajectory coupling is calculated using the apparent affinity.

5. The method according to any one of claims 1 to 4, characterized in that: The affinity also includes trajectory smoothness affinity and / or motion continuity affinity.

6. A construction turnover material trajectory tracking system, characterized in that: The system includes the following modules: A target recognition module is used to construct a probability occupancy map using video images captured by multiple cameras at the construction site, establish a correspondence between each camera shooting area and a grid in the probability occupancy map, obtain a probability occupancy map of the camera shooting area based on the correspondence, obtain a heat map of the video image using a target recognition method, and obtain the target and appearance features based on the probability occupancy map and the heat map of the camera shooting area; A single camera trajectory acquisition module is used to obtain the trajectory and trajectory appearance feature set in each camera using the target and appearance features of the video image in each camera; A cross-camera trajectory acquisition module is used to obtain trajectory coupling based on the trajectories of all the cameras, obtain the sub-trajectory apparent feature set of each trajectory within the common existence time of the trajectories in the trajectory coupling, calculate the affinity of the trajectory coupling using the sub-trajectory apparent feature set, use the trajectory coupling as a hypergraph node to obtain a spatiotemporal view hypergraph, and obtain the trajectory of the same turnover material in different cameras through the spatiotemporal view hypergraph.

7. The system according to claim 6, characterized in that The target and appearance features are obtained according to the probability occupancy map and heat map of the camera shooting area, specifically: Convert the probability occupancy map of the camera shooting area to the same size as the heat map; Normalize the heat map and probability occupancy map respectively and add them bit by bit to get the updated heat map; The updated heatmap is used to determine the apparent characteristics of the target and the object.

8. The system according to claim 6, characterized in that The target and appearance features of the video image in each camera are used to obtain the trajectory and trajectory appearance feature set in each camera, specifically: Use Kalman filter to get the prediction box of the trajectory; The prediction box and the detection box are matched for apparent features. For the unmatched trajectories and targets, IOU matching is used for further matching. If the appearance feature match or IOU match is successful, the appearance feature of the detection box is added to the appearance feature set of the trajectory; for the trajectory that is not successfully matched, if the average probability occupancy map of the predicted box of the trajectory is greater than the average probability occupancy map of the predicted box of the successfully matched trajectory, and the trajectory is in an unconfirmed state, the trajectory is converted to a confirmed state.

9. The system according to claim 6, characterized in that The affinity of trajectory coupling is calculated by using the sub-trajectory apparent feature set, specifically: Converting the sub-trajectory apparent feature set of the trajectory into an apparent feature matrix of the trajectory; Calculate the similarity of the apparent feature matrix to obtain a similarity matrix, wherein the size of the similarity matrix is ​​N×N, where N is the number of trajectories in the trajectory coupling; The average value of the sum of all elements of the similarity matrix is ​​taken as the apparent affinity, and the affinity of trajectory coupling is calculated using the apparent affinity.

10. The system according to any one of claims 6 to 9, characterized in that: The affinity also includes trajectory smoothness affinity and / or motion continuity affinity.

Images