Target tracking method and device based on multi-camera coordinate conversion, computer equipment and computer program product

Abstract

The invention relates to a target tracking method and device based on multi-camera coordinate conversion, computer equipment and a computer program product. The method comprises the following steps: respectively acquiring three-dimensional coordinates of a plurality of target angular points under a master camera coordinate system and a slave camera coordinate system; target re-identification is carried out on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the target salient angular point pair in the main camera coordinate system and the three-dimensional coordinates of the target salient angular point pair in the slave camera coordinate system; converting the target salient angular point pairs of the slave camera coordinate system into a master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and fusing all target salient angular point pairs in the master camera coordinate system to obtain a target motion track; and updating a secondary coordinate transformation mapping matrix among the cameras in real time, and recalculating and fusing all the target salient angular point pairs to obtain an updated target motion track and continuously tracking. By adopting the method, the coordinate system can be automatically converted and the target motion track can be updated in real time.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target tracking method, device, computer equipment and computer program product based on multi-camera coordinate transformation. Background Art

[0002] With the continuous development of artificial intelligence technology, the intelligence level of camera target tracking technology is also increasing. Modern camera target tracking systems can achieve more advanced target detection, image recognition, target tracking and anomaly detection functions.

[0003] In a multi-camera system, target tracking technology faces more challenges and opportunities. Through the collaborative work of multiple cameras, all-round, multi-angle tracking and positioning of the target object can be achieved. With a multi-camera solution, each perspective is attached to its own coordinate system. If it is combined for detection and tracking, it involves the problem of unified coordinate system. This problem can be solved by calibrating the camera, converting the coordinate system, and dynamically updating the strategy. However, once the relative position between the cameras changes, the original calibration results are no longer applicable and need to be recalibrated. At this time, it is necessary to recalibrate the camera and convert the coordinate system to ensure the accuracy and reliability of target tracking. This usually requires the design of a mechanism that can automatically detect changes in camera position and trigger recalibration. Summary of the invention

[0004] Based on this, it is necessary to provide a target tracking method, device, computer equipment and program product that can automatically convert the coordinate system and update the target motion trajectory in real time to address the above technical problems.

[0005] On the one hand, a target tracking method based on multi-camera coordinate transformation is provided, the method comprising:

[0006] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0007] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0008] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0009] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the salient corner point pairs of the target after fusion are recalculated to obtain the updated target motion trajectory and continue tracking.

[0010] In one embodiment, it also includes:

[0011] Get the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively, including:

[0012] The main camera and the slave camera collect the two-dimensional image and three-dimensional cloud point information of the target respectively;

[0013] Inputting the two-dimensional image into a pre-trained target detection deep neural network model to obtain a position bounding box of the target in the two-dimensional image;

[0014] The target position bounding box of the two-dimensional image corresponds to the target area in the three-dimensional cloud point information of the camera;

[0015] The three-dimensional point cloud information in the target area is input into a pre-trained key point detection deep neural network model, a group of corner points corresponding to the target are extracted, and the three-dimensional coordinates of the group of corner points in the current camera coordinate system are obtained.

[0016] In one of the embodiments, the method further includes performing target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of target salient corner points in the main camera coordinate system and a pair of target salient corner points in the slave camera coordinate system, including:

[0017] Clipping the target two-dimensional image according to the target position boundary box to obtain the main camera target area image and the slave camera target area image;

[0018] Obtaining a main camera target area image feature and a slave camera target area image feature according to the target area image and the target corner point;

[0019] Obtaining the same target corner point of the main camera and the slave camera according to the distance between the target area image feature of the main camera and the target area image feature of the slave camera;

[0020] Filtering out the target corner points with lower confidence levels to obtain the significant corner points of the master camera and the significant corner points of the slave camera;

[0021] The significant corner point pairs of the target corresponding positions are matched according to the significant corner points of the main camera and the significant corner points of the slave camera, so as to obtain the three-dimensional coordinates of the significant corner point pairs in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

[0022] In one embodiment, filtering out the target corner points with low confidence comprises:

[0023] Calculate the response values ​​of all target corner points through the Harris corner detection algorithm;

[0024] Setting response thresholds;

[0025] Comparing the response value of the target corner point with the response threshold;

[0026] If the response value of the target corner point is lower than the response threshold, the target corner point is filtered out.

[0027] In one embodiment, the method further includes converting the target salient corner point pair in the slave camera coordinate system to the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, including:

[0028] The homogeneous coordinate transformation mapping formula from the i-th slave camera coordinate system to the master camera coordinate system is as follows:

[0029]

[0030] Among them, Q i is the homogeneous coordinate transformation mapping matrix from the i-th camera,

[0031]

[0032] is the homogeneous coordinate corresponding to a point in the i-th camera coordinate system,

[0033]

[0034] The coordinates in the main camera coordinate system after conversion.

[0035] In one embodiment, the method further includes fusing all pairs of salient corner points of the target in the main camera coordinate system to obtain a target motion trajectory, including:

[0036] Calculate the mean of all coordinates in the main camera coordinate system coordinate set;

[0037] Calculate the distance between all transformed coordinates and the mean coordinate in the main camera coordinate system coordinate set;

[0038] Filter out abnormal outlier coordinates to obtain the filtered coordinate set of the significant corner point in the main camera coordinate system;

[0039] All coordinates in the filtered coordinate set are weighted averaged to obtain the fused coordinates of the significant corner point in the main camera coordinate system.

[0040] In one embodiment, the method further includes updating the coordinate transformation mapping matrix between the cameras in real time, and recalculating all the fused significant corner point pairs of the target to obtain an updated target motion trajectory and continuously track the target, including:

[0041] The main camera and the slave camera collect the three-dimensional coordinate information of all the targets to be tracked in real time;

[0042] Set time thresholds;

[0043] If the same target to be tracked is detected, after the time threshold is reached, the homogeneous coordinate transformation mapping matrix from the slave camera coordinate system to the master camera coordinate system is recalculated and updated;

[0044] Recalculate the fusion coordinates of the significant corner points of the target to be tracked in the main camera coordinate system according to the updated homogeneous coordinate transformation mapping matrix;

[0045] Get the updated target's motion trajectory and continue tracking it.

[0046] On the other hand, a target tracking device applied to a multi-camera spatial coordinate system is provided, the device comprising:

[0047] A coordinate acquisition module is used to obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system;

[0048] A corner point matching module is used to perform target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of significant corner points of the target in the main camera coordinate system and significant corner points of the target in the slave camera coordinate system;

[0049] A coordinate conversion module, used to convert the target significant corner point pairs in the slave camera coordinate system to the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and fuse all target significant corner point pairs in the master camera coordinate system to obtain a target motion trajectory;

[0050] The update calculation module is used to update the coordinate transformation mapping matrix between the cameras in real time, and recalculate all the salient corner point pairs of the target after fusion to obtain the updated target motion trajectory and continue tracking.

[0051] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0052] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0053] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0054] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0055] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the salient corner point pairs of the target after fusion are recalculated to obtain the updated target motion trajectory and continue tracking.

[0056] In another aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

[0057] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0058] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0059] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0060] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the salient corner point pairs of the target after fusion are recalculated to obtain the updated target motion trajectory and continue tracking.

[0061] The target tracking method, device, computer equipment and storage medium obtain the coordinates of the target corner points through the main camera and the slave camera. The targets in the main camera and slave camera coordinate systems are re-identified. Through the re-identification technology, different targets can be accurately distinguished to confirm the corresponding relationship of the same target under different cameras, and the target significant corner point pairs are extracted from them. These significant corner point pairs remain relatively stable during the target movement, which is helpful for subsequent target tracking and trajectory reconstruction. The significant corner point pairs in the slave camera coordinate system are converted to the main camera coordinate system using the homogeneous coordinate transformation mapping matrix. The significant corner point pairs of all targets in the main camera coordinate system are fused to form the target motion trajectory. Through coordinate transformation, seamless fusion of multi-camera data is achieved, providing a unified basis for subsequent analysis; fusion of data under multiple camera perspectives can eliminate errors under a single perspective and improve the accuracy of the target motion trajectory. After a fixed interval, the homogeneous coordinate transformation mapping matrix between the cameras is updated to adapt to changes in camera position or posture, and the fused target significant corner point pairs are recalculated to obtain an updated target motion trajectory, and continuous tracking is performed.

[0062] This method can adaptively calculate and establish the mapping relationship between camera coordinate systems of different perspectives and update it in real time, thereby achieving the organic integration of the position information of the target to be tracked under different perspectives. Even if the relative position between the cameras changes, there is no need to recalibrate, which saves unnecessary operation processes and improves the application experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A diagram showing an application environment of a target tracking method in an embodiment;

[0064] Figure 2 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0065] Figure 3 is a structural block diagram of a target tracking device of a multi-camera spatial coordinate system in one embodiment;

[0066] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] The target tracking method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the detection terminal 101 communicates with the server 102 through the network. When the detection terminal 101 recognizes that a human body enters the channel, the collected information is sent to the server 102, and the server 102 returns the detection result after calculation. Among them, the detection terminal 101 can be but not limited to a camera, an image acquisition device, and a recognition lens, and the server 102 can be implemented by an independent server or a server cluster composed of multiple servers.

[0069] In one embodiment, Figure 2 As shown in FIG. 1 , a target tracking method based on multi-camera coordinate transformation is provided. The method is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0070] Step 202, respectively obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system.

[0071] Among them, there is only one main camera, which is located at the front perspective of the target. There are multiple slave cameras, which are located at the side or top perspective of the target, etc. The coordinate system with the main camera as the coordinate origin is the main camera coordinate system, and there is only one. The coordinate system with the slave camera as the coordinate origin is the slave camera coordinate system, and there are multiple.

[0072] Specifically, the information of the target corner points is collected by the main camera and the slave camera to obtain the three-dimensional coordinates of the target corner points. Through the multi-camera perspective, more comprehensive target information can be obtained, reducing occlusion and errors under a single perspective.

[0073] Step 204 , re-identify the target in the master camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the target's salient corner point pairs in the master camera coordinate system and the three-dimensional coordinates of the target's salient corner point pairs in the slave camera coordinate system.

[0074] Among them, target re-identification is a computer vision task that aims to continuously and stably identify specific target objects, such as pedestrians, vehicles, etc., from different images or video sequences taken by multiple non-overlapping cameras.

[0075] Specifically, by re-identifying the target, the target feature vector, i.e., the salient corner points, can be extracted; and the salient corner point pairs of different cameras can be matched and extracted. By extracting the salient corner point pairs, redundant data is reduced, and the efficiency and accuracy of subsequent processing are improved.

[0076] Step 206: transform the target salient corner point pairs in the slave camera coordinate system into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and fuse all target salient corner point pairs in the master camera coordinate system to obtain a target motion trajectory.

[0077] Among them, the homogeneous coordinate mapping transformation matrix is ​​a mathematical tool widely used in computer graphics, robotics and many other fields. It is used to represent the transformation of a point, vector or coordinate system in space from one state to another.

[0078] Homogeneous Coordinates: In two or three dimensional space, traditional coordinate representations such as (x,y) or (x,y,z) are non-homogeneous. Homogeneous coordinates extend these coordinates by adding an extra dimension, usually w, with w≠0, to form a form such as (x,y,1) or (x,y,z,1). This extra dimension plays an important role in the transformation process, allowing translation operations to be implemented through matrix multiplication. Homogeneous Transformation Matrix: A homogeneous transformation matrix is ​​a special matrix, usually of dimension 4x4, in three dimensional space. This matrix contains all the information for transformations such as rotation, translation, and possibly scaling, which can transform a point or vector from one coordinate system to another.

[0079] Specifically, by converting the significant target corner point pairs in the slave camera coordinate system to the master camera coordinate system and fusing data from multiple camera perspectives, errors in a single perspective can be eliminated and the accuracy of the motion trajectory can be improved.

[0080] Step 208, updating the coordinate transformation mapping matrix between the cameras in real time, and recalculating all the fused significant corner point pairs of the target, obtaining the updated target motion trajectory and continuously tracking it.

[0081] Specifically, the coordinate information collected by each camera is used to update the homogeneous coordinate transformation mapping matrix between the cameras in real time to adapt to the change of the position or posture of the camera. The fused target salient corner point pairs are recalculated to obtain the updated target motion trajectory, and the tracking is continued.

[0082] In the above target tracking method, the coordinates of the target corner points are obtained through the main camera and the slave camera. The targets in the main camera and slave camera coordinate systems are re-identified. Through the re-identification technology, different targets can be accurately distinguished to confirm the corresponding relationship of the same target under different cameras, and the target salient corner point pairs are extracted from them. These salient corner point pairs remain relatively stable during the target movement, which is helpful for subsequent target tracking and trajectory reconstruction. The salient corner point pairs in the slave camera coordinate system are converted to the main camera coordinate system using the homogeneous coordinate transformation mapping matrix. The salient corner point pairs of all targets in the main camera coordinate system are fused to form the target motion trajectory. Through coordinate transformation, seamless fusion of multi-camera data is achieved, providing a unified basis for subsequent analysis; fusion of data from multiple camera perspectives can eliminate errors in a single perspective and improve the accuracy of the target motion trajectory. After a fixed interval, the homogeneous coordinate transformation mapping matrix between the cameras is updated to adapt to changes in camera position or posture, and the fused target salient corner point pairs are recalculated to obtain the updated target motion trajectory, and continuous tracking is performed.

[0083] This method can adaptively calculate and establish the mapping relationship between camera coordinate systems of different perspectives and update it in real time, thereby achieving the organic integration of the position information of the target to be tracked under different perspectives. Even if the relative position between the cameras changes, there is no need to recalibrate, which saves unnecessary operation processes and improves the application experience.

[0084] In one implementation, the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system are obtained respectively, including:

[0085] The main camera and the slave camera collect the two-dimensional image and three-dimensional cloud point information of the target respectively;

[0086] Inputting the two-dimensional image into a pre-trained target detection deep neural network model to obtain a position bounding box of the target in the two-dimensional image;

[0087] The target position bounding box of the two-dimensional image corresponds to the target area in the three-dimensional cloud point information of the camera;

[0088] The three-dimensional point cloud information in the target area is input into a pre-trained key point detection deep neural network model, a group of corner points corresponding to the target are extracted, and the three-dimensional coordinates of the group of corner points in the current camera coordinate system are obtained.

[0089] Among them, the target detection deep neural network model is mainly used to automatically find and identify the location and category of a specific target in an image or video, and mark its location with a bounding box, while giving the category information of the object. The key point detection deep neural network model is mainly used to automatically detect and locate specific key points in an image or video, such as the joints of the human body and the locations of facial features. These features are used to predict the location of key points in the image.

[0090] Specifically, the two-dimensional image information of the target is collected by the camera and input into the pre-trained target detection deep neural network model. The bounding box of the target is marked through model training to obtain the position bounding box of the target. The obtained two-dimensional image bounding box is mapped to the collected three-dimensional cloud point information, and the three-dimensional cloud point target area is obtained after matching. The three-dimensional cloud point information in the obtained target area is input into the pre-trained key point detection deep neural network model, and a group of significant human skeleton corner points of the corresponding target, i.e., significant corner points, are extracted through the model. The three-dimensional coordinates of the group of significant corner points are obtained.

[0091] By mapping the target position bounding box in the two-dimensional image to the three-dimensional cloud point information, accurate conversion from two-dimensional to three-dimensional is achieved. This helps to accurately locate the target in three-dimensional space and provides a basis for subsequent three-dimensional processing and analysis. Combining two-dimensional images and three-dimensional point cloud information, the system is able to obtain more comprehensive and in-depth perception capabilities. Two-dimensional images provide rich texture and color information, while three-dimensional point clouds provide accurate spatial position and shape information. This fusion of multi-source data enables the system to better understand and analyze the target. The entire information collection process is highly automated, from image acquisition to target detection and three-dimensional shape extraction, which can be completed in a short time. Efficient, accurate and comprehensive perception and analysis of the target is achieved.

[0092] In one implementation, performing target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of significant corner points of the target in the main camera coordinate system and a pair of significant corner points of the target in the slave camera coordinate system includes:

[0093] Step 302 : clipping the target two-dimensional image according to the target position boundary box to obtain the master camera target area image and the slave camera target area image.

[0094] Specifically, through cropping, only the target area of ​​interest in the image is retained, reducing the computational complexity of subsequent image processing, such as feature extraction and target recognition. Cropping removes background information in the image that is not related to the target, reducing the interference of this background information on the target detection algorithm and improving the accuracy of target detection.

[0095] Step 304 , obtaining the main camera target area image features and the slave camera target area image features according to the target area image and the target corner points.

[0096] Through the target area image, local features closely related to the target can be accurately extracted. Corner points, as important local features in the image, usually contain rich information, such as direction, scale, etc. Detecting corner points in the target area and extracting their features can further improve the accuracy and robustness of feature description.

[0097] Step 306 : obtaining the same target corner point of the master camera and the slave camera according to the distance between the target area image feature of the master camera and the target area image feature of the slave camera.

[0098] Specifically, corner detection algorithms such as Harris corner detection, SIFT, SURF, etc. are used to detect corners in the target area image and extract their descriptors such as feature vectors. The features extracted by the main camera and the slave camera are matched. This is done by comparing the similarity between feature descriptors, such as using metrics such as Euclidean distance and Hamming distance. The same target corners are obtained from the perspective of the main camera and the perspective of the slave camera.

[0099] Step 308 , filtering out the target corner points with lower confidence, and obtaining the significant corner points of the master camera and the significant corner points of the slave camera.

[0100] Specifically, filtering out corner points with low confidence reduces the amount of data that needs to be processed in subsequent processing, thereby saving computing resources. As the amount of data processed is reduced, the execution speed of the algorithm will also increase accordingly, speeding up the entire processing flow.

[0101] Step 310, matching the significant corner point pairs of the target corresponding positions according to the significant corner points of the master camera and the significant corner points of the slave camera, and obtaining the three-dimensional coordinates of the significant corner point pairs in the master camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

[0102] This image processing process significantly improves the efficiency and accuracy of data processing through steps such as cropping target areas, extracting features, matching corner points, filtering out low-confidence corner points, and calculating three-dimensional coordinates. It also enhances the robustness and stability of the system, helps to achieve the fusion of multi-perspective information, and improves the accuracy and reliability of three-dimensional reconstruction and positioning.

[0103] In one embodiment, filtering out the target corner points with low confidence includes:

[0104] Calculate the response values ​​of all target corner points through the Harris corner detection algorithm;

[0105] Setting response thresholds;

[0106] Comparing the response value of the target corner point with the response threshold;

[0107] If the response value of the target corner point is lower than the response threshold, the target corner point is filtered out.

[0108] Among them, the Harris corner detection algorithm is a corner detection algorithm based on the change of image grayscale intensity. It calculates the local autocorrelation matrix of the window around each pixel in the image and judges whether the point is a corner point based on it. Corner points are usually points in the image with large grayscale changes in two orthogonal directions. Response value: The Harris algorithm assigns a response value to each detected candidate corner point, which reflects the strength or credibility of the point as a corner point. Generally, the higher the response value, the higher the credibility of the point as a corner point.

[0109] Specifically, the Harris corner detection algorithm is used to effectively identify potential corner points in the image, and by setting a response threshold and comparing the response value of the target corner point, if the response value of the target corner point is lower than the response threshold, the corner points with low current reliability or that may be caused by noise or image blur are removed, thereby improving the accuracy and reliability of corner detection.

[0110] In one embodiment, the target significant corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target significant corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory, including:

[0111] A homogeneous coordinate transformation mapping matrix is ​​set under each camera;

[0112] Calculate the mean of all coordinates in the main camera coordinate system coordinate set;

[0113] Calculate the distance between all transformed coordinates and the mean coordinate in the main camera coordinate system coordinate set;

[0114] Filter out abnormal outlier coordinates to obtain the filtered coordinate set of the significant corner point in the main camera coordinate system;

[0115] All coordinates in the filtered coordinate set are weighted averaged to obtain the fused coordinates of the significant corner point in the main camera coordinate system.

[0116] Specifically, the homogeneous coordinate transformation mapping formula from the i-th slave camera coordinate system to the master camera coordinate system is as follows:

[0117]

[0118] in,

[0119]

[0120] Represents the coordinates of a point on the x, y, and z axes from the i-th camera coordinate system;

[0121]

[0122] Represents the corresponding homogeneous coordinates

[0123]

[0124] Indicates the coordinates of the corresponding point on the x, y, and z axes in the main camera coordinate system;

[0125]

[0126] represents the corresponding homogeneous coordinates;

[0127]

[0128] Represents the homogeneous coordinate transformation mapping matrix from the i-th slave camera coordinate system to the master camera coordinate system;

[0129]

[0130] represents the translation transformation from the i-th slave camera coordinate system to the master camera coordinate system, are the translation amounts in the x, y, and z axis directions respectively;

[0131]

[0132] represents the rotation transformation from the i-th slave camera coordinate system to the master camera coordinate system, α i , β i , γ i are the rotation angles around the x, y, and z axes respectively;

[0133]

[0134] represents the rotation transformation matrix from the i-th slave camera coordinate system to the master camera coordinate system around the x-axis, α i is the rotation angle around the x-axis;

[0135]

[0136] represents the rotation transformation matrix around the y-axis from the i-th slave camera coordinate system to the master camera coordinate system, β i is the rotation angle around the y axis;

[0137]

[0138] represents the rotation transformation matrix from the i-th slave camera coordinate system to the master camera coordinate system around the z axis, γ i is the rotation angle around the z axis;

[0139] Then the formula:

[0140] is the homogeneous coordinate transformation mapping formula from the i-th slave camera coordinate system to the master camera coordinate system.

[0141] Assume the three-dimensional coordinates of the significant bone corner point pairs in the front main camera coordinate system and the left slave camera coordinate system, and assume that a total of K pairs of significant bone corner points are matched in the two sets of significant bone corner points. Then the coordinates of the kth pair of significant bone corner points in the front main camera coordinate system and the left slave camera coordinate system can be recorded as:

[0142]

[0143] Substitute the coordinates of the K pairs of significant bone corner points in the front main camera coordinate system and the left slave camera coordinate system into the homogeneous coordinate transformation mapping formula from the left slave camera coordinate system to the front main camera coordinate system, and obtain K sets of equations:

[0144]

[0145] Based on the least squares method, the three translation transformation parameters from the left slave camera coordinate system to the front main camera coordinate system are solved respectively. With three rotation transformation parameters (α 左 , β 左 , γ 左 ), and obtain the corresponding homogeneous coordinate transformation mapping matrix (Q 左 );

[0146] The same is true for the right side from the camera, so I won’t go into details here;

[0147] The two solved left and right homogeneous coordinate transformation mapping matrices transform the coordinates of each significant bone corner point in the left and right camera coordinate systems to the front main camera coordinate system;

[0148] That is, convert the left and right coordinates from the camera coordinate system:

[0149]

[0150] Calculate the conversion coordinates from the left and right camera coordinate systems to the front main camera coordinate system, and then integrate the coordinates of the front main camera coordinate system to obtain the coordinate set of the significant bone corner points in the front main camera coordinate system:

[0151]

[0152] For each significant bone corner point, calculate the mean of all coordinates in the coordinate set of the front main camera coordinate system, including its own coordinates and all transformed coordinates:

[0153]

[0154] On this basis, the distance between all transformed coordinates and the mean coordinates in the coordinate set of the front main camera coordinate system is calculated:

[0155]

[0156] Filter out the abnormal outlier coordinates whose offset distance exceeds a predetermined threshold, such as 20 cm, and obtain the filtered coordinate set of the significant bone corner points in the front main camera coordinate system:

[0157]

[0158] Assume that after filtering, at least one set of left or right coordinates from the camera coordinate system remains;

[0159] Perform weighted averaging on all coordinates in the filtered coordinate set, and finally obtain the fused coordinates of the significant bone corner point in the front main camera coordinate system:

[0160]

[0161] The above calculation is performed on all significant bone corner points of the person to be tracked to obtain the fusion coordinates of all significant bone corner points in the front main camera coordinate system, which are used to characterize the motion trajectory of the person to be tracked in the current security inspection channel with the front main camera as the coordinate origin.

[0162] In one embodiment, the coordinate transformation mapping matrix between the cameras is updated in real time, and all the salient corner point pairs of the target after fusion are recalculated to obtain the updated target motion trajectory and continue tracking, including:

[0163] The main camera and the slave camera collect the three-dimensional coordinate information of all the targets to be tracked in real time;

[0164] Set time thresholds;

[0165] If the same target to be tracked is detected, after the time threshold is reached, the homogeneous coordinate transformation mapping matrix from the slave camera coordinate system to the master camera coordinate system is recalculated and updated;

[0166] Recalculate the fusion coordinates of the significant corner points of the target to be tracked in the main camera coordinate system according to the updated homogeneous coordinate transformation mapping matrix;

[0167] Get the updated target's motion trajectory and continue tracking it.

[0168] Specifically, the main camera and the slave camera continuously collect the image target position bounding box and the three-dimensional coordinates of the significant bone corner points of all the persons to be tracked in the scene in each camera perspective. If a pedestrian is detected in the channel, the homogeneous coordinate transformation mapping matrix from the camera coordinate system to the front main camera coordinate system is recalculated and updated according to the above method at a predetermined interval, such as 2 minutes.

[0169] The fusion coordinates of all significant skeletal corner points of all targets in the front main camera coordinate system are recalculated through the updated homogeneous coordinate transformation mapping matrix. Combined with the identity information of the person to be tracked identified based on the front main camera image, the motion trajectory of each person to be tracked in the current security inspection channel is represented, thereby achieving structured stable and continuous tracking.

[0170] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0171] In one embodiment, Figure 3 As shown, a target tracking device applied to a multi-camera space coordinate system is provided, including: a coordinate acquisition module 402, a corner point matching module 404, a coordinate conversion module 406 and an update calculation module 408, wherein:

[0172] A coordinate acquisition module 402 is used to acquire the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system;

[0173] A corner point matching module 404 is used to perform target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of significant corner points of the target in the main camera coordinate system and significant corner points of the target in the slave camera coordinate system;

[0174] A coordinate conversion module 406 is used to convert the target significant corner point pairs in the slave camera coordinate system to the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and fuse all target significant corner point pairs in the master camera coordinate system to obtain a target motion trajectory;

[0175] The update calculation module 408 is used to update the coordinate transformation mapping matrix between the cameras in real time, and recalculate all the fused significant corner point pairs of the target to obtain the updated target motion trajectory and continue tracking.

[0176] In one embodiment, the coordinate acquisition module 402 includes:

[0177] The main camera and the slave camera are used to collect the two-dimensional image and three-dimensional cloud point information of the target respectively;

[0178] For inputting the two-dimensional image into a pre-trained target detection deep neural network model to obtain a position bounding box of the target in the two-dimensional image;

[0179] For making the target position bounding box of the two-dimensional image correspond to the target area in the three-dimensional cloud point information of the camera;

[0180] It is used to input the three-dimensional point cloud information in the target area into a pre-trained key point detection deep neural network model, extract a group of corner points corresponding to the target, and obtain the three-dimensional coordinates of the group of corner points in the current camera coordinate system.

[0181] In one embodiment, the update calculation module 408 includes:

[0182] Used to calculate the mean of all coordinates in the main camera coordinate system coordinate set.

[0183] Used to calculate the distance between all transformed coordinates and the mean coordinate in the main camera coordinate system coordinate set.

[0184] It is used to filter out abnormal outlier coordinates and obtain the filtered coordinate set of the significant corner point in the main camera coordinate system.

[0185] It is used to perform weighted averaging on all coordinates in the filtered coordinate set to obtain the fused coordinates of the significant corner point in the main camera coordinate system.

[0186] The main camera and the slave camera are used to collect the three-dimensional coordinate information of all targets to be tracked in real time.

[0187] Used to set time threshold.

[0188] Used to, if the same target to be tracked is detected, after reaching the time threshold, recalculate and update the homogeneous coordinate transformation mapping matrix from the slave camera coordinate system to the master camera coordinate system.

[0189] Used to recalculate the fusion coordinates of the salient corner points of the target to be tracked in the main camera coordinate system according to the updated homogeneous coordinate transformation mapping matrix.

[0190] Used to obtain the updated motion trajectory of the target and continue tracking it.

[0191] For the specific definition of a target tracking device applied to a multi-camera spatial coordinate system, please refer to the definition of the target tracking method above, which will not be repeated here. Each module in the above-mentioned target tracking device applied to a multi-camera spatial coordinate system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0192] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target tracking method based on multi-camera coordinate transformation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0193] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0194] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0195] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0196] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0197] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0198] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the significant corner point pairs of the target are recalculated and fused to obtain the updated target motion trajectory and continue tracking.

[0199] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0200] Clipping the target two-dimensional image according to the target position boundary box to obtain the main camera target area image and the slave camera target area image;

[0201] Obtaining a main camera target area image feature and a slave camera target area image feature according to the target area image and the target corner point;

[0202] Obtaining the same target corner point of the main camera and the slave camera according to the distance between the target area image feature of the main camera and the target area image feature of the slave camera;

[0203] Filtering out the target corner points with lower confidence levels to obtain the significant corner points of the master camera and the significant corner points of the slave camera;

[0204] The significant corner point pairs of the target corresponding positions are matched according to the significant corner points of the main camera and the significant corner points of the slave camera, so as to obtain the three-dimensional coordinates of the significant corner point pairs in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

[0205] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0206] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0207] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0208] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0209] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the significant corner point pairs of the target are recalculated and fused to obtain the updated target motion trajectory and continue tracking.

[0210] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0211] Clipping the target two-dimensional image according to the target position boundary box to obtain the main camera target area image and the slave camera target area image;

[0212] Obtaining a main camera target area image feature and a slave camera target area image feature according to the target area image and the target corner point;

[0213] Obtaining the same target corner point of the main camera and the slave camera according to the distance between the target area image feature of the main camera and the target area image feature of the slave camera;

[0214] Filtering out the target corner points with lower confidence levels to obtain the significant corner points of the master camera and the significant corner points of the slave camera;

[0215] The significant corner point pairs of the target corresponding positions are matched according to the significant corner points of the main camera and the significant corner points of the slave camera, so as to obtain the three-dimensional coordinates of the significant corner point pairs in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

[0216] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0217] Clipping the target two-dimensional image according to the target position boundary box to obtain the main camera target area image and the slave camera target area image;

[0218] Obtaining a main camera target area image feature and a slave camera target area image feature according to the target area image and the target corner point;

[0219] Obtaining the same target corner point of the main camera and the slave camera according to the distance between the target area image feature of the main camera and the target area image feature of the slave camera;

[0220] Filtering out the target corner points with lower confidence levels to obtain the significant corner points of the master camera and the significant corner points of the slave camera;

[0221] The significant corner point pairs of the target corresponding positions are matched according to the significant corner points of the main camera and the significant corner points of the slave camera, so as to obtain the three-dimensional coordinates of the significant corner point pairs in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

[0222] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0223] Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively;

[0224] Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system;

[0225] The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory;

[0226] The coordinate transformation mapping matrix between the cameras is updated in real time, and all the significant corner point pairs of the target are recalculated and fused to obtain the updated target motion trajectory and continue tracking.

[0227] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0228] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0229] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A target tracking method based on multi-camera coordinate transformation, characterized in that: The method is applied to a multi-camera space coordinate system, comprising: Obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively; Re-identify the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain the three-dimensional coordinates of the significant corner point pairs of the target in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs of the target in the slave camera coordinate system; The target salient corner point pairs in the slave camera coordinate system are transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and all target salient corner point pairs in the master camera coordinate system are integrated to obtain the target motion trajectory; The coordinate transformation mapping matrix between the cameras is updated in real time, and all the significant corner point pairs of the target are recalculated and fused to obtain the updated target motion trajectory and continue tracking.

2. The target tracking method based on multi-camera coordinate transformation according to claim 1, characterized in that: Get the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system respectively, including: The main camera and the slave camera collect the two-dimensional image and three-dimensional cloud point information of the target respectively; Inputting the two-dimensional image into a pre-trained target detection deep neural network model to obtain a position bounding box of the target in the two-dimensional image; The target position bounding box of the two-dimensional image corresponds to the target area in the three-dimensional cloud point information of the camera; The three-dimensional point cloud information in the target area is input into a pre-trained key point detection deep neural network model, a group of corner points corresponding to the target are extracted, and the three-dimensional coordinates of the group of corner points in the current camera coordinate system are obtained.

3. The target tracking method based on multi-camera coordinate transformation according to claim 2, characterized in that: Performing target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of target salient corner points in the main camera coordinate system and a pair of target salient corner points in the slave camera coordinate system, including: Clipping the target two-dimensional image according to the target position boundary box to obtain the main camera target area image and the slave camera target area image; Obtaining a main camera target area image feature and a slave camera target area image feature according to the target area image and the target corner point; Obtaining the same target corner point of the main camera and the slave camera according to the distance between the target area image feature of the main camera and the target area image feature of the slave camera; Filtering out the target corner points with lower confidence levels to obtain the significant corner points of the master camera and the significant corner points of the slave camera; The significant corner point pairs of the target corresponding positions are matched according to the significant corner points of the main camera and the significant corner points of the slave camera, so as to obtain the three-dimensional coordinates of the significant corner point pairs in the main camera coordinate system and the three-dimensional coordinates of the significant corner point pairs in the slave camera coordinate system.

4. The target tracking method based on multi-camera coordinate transformation according to claim 3, characterized in that: Filtering out the target corner points with low confidence includes: Calculate the response values ​​of all target corner points through the Harris corner detection algorithm; Setting response thresholds; Comparing the response value of the target corner point with the response threshold; If the response value of the target corner point is lower than the response threshold, the target corner point is filtered out.

5. The target tracking method based on multi-camera coordinate transformation according to claim 1, characterized in that: The target salient corner point pair in the slave camera coordinate system is transformed into the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, including: The homogeneous coordinate transformation mapping formula from the i-th slave camera coordinate system to the master camera coordinate system is as follows: Among them, Q i is the homogeneous coordinate transformation mapping matrix from the i-th camera, is the homogeneous coordinate corresponding to a point in the i-th camera coordinate system, The coordinates in the main camera coordinate system after conversion.

6. The target tracking method based on multi-camera coordinate transformation according to claim 1, characterized in that: All the salient corner point pairs of the target in the main camera coordinate system are integrated to obtain the target motion trajectory, including: Calculate the mean of all coordinates in the main camera coordinate system coordinate set; Calculate the distance between all transformed coordinates and the mean coordinate in the main camera coordinate system coordinate set; Filter out abnormal outlier coordinates to obtain the filtered coordinate set of the significant corner point in the main camera coordinate system; All coordinates in the filtered coordinate set are weighted averaged to obtain the fused coordinates of the significant corner point in the main camera coordinate system.

7. The target tracking method based on multi-camera coordinate transformation according to claim 1, characterized in that: Update the coordinate transformation mapping matrix between the cameras in real time, and recalculate and fuse all the significant corner point pairs of the target to obtain the updated target motion trajectory and continue tracking, including: The main camera and the slave camera collect the three-dimensional coordinate information of all the targets to be tracked in real time; Set time thresholds; If the same target to be tracked is detected, after the time threshold is reached, the homogeneous coordinate transformation mapping matrix from the slave camera coordinate system to the master camera coordinate system is recalculated and updated; Recalculate the fusion coordinates of the significant corner points of the target to be tracked in the main camera coordinate system according to the updated homogeneous coordinate transformation mapping matrix; Get the updated target's motion trajectory and continue tracking it.

8. A target tracking device applied to a multi-camera spatial coordinate system, characterized in that: The device comprises: A coordinate acquisition module is used to obtain the three-dimensional coordinates of several target corner points in the main camera coordinate system and the slave camera coordinate system; A corner point matching module is used to perform target re-identification on the target in the main camera coordinate system and the target in the slave camera coordinate system to obtain a pair of significant corner points of the target in the main camera coordinate system and significant corner points of the target in the slave camera coordinate system; A coordinate conversion module, used to convert the target significant corner point pairs in the slave camera coordinate system to the master camera coordinate system through a homogeneous coordinate transformation mapping matrix, and fuse all target significant corner point pairs in the master camera coordinate system to obtain a target motion trajectory; The update calculation module is used to update the coordinate transformation mapping matrix between the cameras in real time, and recalculate all the salient corner point pairs of the target after fusion to obtain the updated target motion trajectory and continue tracking.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images