Multi-view combined tracking method for complex target in industrial production scene

Abstract

The invention relates to a multi-view combined tracking method for a complex target in an industrial production scene, and the method comprises the following steps: S1, carrying out the calibration of internal parameters and external parameters of all cameras, constructing a physical space 3D model of a target production region, and forming a factory coordinate system; s2, collecting a multi-view image frame through the calibrated camera, and carrying out the preprocessing of the multi-view image frame; s3, detecting the position of each frame of target by using a deep learning detection model Detectron2, performing monocular inter-frame association on the detected target to obtain a local trajectory, and extracting a depth feature vector for each target; s4, when the same target appears at a plurality of visual angles, fusing the space coordinates to obtain accurate 3D positioning, and obtaining a physical 3D space coordinate candidate set of each actual target; and S5, according to the physical 3D space coordinate candidate set, constructing a matching candidate set, and adopting a joint matching algorithm to realize trajectory merging to obtain a complete 3D trajectory. According to the invention, multi-target efficient and multi-view collaborative accurate tracking in a complex industrial scene is realized.

Description

Technical Field

[0001] This invention relates to the field of video processing, and more particularly to a multi-view joint tracking method for complex targets in industrial production scenarios. Background Technology

[0002] Modern industrial production environments are becoming increasingly complex, leading to a growing demand for automation, precision, and intelligence. Against the backdrop of smart manufacturing, Industry 4.0, and flexible production lines, real-time and accurate tracking of production objects (such as workpieces, AGVs, robotic arms, and personnel) within factory workshops has become a key foundational technology for improving production efficiency, ensuring safe production, and enabling intelligent decision-making.

[0003] Traditional industrial target tracking methods mainly rely on a single sensor or a limited field of view, which often face the following challenges: First, in factory environments with dense equipment, complex lighting, and frequent occlusion, targets are easily lost or confused under a single field of view; second, industrial objects are diverse and have similar appearances, making it difficult to maintain stability in identification based on a single feature; third, the lack of accurate three-dimensional spatial positioning makes it impossible to support high-precision navigation and fine operation; and fourth, a single system cannot achieve full-process, full-domain monitoring, resulting in data fragmentation. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a multi-view joint tracking method for complex targets in industrial production scenarios, achieving efficient and accurate tracking of multiple targets in complex industrial settings through multi-view collaboration.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multi-view joint tracking method for complex targets in industrial production scenarios includes the following steps: S1: Calibrate the intrinsic and extrinsic parameters of all cameras, construct a 3D physical space model of the target production area, form a factory coordinate system, and obtain the geometric mapping parameters between each camera and the physical space; S2: Acquire multi-view image frames through the calibrated camera, obtain multi-view synchronized raw image frames with timestamps, and perform preprocessing to obtain preprocessed image frames; S3: Based on the preprocessed image frames, use the deep learning detection model Detectron2 to detect the target position in each frame, perform monocular inter-frame correlation on the detected targets to obtain local trajectories, extract depth feature vectors for each target, and obtain the 2D detection boxes, local 2D trajectory IDs and appearance features of all targets in the current frame under each camera. S4: Back-project the 2D detection box of the target in the current frame to the physical 3D space of the factory. When the same target appears in multiple viewpoints, fuse the spatial coordinates to obtain accurate 3D positioning and obtain a candidate set of physical 3D spatial coordinates for each actual target. S5: Based on the candidate set of physical 3D spatial coordinates, multimodal fusion is performed using spatial distance, temporal consistency and appearance features to construct a matching candidate set. A joint matching algorithm is then used to achieve global ID allocation of the target across different cameras and time slices, and the trajectories are merged to obtain a complete 3D trajectory.

[0006] Furthermore, intrinsic and extrinsic parameters were calibrated for all cameras, as follows: The camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, using multiple checkerboard images taken from different angles by the same camera. Camera model: ; Where K is the intrinsic parameter matrix; fx, fy are focal lengths, cx, cy are principal point coordinates; rotation matrix R and translation vector t are extrinsic parameters; (X w ,Y w Z w () represents the coordinates of a three-dimensional point in the world coordinate system; Distortion model: ; in, k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; In a factory setting, a checkerboard calibration board of known size is fixed, ensuring that all cameras can observe it simultaneously. Using the PnP algorithm, the extrinsic parameters are calculated based on the 3D world coordinates and corresponding 2D pixel coordinates of the calibration board's corner points. ; Among them, P i 3D world coordinates of the corner points of the calibration board; p i Here are the corresponding 2D pixel coordinates; π(⋅) is the projection function; Use OpenCV's solvePnP function to calculate the extrinsic parameter matrices R and t for each camera.

[0007] Furthermore, a 3D physical space model of the target production area is constructed to form a factory coordinate system, as follows: The scene is obtained by scanning with LiDAR, and the world coordinate system of the calibration board is aligned with the global coordinate system of the factory 3D model. If the position of the calibration plate in the factory coordinate system is [R] b ,t b If the camera extrinsic parameters are then converted to:

[0008] The projection matrix P=K[R] of all cameras in the factory's global coordinate system factory |t factory]; Each camera independently acquires images and detects the pixel position of the same physical target in its own image. Using the intrinsic and extrinsic parameters of each camera, the 3D spatial coordinates of the observed target point are calculated by back-projection. Compare the 3D coordinates calculated by each camera with the actual known coordinates to see if they are consistent, or compare the 3D coordinates of the target calculated by multiple cameras to see if the error is controlled within the allowable range of engineering. If significant drift or stacking errors are found, check the calibration process, distortion correction or mechanical adjustment accuracy of each camera.

[0009] Furthermore, multi-view image frames are acquired using calibrated cameras to obtain timestamped, synchronized original multi-view image frames, which are then preprocessed as follows: All cameras are connected via trigger signal lines to achieve hardware-level synchronized exposure. Each camera synchronizes its time using the SDK controlled by the same host computer before initiating a unified acquisition command. A timestamp is automatically added to each acquired image frame, and at each moment, original frames from all cameras are acquired. The preprocessing includes grayscale correction, filtering, and contrast enhancement. Grayscale correction is achieved by calculating the mean (μ) of the R, G, and B channels of the original image. R ,μ G ,μ B The offset of each channel is corrected to the overall average level M; the corrected pixel values ​​are subjected to non-linear transformation to adjust the brightness distribution of the overall image; the image after brightness adjustment is filtered by Gaussian filtering; the filtered image is subjected to global or local contrast stretching, and adaptive enhancement is used to divide the image into small regions to increase the contrast of each region.

[0010] Furthermore, based on the preprocessed image frames, the Detectron2 deep learning detection model is used to detect the target position in each frame. Monocular inter-frame correlation is performed on the detected targets to obtain local trajectories. Depth feature vectors are extracted for each target, and the 2D detection boxes, local 2D trajectory IDs, and appearance features of all targets in the current frame are obtained for each camera, as detailed below: For each preprocessed image frame, YOLOv8 is used for multi-object detection, and the 2D bounding box, class label and confidence score of each object are output. For each 2D detection box, the target region is cropped and fed into the appearance feature extraction deep network to obtain the Re-ID feature vector; Single-target tracking based on motion and appearance is performed on the target detection results of consecutive frames from the same camera. BYTETrack is used to realize inter-frame target association. A unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. The output of each camera and each frame is the target set:

[0011] Among them, b i s is the two-dimensional detection bounding box for the i-th target; i Confidence score; id i f is the local trajectory ID at the current moment under this camera; i For target appearance Re-ID features.

[0012] Furthermore, for each preprocessed frame, YOLOv8 is used for multi-object detection, outputting the 2D bounding box, class label, and confidence score for each object, as follows: Preprocessed image I for each frame in Normalize and scale (x, y, c): ; in, W raw H raw W represents the width and height of the original image. model H model Provides the required input width and height for YOLOv8; The processed image is input into a pre-trained YOLOv8 model. Based on a deep convolutional network, the model extracts multi-level features from the image and predicts the location and category information of all objects in the image at different spatial scales. For each feature point, the YOLOv8 output is: O=(x norm ,y norm ,w norm ,h norm ,s,p1,...,p k ,...,p C ); Where, x norm ,y norm The relative coordinates of the center point of the bounding box; w norm ,h norm s represents the bounding box width and height, relative coordinates; s represents the confidence score; p k Let be the category probability of belonging to the kk-th target class; The relative coordinates of the bounding box center point are restored to the actual pixel coordinates of the original image. By setting a confidence threshold, only detection results with sufficiently high confidence are retained. Then, a non-maximum suppression algorithm is used to remove multiple highly overlapping detection boxes, retaining only the highest score in similar regions. Finally, the two-dimensional box position, category label, and confidence score of all targets are returned for each frame.

[0013] Furthermore, single-target tracking based on motion and appearance is performed on the target detection results of consecutive frames from the same camera. BYTETrack is used to achieve inter-frame target association, and a unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. Based on the acquired 2D bounding box, category, and confidence score for each frame, as well as the Re-ID feature vector extracted for each detected target; BYTETrack is used to divide the detection results into high-confidence candidate HSD and low-confidence candidate LSD. HSD is used to update active trajectories, while LSD can supplement trajectory interruptions caused by missed detections. For all tracked targets, use Kalman filtering to predict their state and predict their possible locations in the current frame. A two-level data association method is adopted. First, Hungarian matching is performed in HSD and Track using IoU overlap. For all unassigned detections and trajectories, supplement the matching using appearance feature distance; If a Track can be associated with a new detection, the original ID is maintained and the status and appearance features are updated; unmatched detections create new tracks and are assigned new IDs; tracks that have not matched for a long time will be terminated. Each frame assigns a unique local 2D trajectory ID to all tracked targets and outputs a 2D detection box, classification, confidence score, trajectory ID, and appearance features.

[0014] Furthermore, the 2D detection box of the target in the current frame is back-projected onto the factory's physical 3D space. When the same target appears in multiple viewpoints, the spatial coordinates are fused to obtain accurate 3D positioning, resulting in a candidate set of physical 3D spatial coordinates for each actual target, as follows: Obtain the intrinsic parameter matrix K and extrinsic parameter (R,t) for each camera, and backproject the center point of the 2D detection box onto a 3D ray; for each detected target, backproject the center point of the 2D detection box and the camera's intrinsic and extrinsic parameters onto the image of each camera as a ray in the physical space of the factory, rather than a single point; Based on Re-ID feature vectors and spatiotemporal constraints, the 2D detection of the same physical target under different cameras is matched to obtain multiple ray groups. Let L be the spatial ray L generated by the same target in the j-th camera. j : ; Among them, C j Let d be the optical center in the world coordinate system of the j-th camera; j The ray direction vector; By using rays from the same target viewed from different cameras, the closest point of the ray in space is determined, which serves as the three-dimensional spatial positioning result of the object being measured. If the candidate point lies on N camera rays, finding the nearest point minimizes the following loss: ; Among them, Q 3D represents the coordinates of the 3D point to be solved; N is the number of rays; For the first The three-dimensional coordinates of the optical center of a camera in the world coordinate system; No. The direction vector of the ray; A candidate set of physical 3D spatial coordinates is obtained.

[0015] Furthermore, based on the candidate set of physical 3D spatial coordinates, multimodal fusion is performed using spatial distance, temporal consistency, and appearance features to construct a matching candidate set. A joint matching algorithm is then used to achieve global ID allocation of the target across different cameras and time slices. Trajectories are merged to obtain the complete 3D trajectory, as detailed below: Based on the candidate set of physical 3D spatial coordinates, multimodal features, including three dimensions of spatial distance, time interval, and appearance features, are used to construct a pair matching candidate set for all physical target candidate points, which can be used to determine whether a group of targets are generated by the same real object. Define a joint matching cost function for global association, integrating spatial, temporal, and appearance information: construct a cost matrix C for paired candidates, where each element is the matching cost Cost(A) between target pairs. a B b Joint matching optimization problem:

[0016] in, ∈{0,1} indicates whether group a and group b are selected as one; satisfying the following constraint: each target candidate belongs to at most one global trajectory; Construct a cost graph and use the Hungarian algorithm to achieve global optimal allocation; Assign a globally unique target ID based on the matching results, merge all matching 3D trajectory fragments, and output a complete global 3D target trajectory list. Each physical target has a unique ID, which includes its complete 3D trajectory at different viewpoints and times.

[0017] A multi-view joint tracking system for complex targets in an industrial production scenario includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the multi-view joint tracking method for complex targets in an industrial production scenario as described above.

[0018] The present invention has the following beneficial effects: 1. This invention, through multi-camera calibration and multi-view fusion, can overcome the limitations of single-camera field of view and accuracy, and achieve high-precision three-dimensional positioning of targets in the factory coordinate system. It effectively solves the problem of monocular tracking caused by occlusion, object stacking and complex movement in the production site, and ensures that each target is always within the visible and trackable range of the system. 2. This invention effectively improves the accuracy of target identification across perspectives, time periods, and regions. Especially in industrial environments where targets have highly similar appearances, frequent occlusion, or complex movement trajectories, it significantly reduces ID drift and mismatch probability, and enhances global trajectory consistency and continuity. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 In this embodiment, a multi-view joint tracking method for complex targets in an industrial production scenario is provided, characterized by the following steps: S1: Calibrate the intrinsic and extrinsic parameters of all cameras, construct a 3D physical space model of the target production area, form a factory coordinate system, and obtain the geometric mapping parameters between each camera and the physical space; S2: Acquire multi-view image frames through the calibrated camera, obtain multi-view synchronized raw image frames with timestamps, and perform preprocessing to obtain preprocessed image frames; S3: Based on the preprocessed image frames, use the deep learning detection model Detectron2 to detect the target position in each frame, perform monocular inter-frame correlation on the detected targets to obtain local trajectories, extract depth feature vectors for each target, and obtain the 2D detection boxes, local 2D trajectory IDs and appearance features of all targets in the current frame under each camera. S4: Back-project the 2D detection box of the target in the current frame to the physical 3D space of the factory. When the same target appears in multiple viewpoints, fuse the spatial coordinates to obtain accurate 3D positioning and obtain a candidate set of physical 3D spatial coordinates for each actual target. S5: Based on the candidate set of physical 3D spatial coordinates, multimodal fusion is performed using spatial distance, temporal consistency and appearance features to construct a matching candidate set. A joint matching algorithm is then used to achieve global ID allocation of the target across different cameras and time slices, and the trajectories are merged to obtain a complete 3D trajectory.

[0021] In this embodiment, intrinsic and extrinsic parameters are calibrated for all cameras, as follows: The camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method (chessboard calibration method), which uses multiple chessboard images taken from different angles by the same camera. Camera model: ; Where K is the intrinsic parameter matrix; fx, fy are focal lengths, cx, cy are principal point coordinates; rotation matrix R and translation vector t are extrinsic parameters; (X w ,Y w Z w () represents the coordinates of a three-dimensional point in the world coordinate system (i.e., the factory's physical space coordinate system); Distortion model (radial and tangential distortion): ; in, k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; In a factory setting, a checkerboard calibration board of known size is fixed, ensuring that all cameras can observe it simultaneously. The Perspective-n-Point (PnP) algorithm is used to solve for the extrinsic parameters based on the 3D world coordinates (in the factory coordinate system) and corresponding 2D pixel coordinates of the calibration board's corner points. ; Among them, P i 3D world coordinates of the corner points of the calibration board; p i Here are the corresponding 2D pixel coordinates; π(⋅) is the projection function; Use OpenCV's solvePnP function (based on Levenberg-Marquardt optimization or RANSAC) to compute the extrinsic parameter matrices R and t for each camera.

[0022] In this embodiment, a 3D physical space model of the target production area is constructed to form a factory coordinate system, as detailed below: The scene's 3D point cloud model is obtained through LiDAR scanning, and the world coordinate system (origin in the external parameter calibration) of the calibration board is aligned with the global coordinate system of the factory's 3D model. If the position of the calibration plate in the factory coordinate system is [R] b ,t b If the camera extrinsic parameters are then converted to:

[0023] The projection matrix P=K[R] of all cameras in the factory's global coordinate system factory |t factory ]; Each camera independently acquires images and detects the pixel position of the same physical target in its own image. Using the intrinsic and extrinsic parameters of each camera, the 3D spatial coordinates of the observed object point are calculated by back projection (such as triangulation). Compare the 3D coordinates calculated by each camera with the actual known coordinates to see if they are consistent, or compare the 3D coordinates of the target calculated by multiple cameras to see if the error is controlled within the allowable range of engineering. If significant drift or stacking errors are found, check the calibration process, distortion correction or mechanical adjustment accuracy of each camera.

[0024] In this embodiment, multi-view image frames are acquired using calibrated cameras to obtain timestamped, synchronized original multi-view image frames, which are then preprocessed as follows: All cameras are connected via trigger signal lines to achieve hardware-level synchronized exposure. Each camera synchronizes its time using the SDK controlled by the same host computer, and then a unified acquisition command is initiated. A timestamp is automatically added to each acquired image frame, and at each moment, original frames from all cameras are acquired. The preprocessing includes grayscale correction, filtering, and contrast enhancement. The grayscale correction is achieved by calculating the mean (μ) of the R, G, and B channels of the original image. R ,μ G ,μ B The process involves correcting the offset of each channel to the overall average level M; performing a non-linear transformation on the corrected pixel values ​​to adjust the overall brightness distribution of the image; applying Gaussian filtering to the brightness-adjusted image; performing global or local contrast stretching on the filtered image; and using adaptive enhancement (such as CLAHE) to divide the image into small regions and increase the contrast of each region; preventing local overexposure or underexposure and making subtle changes (such as object edges, blemishes, etc.) clearer.

[0025] In this embodiment, based on the preprocessed image frames, the Detectron2 deep learning detection model is used to detect the target position in each frame. Monocular inter-frame correlation is performed on the detected targets to obtain local trajectories. Depth feature vectors are extracted for each target, and the 2D detection bounding boxes, local 2D trajectory IDs, and appearance features of all targets in the current frame are obtained for each camera, as detailed below: For each preprocessed image frame, YOLOv8 is used for multi-object detection, and the 2D bounding box, class label and confidence score of each object are output. For each 2D detection box, the target region is cropped and fed into the appearance feature extraction deep network to obtain the Re-ID feature vector, which is used to characterize the target identity and re-identify across frames / cameras; Single-target tracking based on motion and appearance is performed on the target detection results of consecutive frames from the same camera. BYTETrack is used to realize inter-frame target association. A unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. The output of each camera and each frame is the target set:

[0026] Among them, b i s is the two-dimensional detection bounding box for the i-th target; i Confidence score; id i f is the local trajectory ID at the current moment under this camera (obtained after monocular association); i For target appearance Re-ID features.

[0027] In this embodiment, YOLOv8 is used to perform multi-object detection on each preprocessed image frame, and the 2D bounding box, class label, and confidence score of each object are output as follows: Preprocessed image I for each frame in Normalize and scale (x, y, c): ; in, W raw H raw W represents the width and height of the original image. model H model Provides the required input width and height for YOLOv8; The processed image is input into a pre-trained YOLOv8 model. Based on a deep convolutional network, the model extracts multi-level features from the image and predicts the location and category information of all objects in the image at different spatial scales. For each feature point, the YOLOv8 output is: O=(x norm ,y norm ,w norm ,h norm ,s,p1,...,p k ,...,p C ); Where, x norm ,y norm The relative coordinates of the center point of the bounding box; w norm ,h norm s represents the bounding box width and height, relative coordinates; s represents the confidence score; p k Let be the category probability of belonging to the kk-th target class; The relative coordinates of the bounding box center point are restored to the actual pixel coordinates of the original image. By setting a confidence threshold, only detection results with sufficiently high confidence are retained. Then, the non-maximum suppression (NMS) algorithm is used to remove multiple highly overlapping detection boxes, retaining only the highest score in similar regions. Finally, the two-dimensional box position, class label and confidence score of all targets are returned for each frame.

[0028] In this embodiment, single-target tracking based on motion and appearance is performed on the target detection results of consecutive frames from the same camera. BYTETrack is used to achieve inter-frame target association, and a unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. Based on the acquired 2D bounding box, category, and confidence score for each frame, as well as the Re-ID feature vector extracted for each detected target; BYTETrack is used to divide the detection results into high-confidence candidate HSD and low-confidence candidate LSD. HSD is used to update active trajectories, while LSD can supplement trajectory interruptions caused by missed detections. Kalman filtering is used to predict the state of all tracked targets (trajectories from the previous frame) to predict the possible location of the targets in the current frame. A two-level data association method is adopted. First, Hungarian matching is performed in HSD and Track using IoU overlap. For all unassigned detections and trajectories, supplement the matching using appearance feature distances (such as cosine distances); If a Track can be associated with a new detection, the original ID is maintained and the status and appearance features are updated; unmatched detections create new tracks and are assigned new IDs; tracks that have not matched for a long time will be terminated. Each frame assigns a unique local 2D trajectory ID to all tracked targets and outputs a 2D detection box, classification, confidence score, trajectory ID, and appearance features.

[0029] In this embodiment, the 2D detection box of the target in the current frame is back-projected onto the physical 3D space of the factory. When the same target appears in multiple viewpoints, the spatial coordinates are fused to obtain accurate 3D positioning, resulting in a candidate set of physical 3D spatial coordinates for each actual target, as follows: Obtain the intrinsic parameter matrix K and extrinsic parameter (R,t) for each camera, and backproject the center point of the 2D detection box onto a 3D ray; for each detected target, backproject the center point (or multiple points) of the 2D detection box onto the image of each camera using the camera's intrinsic and extrinsic parameters, and project it into a ray in the physical space of the factory, rather than a single point; Based on Re-ID feature vectors and spatiotemporal constraints, the 2D detection of the same physical target under different cameras is matched to obtain multiple ray groups. Let L be the spatial ray L generated by the same target in the j-th camera. j : ; Among them, C j Let d be the optical center in the world coordinate system of the j-th camera; j The ray direction vector; By using rays from the same target viewed from different cameras, the closest point of the ray in space is determined, which serves as the three-dimensional spatial positioning result of the object being measured. If the candidate point lies on N camera rays, finding the nearest point minimizes the following loss: ; Among them, Q 3D represents the coordinates of the 3D point to be solved; N is the number of rays; For the first The three-dimensional coordinates of the optical center of a camera in the world coordinate system; No. The direction vector of the ray; A candidate set of physical 3D spatial coordinates is obtained.

[0030] In this embodiment, based on the candidate set of physical 3D spatial coordinates, multimodal fusion is performed using spatial distance, temporal consistency, and appearance features to construct a matching candidate set. A joint matching algorithm is then used to achieve global ID allocation of the target across different cameras and time slices. Trajectories are merged to obtain a complete 3D trajectory, as detailed below: Based on the candidate set of physical 3D spatial coordinates, multimodal features, including three dimensions of spatial distance, time interval, and appearance features, are used to construct a pair matching candidate set for all physical target candidate points, which can be used to determine whether a group of targets are generated by the same real object. Define a joint matching cost function for global association, integrating spatial, temporal, and appearance information: construct a cost matrix C for paired candidates, where each element is the matching cost Cost(A) between target pairs. a B b Joint matching optimization problem:

[0031] in, ∈{0,1} indicates whether group a and group b are selected as one; satisfying the following constraint: each target candidate belongs to at most one global trajectory; Construct a cost graph and use the Hungarian algorithm to achieve global optimal allocation; Assign a globally unique target ID based on the matching results, merge all matching (spatiotemporally continuous or overlapping and with consistent appearance) 3D trajectory fragments, and output a complete global 3D target trajectory list. Each physical target has a unique ID, which includes its complete three-dimensional trajectory at different viewpoints and times.

[0032] A multi-view joint tracking system for complex targets in an industrial production scenario includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps described above in a multi-view joint tracking method for complex targets in an industrial production scenario. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0033] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0034] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0035] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multi-view joint tracking method for complex targets in industrial production scenarios, characterized in that, Includes the following steps: S1: Calibrate the intrinsic and extrinsic parameters of all cameras, construct a 3D physical space model of the target production area, form a factory coordinate system, and obtain the geometric mapping parameters between each camera and the physical space; S2: Acquire multi-view image frames through the calibrated camera, obtain multi-view synchronized raw image frames with timestamps, and perform preprocessing to obtain preprocessed image frames; S3: Based on the preprocessed image frames, use the deep learning detection model Detectron2 to detect the target position in each frame, perform monocular inter-frame correlation on the detected targets to obtain local trajectories, extract depth feature vectors for each target, and obtain the 2D detection boxes, local 2D trajectory IDs and appearance features of all targets in the current frame under each camera. S4: Back-project the 2D detection box of the target in the current frame to the physical 3D space of the factory. When the same target appears in multiple viewpoints, fuse the spatial coordinates to obtain accurate 3D positioning and obtain a candidate set of physical 3D spatial coordinates for each actual target. S5: Based on the candidate set of physical 3D spatial coordinates, multimodal fusion is performed using spatial distance, temporal consistency and appearance features to construct a matching candidate set. A joint matching algorithm is then used to achieve global ID allocation of the target across different cameras and time slices, and the trajectories are merged to obtain a complete 3D trajectory.

2. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 1, characterized in that, The intrinsic and extrinsic parameters of all cameras are calibrated as follows: The camera intrinsic parameter calibration adopts the Zhang Zhengyou calibration method, using multiple checkerboard images taken from different angles by the same camera. Camera model: ； Where K is the intrinsic parameter matrix; fx, fy are focal lengths, cx, cy are principal point coordinates; rotation matrix R and translation vector t are extrinsic parameters; (X w ,Y w Z w () represents the coordinates of a three-dimensional point in the world coordinate system; Distortion model: ； in, k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients; In a factory setting, a checkerboard calibration board of known size is fixed, ensuring that all cameras can observe it simultaneously. Using the PnP algorithm, the extrinsic parameters are calculated based on the 3D world coordinates and corresponding 2D pixel coordinates of the calibration board's corner points. ； Among them, P i 3D world coordinates of the corner points of the calibration board; p i Here are the corresponding 2D pixel coordinates; π(⋅) is the projection function; Use OpenCV's solvePnP function to calculate the extrinsic parameter matrices R and t for each camera.

3. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 2, characterized in that, The construction of a 3D physical space model of the target production area, forming a factory coordinate system, is as follows: The scene is obtained by scanning with LiDAR, and the world coordinate system of the calibration board is aligned with the global coordinate system of the factory 3D model. If the position of the calibration plate in the factory coordinate system is [R] b ,t b If the camera extrinsic parameters are then converted to: ； The projection matrix P=K[R] of all cameras in the factory's global coordinate system factory |t factory ]; Each camera independently acquires images and detects the pixel position of the same physical target in its own image. Using the intrinsic and extrinsic parameters of each camera, the 3D spatial coordinates of the observed target point are calculated by back-projection. Compare the 3D coordinates calculated by each camera with the actual known coordinates to see if they are consistent, or compare the 3D coordinates of the target calculated by multiple cameras to see if the error is controlled within the allowable range of engineering. If significant drift or stacking errors are found, check the calibration process, distortion correction or mechanical adjustment accuracy of each camera.

4. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 1, characterized in that, The process involves acquiring multi-view image frames using calibrated cameras, obtaining timestamped, synchronized raw image frames, and preprocessing them as follows: All cameras are connected via trigger signal lines to achieve hardware-level synchronized exposure. Each camera synchronizes its time using the SDK controlled by the same host computer before initiating a unified acquisition command. A timestamp is automatically added to each acquired image frame, and raw frames from all cameras are acquired at each moment. The preprocessing includes grayscale correction, filtering, and contrast enhancement. Grayscale correction is achieved by calculating the mean (μ) of the R, G, and B channels of the raw image. R ,μ G ,μ B The offset of each channel is corrected to the overall average level M; the corrected pixel values ​​are subjected to non-linear transformation to adjust the brightness distribution of the overall image; and the image after brightness adjustment is filtered by Gaussian filtering. The filtered image is subjected to global or local contrast stretching, and adaptive enhancement is used to divide the image into small regions and increase the contrast of each region.

5. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 1, characterized in that, The process involves using the Detectron2 deep learning detection model to detect the target position in each frame based on the preprocessed image frames. Monocular inter-frame correlation is then performed on the detected targets to obtain local trajectories. Depth feature vectors are extracted for each target, and the 2D detection bounding boxes, local 2D trajectory IDs, and appearance features of all targets in the current frame are obtained for each camera, as detailed below: For each preprocessed image frame, YOLOv8 is used for multi-object detection, and the 2D bounding box, class label and confidence score of each object are output. For each 2D detection box, the target region is cropped and fed into the appearance feature extraction deep network to obtain the Re-ID feature vector; Single-target tracking based on motion and appearance is performed on the target detection results of consecutive frames from the same camera. BYTETrack is used to realize inter-frame target association. A unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. The output of each camera and each frame is the target set: ； Among them, b i s is the two-dimensional detection bounding box for the i-th target; i Confidence score; id i f is the local trajectory ID at the current moment under this camera; i For target appearance Re-ID features.

6. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 5, characterized in that, For each preprocessed image frame, YOLOv8 is used for multi-object detection, outputting the 2D bounding box, class label, and confidence score for each object, as detailed below: Preprocessed image I for each frame in Normalize and size-transform (x, y, c): ； in, W raw H raw W represents the width and height of the original image. model H model The required input width and height for YOLOv8; The processed image is input into a pre-trained YOLOv8 model. Based on a deep convolutional network, the model extracts multi-level features from the image and predicts the location and category information of all objects in the image at different spatial scales. For each feature point, the YOLOv8 output is: O=(x norm ,y norm ,w norm ,h norm ,s,p1,...,p k ,...,p C ); Where, x norm ,y norm The relative coordinates of the center point of the bounding box; w norm ,h norm s represents the bounding box width and height, relative coordinates; s represents the confidence score; p k Let be the category probability of belonging to the kk-th target class; The relative coordinates of the bounding box center point are restored to the actual pixel coordinates of the original image. By setting a confidence threshold, only detection results with sufficiently high confidence are retained. Then, a non-maximum suppression algorithm is used to remove multiple highly overlapping detection boxes, retaining only the highest score in similar regions. Finally, the two-dimensional box position, category label, and confidence score of all targets are returned for each frame.

7. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 5, characterized in that, The method involves performing motion and appearance-based single-target tracking on consecutive frames of target detection results from the same camera. BYTETrack is used to associate targets between frames, and a unique local 2D trajectory ID is assigned based on IoU overlap and appearance feature distance. Based on the acquired 2D bounding box, category, and confidence score for each frame, as well as the Re-ID feature vector extracted for each detected target; BYTETrack is used to divide the detection results into high-confidence candidate HSD and low-confidence candidate LSD. HSD is used to update active trajectories, while LSD can supplement trajectory interruptions caused by missed detections. For all tracked targets, use Kalman filtering to predict their state and predict their possible locations in the current frame. A two-level data association method is adopted. First, Hungarian matching is performed in HSD and Track using IoU overlap. For all unassigned detections and trajectories, supplement the matching using appearance feature distance; If a Track can be associated with a new detection, the original ID is maintained and the status and appearance features are updated; unmatched detections create new tracks and are assigned new IDs; tracks that have not matched for a long time will be terminated. Each frame assigns a unique local 2D trajectory ID to all tracked targets and outputs a 2D detection box, classification, confidence score, trajectory ID, and appearance features.

8. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 1, characterized in that, The process involves back-projecting the target's 2D detection bounding box in the current frame to the factory's physical 3D space. When the same target appears from multiple viewpoints, the spatial coordinates are fused to obtain accurate 3D positioning, resulting in a candidate set of physical 3D spatial coordinates for each actual target, as detailed below: Obtain the intrinsic parameter matrix K and extrinsic parameter (R,t) for each camera, and backproject the center point of the 2D detection box onto a 3D ray; for each detected target, backproject the center point of the 2D detection box and the camera's intrinsic and extrinsic parameters onto the image of each camera as a ray in the physical space of the factory, rather than a single point; Based on Re-ID feature vectors and spatiotemporal constraints, the 2D detection of the same physical target under different cameras is matched to obtain multiple ray groups. Let L be the spatial ray L generated by the same target in the j-th camera. j : ; Among them, C j Let d be the optical center of the j-th camera in the world coordinate system; j The ray direction vector; By using rays from the same target viewed from different cameras, the closest point of the ray in space is determined, which serves as the three-dimensional spatial positioning result of the object being measured. If the candidate point lies on N camera rays, finding the nearest point minimizes the following loss: ； Among them, Q 3D represents the coordinates of the 3D point to be solved; N is the number of rays; For the first The three-dimensional coordinates of the optical center of a camera in the world coordinate system; No. The direction vector of the ray; A candidate set of physical 3D spatial coordinates is obtained.

9. The multi-view joint tracking method for complex targets in an industrial production scenario according to claim 1, characterized in that, The process involves constructing a matching candidate set based on a candidate set of physical 3D spatial coordinates, utilizing spatial distance, temporal consistency, and appearance features for multimodal fusion, and employing a joint matching algorithm to achieve global ID allocation for the target across different cameras and time slices. Trajectories are then merged to obtain a complete 3D trajectory, as detailed below: Based on the candidate set of physical 3D spatial coordinates, multimodal features, including three dimensions of spatial distance, time interval, and appearance features, are used to construct a pair matching candidate set for all physical target candidate points, which can be used to determine whether a group of targets are generated by the same real object. Define a joint matching cost function for global association, integrating spatial, temporal, and appearance information: construct a cost matrix C for paired candidates, where each element is the matching cost Cost(A) between target pairs. a B b Joint matching optimization problem: ； in, ∈{0,1} indicates whether group a and group b are selected as one; satisfying the following constraint: each target candidate belongs to at most one global trajectory; Construct a cost graph and use the Hungarian algorithm to achieve global optimal allocation; Assign a globally unique target ID based on the matching results, merge all matching 3D trajectory fragments, and output a complete global 3D target trajectory list. Each physical target has a unique ID, which includes its complete 3D trajectory at different viewpoints and times.

10. A multi-view joint tracking system for complex targets in industrial production scenarios, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the multi-view joint tracking method for complex targets in an industrial production scenario as described in any one of claims 1-9.

Images