Multi-target impurity continuous tracking and trajectory prediction method and system based on deepsort

Abstract

The application discloses a multi-target impurity continuous tracking and trajectory prediction method and system based on DeepSORT, and belongs to the technical field of industrial visual material sorting. The method comprises the following steps: acquiring the inter-frame displacement increment of a conveying belt and the detection position information of impurities in the material; inputting the inter-frame displacement increment and the individual sliding factor into a conveying belt constraint motion model to predict the impurity target position; after multi-dimensional joint matching, the impurity position and the individual sliding factor are synchronously updated, and the updated individual sliding factor is fed back to the motion model for next period position prediction; the time when the impurities reach the removal position is calculated in combination with the updated impurity position and the individual sliding factor, and the removal action is triggered. The application constructs a motion model based on the physical constraints of the conveying belt, identifies the individual sliding factor online, and forms a closed loop self-correction, which can significantly improve the impurity trajectory prediction accuracy and removal timing accuracy, and effectively solves the technical problems of the traditional DeepSORT framework motion model deviating from the physical law and large removal error.

Description

Technical Field

[0001] This invention relates to the field of industrial visual material sorting technology, and specifically discloses a method and system for continuous tracking and rejection trajectory prediction of multi-target impurities based on DeepSORT. Background Technology

[0002] With the rapid development of machine vision and automation control technologies, conveyor belt-based automatic material impurity sorting systems based on vision recognition have been widely used in industrial production. Existing sorting systems generally adopt a standardized technical architecture of "image acquisition - target detection - multi-target tracking - rejection execution". Among them, the multi-target tracking link is mostly based on the DeepSORT algorithm, which relies on the uniform motion model of Kalman filtering to predict the target position, and combines appearance features and Mahalanobis distance to complete the data association between the trajectory and the detected target. Finally, the time for the impurity to reach the rejection mechanism is calculated based on the nominal speed of the conveyor belt, and the rejection action is triggered.

[0003] In practical industrial applications, existing technologies have the following obvious shortcomings: First, the motion model deviates from the physical motion laws of the conveyor belt. DeepSORT's built-in uniform Kalman filter model treats the target as a freely moving point in the image plane, without considering the forced translational constraints of the conveyor belt on the material, and cannot characterize the composite motion characteristics of "overall translation of the conveyor belt + individual sliding disturbance of impurities". As a result, the trajectory prediction results have a continuous systematic deviation, and the tracking accuracy is difficult to meet the requirements of high-precision sorting.

[0004] Second, the robustness of data association is insufficient. Traditional DeepSORT uses single-frame or short-term appearance features for matching, without setting a feature validity screening mechanism. When impurities flip, rotate, occlude, or similar impurities appear densely, false associations are prone to occur, resulting in problems such as trajectory ID switching, trajectory breakage, and repeated counting, which affect the stability of continuous tracking.

[0005] Third, the calculation error for the rejection timing is large. Different impurities vary in material, weight, shape, and surface friction coefficient, causing them to slip relative to the conveyor belt to varying degrees. The actual movement speed does not match the nominal speed of the conveyor belt. Current solutions uniformly use the nominal speed of the conveyor belt to calculate the rejection arrival time, ignoring the individual slip differences of impurities. This leads to the rejection mechanism acting prematurely or lagging behind, resulting in frequent malfunctions such as missed rejection, incorrect rejection, and blown-out, resulting in a low impurity rejection pass rate.

[0006] Fourth, there is a lack of a closed-loop self-correction mechanism. In the existing scheme, motion prediction, parameter estimation, and elimination calculation are unidirectional serial processes. The motion parameters identified during the tracking process cannot be used to correct the motion prediction model in reverse. The system performance has a fixed upper limit and cannot be iteratively optimized with the operation process. It is also difficult to adapt to dynamic working conditions such as conveyor belt speed fluctuations and material type switching.

[0007] Therefore, there is an urgent need for a multi-target tracking and trajectory prediction scheme that conforms to the physical motion laws of conveyor belts, can identify individual sliding parameters online, and has closed-loop self-correction capabilities, in order to overcome the aforementioned shortcomings of existing technologies. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides solutions to the technical problems of existing DeepSORT tracking framework motion models deviating from the physical laws of conveyor belts, individual sliding differences leading to inaccurate removal timing, lack of closed-loop parameter self-correction mechanisms, and insufficient tracking stability, thereby achieving high-precision and high-stability continuous tracking and accurate removal of conveyor belt material impurities.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for predicting the trajectory of continuous tracking and removal of multi-target impurities based on DeepSORT, characterized in that it includes: Acquire the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; The inter-frame displacement increment and the individual slip factor are input into the conveyor belt constrained motion model to predict the target position of the impurity. The predicted impurity target location is matched with the impurity corresponding to the detection location information, and the impurity position and individual slip factor are updated synchronously based on the matching result; The updated individual slip factor is fed back to the conveyor belt constrained motion model for position prediction in the next cycle. The time it takes for an impurity to reach the removal position is calculated based on the updated impurity position and the individual slip factor, triggering the removal action.

[0010] Optionally, obtaining the inter-frame displacement increment of the conveyor belt includes: Based on the synchronous trigger signal, the cumulative displacement of the conveyor belt corresponding to the image timestamp is obtained; The inter-frame displacement increment is obtained by calculating the difference between the cumulative displacements of the conveyor belt in adjacent frames.

[0011] Optionally, before predicting the target location of the impurity, the method further includes: The unit vector of the conveyor belt's motion direction is pre-calibrated offline, and then input into the conveyor belt's constrained motion model.

[0012] Optionally, inputting the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity includes: Construct a three-dimensional joint state vector containing the horizontal and vertical coordinates of the impurities and the individual slip factor; The inter-frame displacement increment, the unit vector of the conveyor belt motion direction, and the historical individual slip factor are input into the conveyor belt constrained motion model to solve for the target position prediction coordinates of the impurity. Based on a symmetric sigma point set, an unscented transformation is performed on the three-dimensional joint state vector to solve for the trajectory prediction position and prediction covariance.

[0013] Optionally, the step of synchronously updating the position and individual slip factor of impurities based on the matching results includes: Construct a linear observation equation for the location coordinates of the impurities; The successfully matched detection location information is used as the observation value, and the state is updated by unscented Kalman filtering to output the updated three-dimensional joint state vector; wherein, the three-dimensional joint state vector includes the impurity location and the individual slip factor.

[0014] Optionally, matching the predicted impurity target location with the impurity corresponding to the detection location information includes: Call the spatial texture joint features obtained from the detection and corresponding to the impurity detection location; Calculate appearance similarity based on the aforementioned spatial texture joint features; A joint cost matrix is ​​constructed by integrating the appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence. Based on the joint cost matrix, the existing impurity tracking trajectories are matched with the detection location information.

[0015] Optionally, calculating the appearance similarity based on the joint spatial texture features includes: Maintain a feature cache queue for each of the impurity tracking trajectories; The spatial texture joint features that satisfy the dynamic consistency test are stored in the feature cache queue; Features in the feature cache queue are aggregated by temporal attention weights to generate a dynamic trajectory appearance vector. The cosine similarity between the query vector obtained from the spatial texture joint feature mapping and the trajectory dynamic appearance vector is calculated as the appearance similarity.

[0016] Optionally, the step of fusing the appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence to construct the joint cost matrix includes: The cost value corresponding to the appearance similarity, motion statistical distance, motion direction consistency and trajectory lifetime confidence is calculated respectively. The total cost of a single set of matching is obtained by weighted summation of each cost value. The joint cost matrix is ​​formed by filling the total cost of matching all trajectories with all detected targets. The trajectory lifetime confidence is calculated based on the number of consecutive trajectory matches, and the corresponding cost is obtained by generating a logarithmic penalty term based on the trajectory lifetime confidence.

[0017] Optionally, the step of calculating the time it takes for the impurity to reach the removal position based on the updated impurity position and the individual slip factor, and triggering the removal action, includes: Establish a mapping relationship between image coordinates and conveyor belt physical coordinates; The actual speed of the impurities is calculated based on the real-time speed of the conveyor belt and the individual slip factor. The arrival time is calculated based on the physical distance from the impurity location to the removal location and the actual movement speed. The rejection trigger time is determined by combining the action delay of the rejection mechanism, and the rejection pulse window is adaptively adjusted based on the estimated variance of the individual sliding factor to output the rejection signal.

[0018] Secondly, the present invention provides a multi-target impurity continuous tracking and removal trajectory prediction system based on DeepSORT, comprising: The information acquisition module is used to acquire the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; The position prediction module is used to input the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity. The matching and updating module matches the predicted impurity target position with the impurity corresponding to the detection position information, and updates the position and individual sliding factor of the impurity synchronously based on the matching result. The feedback module is used to send the updated individual slip factor back to the conveyor belt constrained motion model for position prediction in the next cycle. The action execution module is used to calculate the time it takes for an impurity to reach the removal position based on the updated impurity position and the individual sliding factor, and then trigger the removal action.

[0019] Compared with the closest existing technology, the present invention has the following advantages: This invention proposes a multi-target impurity continuous tracking and removal trajectory prediction method and system based on DeepSORT. On the basis of the DeepSORT framework, the traditional uniform motion model is replaced with a conveyor belt constrained motion model that conforms to the laws of physics. The actual inter-frame displacement increment of the conveyor belt is used as the motion driving input. The sliding disturbance of impurities relative to the conveyor belt is characterized by individual slip factors. The composite motion characteristics of materials in the conveyor belt scenario are accurately restored, effectively eliminating the systematic prediction bias caused by the traditional model, and the trajectory positioning accuracy is significantly improved.

[0020] This invention constructs a four-dimensional joint cost matrix that includes appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence. It also introduces a dynamic appearance feature processing mechanism that combines dynamic consistency testing and temporal attention aggregation. Only valid features that meet motion constraints are included in the appearance cache. Multiple constraints improve the accuracy of data association and effectively reduce trajectory ID switching and breakage problems in scenarios with dense impurities, occlusion, flipping, and changing lighting, thus ensuring the reliability of long-term continuous tracking.

[0021] This invention incorporates individual slip factors into a three-dimensional joint state vector and uses unscented Kalman filtering to identify the unique slip parameters of each impurity in real time online. By combining the individual slip factors with the actual movement speed of the impurities, it replaces the traditional method of calculating uniform nominal speed. This can control the rejection position error to the millimeter level, effectively reducing production failures such as missed rejection, incorrect rejection, and blow-out, and improving the impurity rejection pass rate.

[0022] This invention feeds back the updated individual sliding factor to the conveyor belt constrained motion model for position prediction in the next cycle. The sliding parameters are continuously iterated and optimized during the tracking process, eliminating the need for repeated manual offline calibration. It can adapt to dynamic working conditions such as conveyor belt speed fluctuations and material type switching, making the system more robust. Furthermore, it adaptively adjusts the rejection pulse window based on the estimated variance of the individual sliding factor. The higher the uncertainty of the sliding parameter estimation, the larger the pulse window width, maximizing the rejection hit probability while ensuring rejection accuracy. It can adapt to the impurity sorting needs of different materials and different sliding characteristics. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0024] Figure 1 This is a flowchart of the multi-target impurity continuous tracking and removal trajectory prediction method based on DeepSORT provided by the present invention; Figure 2 A flowchart of a method for predicting the target location of impurities provided in this embodiment of the invention; Figure 3 This is a flowchart of a method for matching the predicted impurity target location with the impurity corresponding to the detection location information, provided in an embodiment of the present invention. Figure 4 This is a block diagram of a multi-target impurity continuous tracking and removal trajectory prediction system based on DeepSORT provided in an embodiment of the present invention. Figure 5This is a diagram of the internal structure of the electronic device provided by the present invention. Detailed Implementation

[0025] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0026] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0027] This invention provides a method and system for continuous tracking and removal trajectory prediction of multi-target impurities based on DeepSORT, which is particularly suitable for various conveyor belt-type material impurity automatic sorting scenarios. The embodiments of this invention are described below with reference to the accompanying drawings.

[0028] Example 1: Combination Figure 1 As shown in the figure, the specific execution steps of the multi-target impurity continuous tracking and removal trajectory prediction method based on DeepSORT provided in this embodiment are as follows: S101 acquires the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; S102 Input the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity; S103 Match the predicted impurity target position with the impurity corresponding to the detection position information, and update the position and individual sliding factor of the impurity synchronously based on the matching result; S104 feeds the updated individual slip factor back to the conveyor belt constrained motion model for position prediction in the next cycle; S105 calculates the time it takes for the impurity to reach the removal position based on the updated impurity position and the individual slip factor, and triggers the removal action.

[0029] In one embodiment, step S101 above, specifically including obtaining the inter-frame displacement increment of the conveyor belt, includes: Based on the synchronous trigger signal, the cumulative displacement of the conveyor belt corresponding to the image timestamp is obtained; The inter-frame displacement increment is obtained by calculating the difference between the cumulative displacements of the conveyor belt in adjacent frames.

[0030] In practice, a synchronous trigger controller is used to synchronize the industrial camera exposure with the encoder pulse latch hardware, and the timestamp of each frame image is recorded as t. img The encoder's cumulative pulse count is obtained by interpolation as t. img The cumulative displacement S of the conveyor belt at any given time k Then, the inter-frame displacement increment is obtained. Where k represents the current frame number. The unit vector of the conveyor belt's direction of motion (u) x u y The data is obtained through offline calibration and used in subsequent motion prediction.

[0031] Simultaneously, an impurity detection network is used to detect the current k-th frame of the image, obtaining the detection location information of each impurity target. In one specific implementation, the impurity detection network uses a lightweight CenterNet network. The input image is mapped to a keypoint heatmap and bounding box size regression results. Simultaneously, the feature map of the region near the center of the detection box is extracted into a 128-dimensional spatial texture joint feature f using RoIAlign. raw This feature integrates the spatial distribution characteristics of impurities with the grayscale characteristics of surface texture, resulting in stronger individual differentiation, which can be used in subsequent data association processes.

[0032] Before predicting the target location of impurities, the unit vector of the conveyor belt's motion direction is pre-calibrated offline, and this unit vector is input into the conveyor belt constrained motion model. Specifically, multiple sets of displacement samples are collected along the material conveying direction of the conveyor belt, and the unit vector (u) of the conveyor belt's motion direction is obtained by fitting and solving the mapping relationship between image coordinates and physical coordinates. x u y It is stored in the system memory as a fixed input parameter for the conveyor belt constrained motion model.

[0033] Combination Figure 2 As shown, in one embodiment, step S102 above, inputting the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity specifically includes: S201, Construct a three-dimensional joint state vector containing the horizontal and vertical coordinates of the impurities and the individual slip factor; S202, input the inter-frame displacement increment, the unit vector of the conveyor belt motion direction, and the historical individual slip factor into the conveyor belt constrained motion model to solve for the target position prediction coordinates of the impurity; S203, perform an unscented transformation on the three-dimensional joint state vector based on a symmetric sigma point set to solve for the trajectory prediction position and prediction covariance.

[0034] The specific implementation is as follows: First, construct a three-dimensional joint state vector. Where x and y are the horizontal and vertical coordinates of the impurity image, The individual slip factor characterizes the ratio of an individual to the conveyor belt slippage. The process noise covariance matrix is ​​set as follows: ,in, This represents the variance of noise in the position process. This represents the process noise variance of the individual sliding factor, which itself follows a noisy, slowly varying model. ,in This refers to noise in the sliding factor process.

[0035] Secondly, the prediction equations for the conveyor belt constrained motion model are:

[0036] in, , This is the center position of the trajectory in the previous frame. , To predict the center position of the current frame trajectory, This is the inter-frame offset increment corresponding to the current frame. , These are the two components of the unit vector representing the direction of movement of the conveyor belt. , This model represents the noise from the positional process. It is constructed by superimposing the translational component of the conveyor belt with the individual sliding disturbance component, accurately characterizing the complex motion law of the material.

[0037] Finally, an unscented transformation is performed on the three-dimensional joint state vector based on a symmetric sigma point set. Using 2n+1 symmetric sampling sigma points (n is the dimension of the state vector, here n=3), the mean of the trajectory prediction position and the prediction covariance matrix P are obtained through weighted calculation, thus completing the probability distribution solution of the prediction step.

[0038] Combination Figure 3 As shown, in one embodiment, step S103 above, specifically matching the predicted impurity target location with the impurity corresponding to the detection location information, includes: S301, call the spatial texture joint feature obtained from the detection and corresponding to the impurity detection position; calculate the appearance similarity based on the spatial texture joint feature; S302, integrate the appearance similarity, motion statistical distance, motion direction consistency and trajectory survival confidence to construct a joint cost matrix; S303, based on the joint cost matrix, match the existing impurity tracking trajectory with the detection position information.

[0039] The specific implementation is as follows: 1. Calculation of distance cost in motion statistics The motion statistical distance uses Mahalanobis distance, which is calculated based on the trajectory prediction position, detection position, and prediction covariance, and is used to measure the degree of statistical matching between the detected target and the trajectory motion prediction result.

[0040] 2. Calculation of appearance similarity cost The calculation of appearance similarity based on the joint features of spatial texture specifically includes: Maintain a feature cache queue for each of the impurity tracking trajectories; The spatial texture joint features that satisfy the dynamic consistency test are stored in the feature cache queue; Features in the feature cache queue are aggregated by temporal attention weights to generate a dynamic trajectory appearance vector. The cosine similarity between the query vector obtained from the spatial texture joint feature mapping and the trajectory dynamic appearance vector is calculated as the appearance similarity.

[0041] Specifically, a fixed-length feature buffer queue is maintained for each existing impurity tracking trajectory, the length of which can be set to... L=30 is used to store historical valid appearance features. A dynamic consistency check is performed on the joint spatial texture features of the currently detected target: the normalized residual distance d between the detection center and the corresponding trajectory prediction position is calculated. res If d is satisfied res <θ gate If the detected feature is found to be consistent with the dynamics, it is stored in the corresponding trajectory's cache queue. Otherwise, it is considered a noise feature and is not stored to avoid invalid features caused by occlusion or false detection interfering with the matching results. Temporal attention weights are assigned to historical features in the cache queue, with higher weights for features closer to the current frame and lower weights for features farther away. A dynamic appearance vector of the trajectory is generated through weighted aggregation. The joint spatial texture features of the currently detected target are mapped into a query vector e of the same dimension through a fully connected mapping layer. det Calculate the cosine similarity between the two and convert it into an appearance similarity cost: .

[0042] 3. Calculation of the cost of consistency in motion direction Motion direction consistency cost measures the degree of deviation between the displacement direction of the detected target and the movement direction of the conveyor belt. During calculation, a displacement vector is first formed based on the detection positions of the same target in the current frame and the previous frame. Then, the angle between this displacement vector and the pre-calibrated conveyor belt movement direction vector is calculated. If the angle is greater than 90 degrees, it is determined to be reverse motion, and the motion direction consistency cost corresponding to the detection and trajectory is maximized to eliminate incorrect matching caused by reverse motion. If the angle is not greater than 90 degrees, the motion direction consistency cost is 0.

[0043] 4. Construct the joint cost matrix A joint cost matrix is ​​constructed by fusing appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence. The specific process is as follows: The cost values ​​of appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence are calculated separately for each trajectory and each detected target. The total cost value of a single match is obtained by weighted summation of the four types of cost values. All trajectories and all detected targets are traversed, and the total cost values ​​of all single matches are filled in the corresponding rows and columns to form a complete joint cost matrix.

[0044] The trajectory survival confidence score is calculated based on the number of consecutive successful matches, ranging from 0 to 1. Specifically, it is calculated as the ratio of the number of consecutive successful matches to a preset stable matching threshold, and this ratio does not exceed 1. The initial confidence score of a newly created trajectory is low; it increases with each successful match and decreases with consecutive non-matches.

[0045] The cost corresponding to the trajectory survival confidence is generated in the form of a logarithmic penalty term: the trajectory survival confidence is superimposed with an extremely small positive number to prevent the logarithmic operation from being singular, and then the logarithm is taken, and the negative value of the result is taken as the survival confidence cost; the lower the confidence, the higher the corresponding penalty cost, and the higher the total matching cost.

[0046] Meanwhile, the matching threshold is adaptively scaled based on the confidence level of the trajectory survival: the Mahalanobis distance matching threshold corresponding to the high-confidence trajectory is the sum of the base threshold multiplied by 1 and 0.5 times the confidence level, so as to achieve adaptive adjustment of relaxing the matching threshold for high-confidence trajectories and tightening the matching threshold for low-confidence trajectories.

[0047] Weight coefficients are assigned to the four types of cost values, and the total cost value of matching a single trajectory with a single detected target is calculated by weighted summation. Based on the generated joint cost matrix, the Hungarian algorithm is used to solve for the optimal matching relationship, and the matching pairs of trajectories and detected targets, unmatched trajectories, and unmatched detected targets are output.

[0048] In one embodiment, step S103 above, specifically updating the impurity position and individual sliding factor based on the matching result, includes: constructing a linear observation equation for observing the impurity position coordinates; using the successfully matched detection position information as the observation value, updating the state through unscented Kalman filtering, and outputting the updated three-dimensional joint state vector; wherein, the three-dimensional joint state vector includes the impurity position and the individual sliding factor.

[0049] Specifically, a linear observation equation is constructed that only observes the location coordinates of impurities. Where the observation vector z represents the detected impurity horizontal and vertical coordinates, and the observation matrix... ; Configure the corresponding observation noise covariance matrix R for the observation noise.

[0050] The successfully matched detection location information is used as the observation value input to the unscented Kalman filter. Measurement prediction and update calculations are performed at the sigma points, the Kalman gain is solved, and the 3D joint state vector and covariance matrix are corrected. The updated 3D joint state vector and posterior covariance are output. The updated 3D joint state vector includes the corrected impurity location and individual sliding factor, and the estimated variance corresponding to the individual sliding factor is also output for subsequent adaptive adjustment of the elimination pulse window.

[0051] In addition, after matching the trajectory with the detection location, trajectory lifecycle management is performed, specifically including: managing the creation and deletion of impurity tracking trajectories based on the matching results and trajectory lifecycle confidence to maintain continuous tracking; specifically, for high-confidence trajectories that do not match the detection target, the state is recursively updated and the lifecycle confidence is reduced through a conveyor belt constrained motion model; impurity tracking trajectories that are continuously unmatched and whose trajectory lifecycle confidence is below a threshold are deleted; new impurity tracking trajectories are created for unmatched detection location information and the individual sliding factor is initialized, with the initial value of the individual sliding factor α being 0 and the initial value of the lifecycle confidence being a low value when the new trajectory is initialized.

[0052] In step S104 above, the individual slip factor updated by the unscented Kalman filter is fed back to the conveyor belt constrained motion model as the historical individual slip factor input for the trajectory prediction of the next cycle, forming a closed-loop self-correction link. As the number of tracking frames increases, the individual slip factor is continuously iteratively optimized, gradually approaching the true slip characteristics of the impurities, and the trajectory prediction accuracy continuously improves with the operation process.

[0053] In one embodiment, step S105 above, which calculates the time it takes for the impurity to reach the removal position based on the updated impurity position and the individual slip factor, and triggers the removal action, specifically includes: establishing a mapping relationship between image coordinates and conveyor belt physical coordinates; and calculating the actual movement speed of the impurity based on the real-time speed of the conveyor belt and the individual slip factor. The arrival time is calculated based on the physical distance from the impurity location to the removal location and the actual movement speed. The rejection trigger time is determined by combining the action delay of the rejection mechanism, and the rejection pulse window is adaptively adjusted based on the estimated variance of the individual sliding factor to output the rejection signal.

[0054] The specific implementation is as follows: First, the mapping relationship between image coordinates and physical coordinates of the conveyor belt is established through offline camera calibration, and the corresponding homography matrix is ​​obtained. The image coordinates of the impurities are converted into physical coordinates of the conveyor belt using the homography matrix, and the physical coordinates of the rejection position corresponding to the rejection mechanism are determined. The physical distance from the current position of the impurity to the rejection position along the conveyor belt conveying direction is then calculated.

[0055] Secondly, the real-time speed of the conveyor belt is calculated based on the encoder pulse frequency. Combined with the individual slip factor updated in the current frame, the actual resultant speed of the impurity along the conveying direction is calculated. The value of the actual resultant speed is the sum of the real-time speed of the conveyor belt multiplied by 1 and the individual slip factor.

[0056] Next, using the physical distance and the actual velocity of motion, the remaining time for the impurity to reach the rejection position is calculated. The value of the remaining time is the ratio of the physical distance to the actual velocity of motion. The inherent motion delay of the rejection mechanism is obtained in advance through high-speed photography calibration. The current time is added to the remaining time, and then the motion delay is subtracted to obtain the precise rejection trigger time.

[0057] Example 2: Based on the same technical concept, Example 2 of this invention also provides a DeepSORT-based multi-target impurity continuous tracking and removal trajectory prediction system, which implements a DeepSORT-based multi-target impurity continuous tracking and removal trajectory prediction method. Figure 4 As shown, it includes: an information acquisition module 210, a location prediction module 220, a matching update module 230, a feedback module 240, and an action execution module 250; Among them, the information acquisition module 210 is used to acquire the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; The position prediction module 220 is used to input the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity. The matching update module 230 matches the predicted impurity target position with the impurity corresponding to the detection position information, and updates the position and individual sliding factor of the impurity synchronously based on the matching result. The feedback module 240 is used to feed the updated individual slip factor back to the conveyor belt constrained motion model for position prediction in the next cycle. The action execution module 250 is used to calculate the time it takes for an impurity to reach the removal position based on the updated impurity position and the individual slip factor, and to trigger the removal action.

[0058] Example 3: In one embodiment, Example 3 of the present invention also provides an electronic device; the electronic device may be a terminal, and its internal structure diagram may be as follows. Figure 5As shown. The electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the DeepSORT-based multi-target impurity continuous tracking and removal trajectory prediction method described in any of steps S101 to S105. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0059] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A multi-target impurity continuous tracking and removal trajectory prediction method based on DeepSORT, characterized in that, include: Acquire the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; The inter-frame displacement increment and the individual slip factor are input into the conveyor belt constrained motion model to predict the target position of the impurity. The predicted impurity target location is matched with the impurity corresponding to the detection location information, and the impurity position and individual slip factor are updated synchronously based on the matching result; The updated individual slip factor is fed back to the conveyor belt constrained motion model for position prediction in the next cycle. The time it takes for an impurity to reach the removal position is calculated based on the updated impurity position and the individual slip factor, triggering the removal action.

2. The method according to claim 1, characterized in that, The process of obtaining the inter-frame displacement increment of the conveyor belt includes: Based on the synchronous trigger signal, the cumulative displacement of the conveyor belt corresponding to the image timestamp is obtained; The inter-frame displacement increment is obtained by calculating the difference between the cumulative displacements of the conveyor belt in adjacent frames.

3. The method according to claim 1, characterized in that, Before predicting the target location of the impurity, the following is also included: The unit vector of the conveyor belt's motion direction is pre-calibrated offline, and then input into the conveyor belt's constrained motion model.

4. The method according to claim 3, characterized in that, The step of inputting the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity includes: Construct a three-dimensional joint state vector containing the horizontal and vertical coordinates of the impurities and the individual slip factor; The inter-frame displacement increment, the unit vector of the conveyor belt motion direction, and the historical individual slip factor are input into the conveyor belt constrained motion model to solve for the target position prediction coordinates of the impurity. Based on a symmetric sigma point set, an unscented transformation is performed on the three-dimensional joint state vector to solve for the trajectory prediction position and prediction covariance.

5. The method according to claim 1, characterized in that, The synchronous update of the impurity position and individual slip factor based on the matching results includes: Construct a linear observation equation for the location coordinates of the impurities; The successfully matched detection location information is used as the observation value, and the state is updated by unscented Kalman filtering to output the updated three-dimensional joint state vector; wherein, the three-dimensional joint state vector includes the impurity location and the individual slip factor.

6. The method according to claim 1, characterized in that, The step of matching the predicted impurity target location with the impurity corresponding to the detection location information includes: Call the spatial texture joint features obtained from the detection and corresponding to the impurity detection location; Calculate appearance similarity based on the aforementioned spatial texture joint features; A joint cost matrix is ​​constructed by integrating the appearance similarity, motion statistical distance, motion direction consistency, and trajectory lifetime confidence. Based on the joint cost matrix, the existing impurity tracking trajectories are matched with the detection location information.

7. The method according to claim 6, characterized in that, The calculation of appearance similarity based on the joint features of spatial texture includes: Maintain a feature cache queue for each of the impurity tracking trajectories; The spatial texture joint features that satisfy the dynamic consistency test are stored in the feature cache queue; Features in the feature cache queue are aggregated by temporal attention weights to generate a dynamic trajectory appearance vector. The cosine similarity between the query vector obtained from the spatial texture joint feature mapping and the trajectory dynamic appearance vector is calculated as the appearance similarity.

8. The method according to claim 6, characterized in that, The method of fusing appearance similarity, motion statistical distance, motion direction consistency, and trajectory survival confidence to construct a joint cost matrix includes: The cost value corresponding to the appearance similarity, motion statistical distance, motion direction consistency and trajectory lifetime confidence is calculated respectively. The total cost of a single set of matching is obtained by weighted summation of each cost value. The joint cost matrix is ​​formed by filling the total cost of matching all trajectories with all detected targets. The trajectory lifetime confidence is calculated based on the number of consecutive trajectory matches, and the corresponding cost is obtained by generating a logarithmic penalty term based on the trajectory lifetime confidence.

9. The method according to claim 1, characterized in that, The step of calculating the time it takes for an impurity to reach the removal position based on the updated impurity position and the individual slip factor, and triggering the removal action, includes: Establish a mapping relationship between image coordinates and conveyor belt physical coordinates; The actual speed of the impurities is calculated based on the real-time speed of the conveyor belt and the individual slip factor. The arrival time is calculated based on the physical distance from the impurity location to the removal location and the actual movement speed. The rejection trigger time is determined by combining the action delay of the rejection mechanism, and the rejection pulse window is adaptively adjusted based on the estimated variance of the individual sliding factor to output the rejection signal.

10. A multi-target impurity continuous tracking and removal trajectory prediction system based on DeepSORT, characterized in that, include: The information acquisition module is used to acquire the inter-frame displacement increment of the conveyor belt and the detection location information of impurities in the material; The position prediction module is used to input the inter-frame displacement increment and the individual slip factor into the conveyor belt constrained motion model to predict the target position of the impurity. The matching and updating module matches the predicted impurity target position with the impurity corresponding to the detection position information, and updates the position and individual sliding factor of the impurity synchronously based on the matching result. The feedback module is used to send the updated individual slip factor back to the conveyor belt constrained motion model for position prediction in the next cycle. The action execution module is used to calculate the time it takes for an impurity to reach the removal position based on the updated impurity position and the individual sliding factor, and then trigger the removal action.

Images