A machine vision processor and method for an industrial robot dynamic sorting system

Abstract

The application discloses a kind of machine vision processor and method for industrial robot dynamic sorting system, including data acquisition module, parallel processing unit, logic control unit and shared memory unit, wherein, data acquisition module is used to real-time acquisition video image sequence;Parallel processing unit is used to extract trajectory based on input video image sequence, detect the appearance feature and motion feature of target, and predict the position distribution of next frame image based on the extracted feature, and calculate the adaptive matching cost matrix between trajectory and detection;Logic control unit is used to match and associate between target trajectory and new detection target based on the adaptive matching cost matrix, and output tracking information;Shared memory unit is used for data sharing and interaction between parallel processing unit and logic control unit.The application effectively solves the general processor computing power bottleneck problem, and is suitable for high-precision dynamic sorting of industrial robot in complex scene.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and more specifically, to a machine vision processor and method for a dynamic sorting system for industrial robots. Background Technology

[0002] In demanding applications such as dynamic sorting in industrial robots, multi-object tracking (MOT) is a core technology for achieving precise grasping and sorting. To address the complexities common in industrial environments, such as occlusion, varying lighting conditions, and variable target motion, academia and industry have developed increasingly sophisticated MOT algorithms. These algorithms employ strategies such as multimodal feature fusion, probabilistic state prediction (e.g., Gaussian Mixture Models), and adaptive data association. While these advanced algorithms theoretically improve tracking robustness and accuracy, their computational complexity also increases dramatically, involving large-scale feature extraction, neural network inference, and complex matrix operations.

[0003] However, existing technologies encounter severe computational bottlenecks and real-time challenges when deploying these advanced algorithms to actual industrial production lines. Currently, industrial sorting systems mainly rely on general-purpose processors (such as standard industrial computer CPUs) or general-purpose vision processing cards (such as general-purpose GPUs).

[0004] When using a general-purpose CPU, its serial processing architecture is severely inadequate in handling the massive parallel computing tasks in the algorithm (such as image feature extraction and multiple hypothesis prediction), resulting in an extremely low frame rate for the entire system, which is far from meeting the millisecond-level response required for industrial sorting.

[0005] When attempting to use a general-purpose GPU for assistance, the system becomes limited by the efficiency of collaboration between the CPU and GPU. General-purpose hardware architectures and drivers are not optimized for the specific data flow of the MOT algorithm (i.e., the feature extraction-fusion-prediction-association loop). The tightly coupled parallel computation (suitable for GPUs) and serial logical decision-making tasks (suitable for CPUs) within the algorithm cannot be effectively decoupled and scheduled in general-purpose systems. This results in frequent data copying overhead between the CPU and GPU, task waiting latency, and bus bottlenecks, causing the overall real-time performance to fall short of expectations despite the use of a GPU. Summary of the Invention

[0006] Based on the analysis of existing technologies, there is an urgent need in industrial settings for a new type of machine vision processor. This processor is no longer a simple collection of general-purpose computing units, but a dedicated processing system designed specifically for advanced MOT algorithms. This system needs to be able to deeply decouple complex tracking algorithm tasks and efficiently map different types of computational tasks to optimal heterogeneous hardware units for execution, thereby completely solving the computing power bottleneck of general-purpose hardware and meeting the stringent real-time requirements of dynamic sorting in industry.

[0007] In view of this, this invention proposes a machine vision processor for a dynamic sorting system of industrial robots. Its inventive point lies in designing a heterogeneous hardware architecture and task mapping scheme tailored to the computational characteristics of multi-target tracking algorithms in industrial sorting. This scheme deeply decouples the core processes of the MOT algorithm—feature extraction → fusion → prediction → association—and maps them to dedicated heterogeneous hardware units: large-scale parallel computing tasks are embedded as the first set of instructions, namely a highly optimized kernel function library burned into the GPU / NPU firmware, executed directly in hardware acceleration; serial decision-making and control tasks are embedded as the second set of instructions, namely a streamlined main control program running on the CPU, achieving precise scheduling through interrupts and state machines. This embedded instruction design ensures the determinism and efficiency of the algorithm execution path, avoiding the uncertain latency caused by dynamic scheduling and drive overhead in general systems. The parallel processing unit and the logic control unit are tightly interconnected through a high-performance system bus with ultra-high bandwidth and extremely low latency, ensuring zero-copy and instantaneous interaction of key intermediate data such as feature vectors and cost matrices in the shared memory unit, completely eliminating the frequent data transfer and synchronization waiting bottlenecks in general CPU+GPU solutions. Through precise mapping of hardware tasks and high-performance interconnection and collaboration, this processor achieves millisecond-level end-to-end real-time tracking of the MOT algorithm in complex industrial scenarios, significantly exceeding the performance limit of general heterogeneous platforms, and providing a robust and deployable visual computing core for dynamic sorting of industrial robots.

[0008] To achieve the above objectives, a first aspect of the present invention provides a machine vision processor for a dynamic sorting system for industrial robots, comprising: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit, composed of a GPU or NPU, is configured with a first set of instructions to extract the trajectory, detect the appearance features and motion features of the target based on the input video image sequence, predict the possible position distribution of the next frame image based on the extracted features, and calculate the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory, and is used to characterize the matching cost between the trajectory and the detection. The logic control unit, composed of a CPU, is configured with a second set of instructions, including a matching association module and a trajectory management and output module. The matching association module is used to match and associate the target trajectory with the newly detected target based on the adaptive matching cost matrix. The trajectory management and output module is used to output tracking information and manage the trajectory according to the matching association result. And a shared memory unit, consisting of a dual-port RAM, for data sharing and interaction between the parallel processing unit and the logic control unit.

[0009] In one implementation, the parallel processing unit includes: The multimodal feature extraction module is used to extract trajectory, appearance features of the detected target, and motion features in parallel from the input video image sequence. The appearance features include size, shape, and texture, and the motion features include displacement, direction, and velocity. The attention fusion and uncertainty quantification module, which is connected to the multimodal feature extraction module, is used to generate trajectory fusion feature vectors for the extracted appearance features and motion features of the detection target; estimate the feature uncertainty of each feature; and generate the reliability weights of the corresponding trajectories. The probabilistic state prediction and data association module, connected to the attention fusion and uncertainty quantification module, is used to predict the probabilistic multi-hypothesis position distribution of the trajectory target in the next frame image based on the motion features, which consists of multiple weighted candidate regions. It also constructs an adaptive cost matrix for all trajectories and all detected targets, where the matching cost between any trajectory and the detected target is jointly determined by the trajectory and the detected target based on reliability weights.

[0010] In one implementation, the trajectory fusion feature vector takes the form of: , in, , These are mapping networks pre-trained for appearance features and motion features, respectively; , The first The first of the trajectories Frame appearance feature vector, motion feature vector; , and These represent the number of extracted appearance features, the number of motion features, and the total number of frames, respectively. , The first The appearance features of the trajectory are fused into feature vectors, and the motion features are fused into feature vectors.

[0011] In one implementation, the reliability weight is calculated as follows: , in, For the first Reliability weights of the trajectories For the Sigmoid function; This is a mapping network pre-trained to address feature biases.

[0012] In one implementation, the probabilistic state prediction and data association module uses a Gaussian mixture model to characterize the probabilistic multi-hypothesis location distribution, for the first... The motion characteristics of the trajectory, and the probability density of the trajectory target at any position in the next frame are: :

[0013] in, To detect the two-dimensional position vector of the target, , These are the GMM component number and the total number, respectively. For GMM The probability density function of each component. , and GMM No. The weights, mean vectors, and covariance matrices of each component are derived from a matrix with the first component as the first component. The motion features of the trajectory are generated by the input prediction network.

[0014] In one implementation, the matching cost is calculated as follows: , in, For the first Reliability weights for each trajectory; For the first Trajectory and the first The distance to the appearance features of each detected target is calculated using the following formula: , in, , The first Trajectory, Number The appearance feature vector of each detected target; To detect the number of targets, For the first Trajectory and the first The motion offset distance of each detected target is calculated using the following formula: , in, , The first The expected position coordinates of the trajectory in the next frame, the first... The measured coordinates of each detection target The calculation formula is: , in, In order to target the The motion characteristics of the trajectory, and the probability density of the trajectory target at any position in the next frame.

[0015] In one implementation, the matching and association module is specifically used for: Data association is completed by running a preset matching algorithm based on the adaptive matching cost matrix. The preset matching algorithm solves the association matrix. The solution method is as follows: , in, To detect the number of targets, For the number of trajectories, It is a binary decision matrix; For binary decision variables, ,in, Indicates the first Trajectory and the first Each detection is associated with, This indicates that they are not related.

[0016] In one implementation, the trajectory management and output module is specifically used for: Based on the matching and association results, update, create, or delete the target trajectory and output tracking information.

[0017] Based on the same inventive concept, a second aspect of the present invention provides a machine vision processing method for a dynamic sorting system of an industrial robot, comprising: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit extracts the trajectory, appearance features and motion features of the detected target based on the input video image sequence, and predicts the possible position distribution of the next frame image based on the extracted features. It also calculates the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory. It is used to characterize the matching cost between the trajectory and the detection. The parallel processing unit is composed of a GPU or NPU and is configured with a first set of instructions. The matching and association module of the logic control unit uses the adaptive matching cost matrix to match and associate the target trajectory with the newly detected target. The trajectory management and output module outputs tracking information and manages the trajectory according to the matching and association results. The logic control unit CPU is configured with a second set of instructions. Data sharing and interaction between the parallel processing unit and the logic control unit are achieved by using a shared memory unit, which consists of a dual-port RAM.

[0018] In one implementation, a parallel processing unit extracts the trajectory, appearance features, and motion features of the detected target based on the input video image sequence, predicts the possible location distribution of the next frame image based on the extracted features, and calculates the adaptive matching cost matrix between the trajectory and the detection, including: The multimodal feature extraction module extracts trajectory, appearance features of the detected target, and motion features in parallel from the input video image sequence. The appearance features include size, shape, and texture, and the motion features include displacement, direction, and velocity. The attention fusion and uncertainty quantization modules are used to generate trajectory fusion feature vectors based on the extracted appearance and motion features of the detected targets; the uncertainty of each feature is estimated, and the reliability weights of the corresponding trajectories are generated. Based on the motion features, the probabilistic state prediction and data association module predicts the probabilistic multi-hypothesis location distribution of the trajectory target in the next frame image, which consists of multiple weighted candidate regions. Furthermore, for all trajectories and all detected targets, an adaptive cost matrix is ​​constructed, where the matching cost between any trajectory and the detected target is jointly determined by the trajectory and the detected target based on reliability weights. Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: This invention proposes a machine vision processor for a dynamic sorting system for industrial robots, which solves the real-time bottleneck of complex algorithms through a heterogeneous collaborative architecture. The core advantage of this processor lies in its heterogeneous collaborative design between the CPU and GPU / NPU, and its clear task division. It fixes or schedules computationally intensive, highly parallel tasks in the MOT algorithm to the GPU / NPU (parallel processing unit) for execution; simultaneously, it delegates complex sequence decision-making, logical judgment, and I / O control tasks to the CPU (logic control unit). This task decoupling and hardware mapping design avoids the scheduling chaos and waiting delays of general-purpose processors under mixed task loads, maximizes hardware utilization efficiency, achieves a qualitative leap in computational efficiency, and ensures that the entire complex tracking process can meet the stringent real-time requirements of industrial sorting.

[0019] Furthermore, the dedicated architecture design achieves a balance between algorithm performance and efficiency. Since this processor is specifically designed for advanced tracking algorithm flows such as "feature extraction-feature fusion-probability prediction-adaptive correlation," its internal data paths and memory architecture are optimized for this specific process. Compared to general-purpose CPU-GPU architectures, this solution significantly reduces unnecessary data transfer and communication overhead between heterogeneous units. This enables advanced, robust algorithms to run efficiently on resource-constrained industrial processors. Therefore, this invention not only solves the real-time problem but also makes it possible to deploy high-precision algorithms in industrial settings, achieving the dual goals of high robustness and high real-time performance.

[0020] Furthermore, this enhances the tracking robustness of industrial robots in complex scenarios. Based on the powerful real-time computing capabilities provided by the aforementioned heterogeneous architecture, this processor can stably run advanced robustness algorithm modules. For example, the attention fusion and uncertainty quantization module and the probabilistic state prediction module can dynamically evaluate feature reliability and predict complex movements in each frame. This enables the machine vision system to maintain stable tracking even when the target is occluded, its appearance changes drastically, or its motion trajectory is nonlinear, significantly reducing the probability of identity switching and trajectory loss, and directly improving the accuracy and reliability of downstream industrial robot sorting operations. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the system architecture of the machine vision processor in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system hardware structure of the machine vision processor in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the interaction between the machine vision processor and external components in an embodiment of the present invention. Figure 4 This is a schematic diagram of the sorting system's working process in an embodiment of the present invention. Detailed Implementation

[0023] This embodiment discloses a machine vision processor and method for a dynamic sorting system of industrial robots. The processor includes a data acquisition module, a parallel processing unit, a logic control unit, and a shared memory unit. The parallel processing unit, including a graphics processing unit (GPU), a neural processing unit (NPU), and a logic control unit (CPU), interacts with the CPU via dual-port random access memory (RAM). The parallel processing unit performs multimodal feature extraction, uncertainty quantization fusion, probabilistic state prediction, and adaptive cost calculation; the logic control unit is used for task scheduling, matching association, and trajectory management. By allocating computationally intensive tasks to the GPU / NPU and logic decision-making tasks to the CPU, algorithm-level task decoupling and hardware mapping collaboration are achieved, significantly improving the real-time performance and robustness of multi-target tracking. This invention effectively solves the shortcomings of traditional methods in handling feature uncertainty, nonlinear motion prediction, and data association robustness, as well as the bottleneck of general-purpose processor computing power. It is suitable for high-precision dynamic sorting of industrial robots in complex scenarios.

[0024] Example 1 This embodiment discloses a machine vision processor for a dynamic sorting system of industrial robots. Please refer to [link to relevant documentation]. Figure 1 ,include: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit, composed of a GPU or NPU, is configured with a first set of instructions to extract the trajectory, detect the appearance features and motion features of the target based on the input video image sequence, predict the possible position distribution of the next frame image based on the extracted features, and calculate the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory, and is used to characterize the matching cost between the trajectory and the detection. The logic control unit, consisting of a CPU, is configured with a second set of instructions, including a matching and association module and a trajectory management and output module. The matching and association module is used to match and associate the target trajectory with the newly detected target based on the adaptive matching cost matrix. The trajectory management and output module is used to output tracking information and manage the trajectory based on the matching and association results. It also includes a shared memory unit, consisting of a dual-port RAM, used for data sharing and interaction between the parallel processing unit and the logic control unit.

[0025] Specifically, the device in this embodiment includes: a data acquisition module, a random access memory (RAM) (shared memory unit), and a processing system (parallel processing unit and logic processing unit). The data acquisition module is used to acquire data, the RAM is used to store image frames, intermediate feature data, and trajectory history information, and the processing system consists of the following core hardware: GPU / NPU: As the primary parallel computing unit, it is responsible for performing large-scale, parallelizable numerical computation tasks, including: all feature extraction (multimodal features), feature uncertainty estimation, GMM parameter prediction, and calculation of distance and position probabilities for all features in the adaptive cost matrix. Its advantage lies in its massively parallel architecture, which greatly accelerates the inference and matrix operations of deep learning models, ensuring that the system can meet real-time processing requirements.

[0026] Central Processing Unit (CPU): As the main control unit, it is responsible for executing sequence decision-making, logical judgment, and I / O control tasks, including: controlling the data acquisition module, overall task scheduling, solving matching algorithms such as data association (belonging to sequence decision-making logic), and trajectory management and output. Its advantage lies in its proficiency in highly complex logic control and task management, ensuring the stable and efficient operation of the entire system.

[0027] In the specific implementation process, the parallel processing unit is specifically configured and has a first set of instructions for executing: based on the input video image sequence, extracting multimodal features of the trajectory and the appearance (size, shape, texture, etc.) and motion (displacement, direction, velocity, etc.) of the detected target in parallel; quantifying the reliability weights of each feature based on preset cue information; predicting the Gaussian mixture model parameters representing the target's multi-hypothetical position in the next frame based on the trajectory target motion features; and calculating the adaptive matching cost matrix between the trajectory and the detection, the mathematical form of which is: , ,in, To match the cost; , These are the trajectory and the detection sequence number, respectively. , These represent the number of trajectories and the number of detections, respectively.

[0028] The logic control unit is specifically configured and has a second set of instructions for executing: scheduling tasks of the parallel processing unit, whose tasks include feature extraction, attention fusion, probabilistic state prediction, uncertainty quantification, etc.; receiving the cost matrix and running matching algorithms to complete data association, wherein the matching algorithms include, but are not limited to, the Hungarian algorithm, the JV algorithm, etc.

[0029] It manages target trajectories based on association results and outputs tracking information. The association results include all trajectory-detection matching pairs, unmatched trajectories, and unmatched detections. Target trajectory management includes four operations: trajectory generation, trajectory update, trajectory maintenance, and trajectory deletion.

[0030] The parallel processing unit, logic control unit, and shared memory unit are interconnected through the system bus. Through the coordinated execution of the aforementioned fixed instructions, real-time and robust tracking of multiple targets in the dynamic sorting scenario of industrial robots can be achieved.

[0031] In one implementation, the parallel processing unit includes: The multimodal feature extraction module is used to extract trajectory, appearance features of detected targets, and motion features in parallel from the input video image sequence. The appearance features include size, shape, and texture, while the motion features include displacement, direction, and velocity. The attention fusion and uncertainty quantification module, which is connected to the multimodal feature extraction module, is used to generate trajectory fusion feature vectors for the extracted appearance and motion features of the detected target; estimate the feature uncertainty of each feature; and generate the reliability weights of the corresponding trajectories. The probabilistic state prediction and data association module, connected to the attention fusion and uncertainty quantification module, is used to predict the probabilistic multi-hypothesis location distribution of the trajectory target in the next frame image based on motion features, which consists of multiple weighted candidate regions. It also constructs an adaptive cost matrix for all trajectories and all detected targets, where the matching cost between any trajectory and the detected target is jointly determined by the trajectory and the detected target based on reliability weights.

[0032] Specifically, the attention fusion and uncertainty quantification module, connected to the multimodal feature extraction module, is used to: generate the first [unclear] based on the target appearance features and motion features extracted from different frame images in the acquired video. Fusion feature vector of trajectories , The expression form is: , in, , These are mapping networks pre-trained for appearance features and motion features, respectively; , The first The first of the trajectories Frame appearance feature vector, motion feature vector; , and These represent the number of extracted appearance features, the number of motion features, and the total number of frames, respectively. The uncertainty of each feature is estimated, and the reliability weights of the corresponding trajectories are generated; the attention fusion and uncertainty quantification module calculates the first... Reliability weight of fusion features of trajectories : , in, For the Sigmoid function; This is a mapping network pre-trained to address feature biases.

[0033] As shown above, the reliability weight is calculated by inputting the appearance feature vector and motion feature vector into a pre-trained mapping network. In, and via activation function Output.

[0034] The probabilistic state prediction and data association module, connected to the attention fusion and uncertainty quantification module, predicts the probabilistic multi-hypothesis location distribution of the trajectory target in the next frame image, which consists of multiple weighted candidate regions. For all trajectories and all detected targets, an adaptive cost matrix is ​​constructed. The matching cost between any trajectory and detection is determined by the reliability weight of the feature and the probability value of the detection falling in the probabilistic multi-hypothesis location distribution of the trajectory. Based on the adaptive cost matrix, the matching association between the trajectory target and the detected target is completed.

[0035] The probabilistic state prediction and data association module uses a Gaussian mixture model (GMM) to characterize the probabilistic multi-hypothesis location distribution, given the first hypothesis. The motion characteristics of the trajectory, the probability density of the trajectory target at any position in the next frame is defined as follows: Its mathematical form is: , in, This is the two-dimensional position vector of the target being detected; , These are the GMM component number and the total number, respectively. For GMM The probability density function (PDF) of each component. , and GMM No. The weights, mean vectors, and covariance matrices of each component are derived from a matrix with the first component as the first component. The motion features of the trajectory are generated by the input prediction network.

[0036] Specifically, a prediction network is used to predict the weights, mean vectors, and covariance matrices of multiple Gaussian components for each trajectory, thereby constructing a GMM that can represent various potential motion modes such as straight-line and turning.

[0037] In one implementation, the matching cost in the adaptive cost matrix Determined by the following function: , in, For the first Reliability weights for each trajectory; For the first Trajectory and the first The distance to the appearance features of each detected target is calculated using the following formula: , in, , The first Trajectory, Number The appearance feature vector of each detected target; For the first Trajectory and the first The motion offset distance of each detected target is calculated using the following formula: , in, , The first The expected position coordinates of the trajectory in the next frame, the first... The measured coordinates of each detection target The calculation formula is: , in, In order to target the The motion characteristics of the trajectory, and the probability density of the trajectory target at any position in the next frame.

[0038] Specifically, the adaptive matching cost calculation ensures that when the reliability weight of the target's appearance features is high, the matching cost is mainly determined by the distance of the appearance features; when the reliability weight is low, the matching cost automatically switches to be mainly determined by the distance of the motion features. At the same time, the cost will be significantly reduced because the detected target's position falls in the high-probability area of ​​the trajectory prediction.

[0039] In one implementation, the matching and association module of the logic control unit is specifically used for: Data association is completed by running a preset matching algorithm based on the adaptive matching cost matrix. The preset matching algorithm solves the association matrix. The solution method is as follows: , in, To detect the number of targets, For the number of trajectories, It is a binary decision matrix; For binary decision variables, ,in, Indicates the first Trajectory and the first Each detection is associated with, This indicates that they are not related.

[0040] The division of labor among the hardware modules in the processing system in this embodiment is as follows: GPUs or NPUs are specifically designed for performing multimodal feature extraction, feature uncertainty estimation, GMM parameter prediction, and parallel computation of feature distance and location probabilities in the adaptive cost matrix. The CPU is specifically used to control the data acquisition module, schedule overall tasks, solve matching algorithms in data association, and manage and output trajectories.

[0041] In other words, the large-scale, parallelizable numerical computation tasks in feature extraction, feature fusion and uncertainty quantification, probabilistic state prediction and adaptive matching cost calculation in this invention are scheduled to be executed on GPU or NPU. The sequence decision-making, logical judgment, and I / O control tasks in trajectory matching and trajectory management are scheduled to be executed by the CPU, thereby maximizing hardware utilization efficiency.

[0042] The machine vision processor provided in this application will be described below through a specific application of the machine vision processor in a sorting system, such as... Figure 4 The diagram illustrates the specific workflow of the machine vision processor provided in this embodiment within the sorting system. Upon material input, the system acquires images via video surveillance. The machine vision processor then performs image stitching and trajectory extraction, target recognition, and position prediction to obtain the target's trajectory and position. The system generates control commands, and the robot autonomously navigates to the material exit according to these commands, completing the sorting process.

[0043] Example 2 Based on the same inventive concept, this embodiment discloses a machine vision processing method for a dynamic sorting system of an industrial robot, including: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit extracts the trajectory, appearance features and motion features of the detected target based on the input video image sequence, and predicts the possible position distribution of the next frame image based on the extracted features. It also calculates the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory. It is used to characterize the matching cost between the trajectory and the detection. The parallel processing unit is composed of a GPU or NPU and is configured with a first set of instructions. The matching and association module of the logic control unit uses the adaptive matching cost matrix to match and associate the target trajectory with the newly detected target. The trajectory management and output module outputs tracking information and manages the trajectory according to the matching and association results. The logic control unit CPU is configured with a second set of instructions. Data sharing and interaction between the parallel processing unit and the logic control unit are achieved by using a shared memory unit, which consists of a dual-port RAM.

[0044] In practical implementation, the method of this embodiment is mainly achieved through the collaborative work of the CPU and GPU / NPU, and the specific steps are as follows: Step a: Parallel extraction of multimodal features Real-time acquired video image sequences and historical trajectory data stored in RAM are fed into the GPU / NPU. The GPU / NPU extracts visual appearance features (representing target identity) and motion features (representing historical motion trends) for all trajectories and detected targets in parallel.

[0045] Step b: Attention Fusion and Uncertainty Quantification Based on the appearance and motion features extracted in step a, the system needs to generate fusion features for each trajectory to characterize the overall characteristics of the trajectory, and quantify its reliability based on the deviation between the features of each frame and the fusion features. The core of this process is to calculate the reliability weight of the trajectory features.

[0046] Step c: Probabilistic state prediction The trajectory fusion feature vector generated in step b is input into a prediction network. This network predicts the probabilistic multi-hypothesis location distribution of the trajectory target in the next frame, represented using a Gaussian mixture model (GMM).

[0047] Step d: Adaptive data association Cost calculation (GPU / NPU): The GPU / NPU performs parallel calculations of the adaptive matching cost between the trajectory and the detected target; Matching solution: The CPU receives the cost matrix calculated by the GPU / NPU, executes the matching algorithm, and completes the optimal matching association between the trajectory and the detection.

[0048] Step d specifically includes: Step d1: Control the GPU or NPU to calculate the adaptive matching cost between the trajectory and the detection. This cost is a dynamic weighted sum of appearance feature distance and motion feature distance, and is adjusted by the probability that the detected target position falls in the trajectory GMM prediction distribution. The weights are the reliability weights calculated in step b. Step d2: Control the CPU to execute the matching algorithm based on the calculated cost matrix to complete the association between the trajectory and the detection. Step e: Track Management and Output Based on the matching and association results of step d, the CPU updates the status of successfully matched trajectories, creates new trajectories for unmatched detections, deletes trajectories that have not been matched for a long time, and finally outputs real-time tracking information.

[0049] This embodiment of the device achieves efficient real-time tracking through heterogeneous collaboration between CPU and GPU / NPU. The GPU / NPU handles high-throughput parallel computing, including feature extraction, feature fusion, uncertainty estimation, GMM prediction, and cost matrix generation, while the CPU handles low-latency logical decision-making, including matching algorithm solving, task scheduling, and trajectory management. The entire method follows a process of "multimodal fusion → dynamic reliability weighting → probabilistic multi-hypothesis prediction → adaptive cost association," achieving robust modeling and high-precision tracking of target identity and motion state in complex scenarios.

[0050] like Figure 2 As shown: Data acquisition: The data acquisition module captures video image sequences in real time and sends them to shared memory (RAM).

[0051] Task Initiation: The CPU's task scheduling module initiates the entire process, sending scheduling instructions to the GPU on one hand and controlling data acquisition on the other.

[0052] GPU parallel computing: The GPU reads the image from RAM and the historical trajectory stored in the previous round by the CPU. It then sequentially executes the fixed steps a, b, and c (feature extraction, fusion, and GMM prediction). Finally, it executes step d1 to calculate the adaptive cost matrix and writes it back to RAM.

[0053] CPU logic processing: The CPU's matching algorithm module (step d2) reads the cost matrix from RAM and completes the data association. The trajectory management module (step e) reads and writes trajectory data from RAM based on the association results, performing updates, creations, or deletions. Output: The CPU finally outputs the processed trajectory results to the industrial robot.

[0054] This process clearly demonstrates the design where the GPU / NPU is responsible for large-scale parallel computing (steps a, b, c, d1), while the CPU is responsible for logic control, scheduling, and serial decision-making (steps d2, e). The two interact efficiently through shared memory to achieve real-time tracing.

[0055] In addition, such as Figure 3 As shown, the core objective of the system is to track moving sorting targets in real time and accurately, and guide industrial robots to operate on them. The entire process begins with external input, namely, the video acquisition device's camera capturing real-time video image sequences of the targets on the conveyor belt, and sending this raw visual data to the core of the system: the machine vision processor.

[0056] Inside the processor, data is first received by the data acquisition module and immediately placed into a shared memory unit (dual-port RAM). This shared memory is the critical data hub of the entire system, acting as a high-speed cache that allows the two different cores in the processor, the logic control unit (CPU) and the parallel processing unit (GPU / NPU), to efficiently share and interact with data, avoiding transmission bottlenecks.

[0057] The brain of the system is the logic control unit (CPU). It plays the role of the overall commander, responsible for task scheduling of the entire system. It reads the processing results of the previous round from shared memory and then decides what needs to be calculated next. The CPU excels at handling serial logic tasks, so it is also responsible for running data association matching algorithms and trajectory management.

[0058] The muscle of the system is the parallel processing unit (GPU / NPU). The CPU schedules computationally intensive, parallelizable tasks to the GPU / NPU for execution. The GPU / NPU utilizes its numerous computing cores to perform intensive processing of video data simultaneously, including: multimodal feature extraction (e.g., analyzing target appearance and motion), uncertainty quantification (e.g., determining feature reliability), GMM parameter prediction (e.g., predicting multiple possible target locations), adaptive cost matrix calculation (e.g., preparing data for the CPU's matching tasks).

[0059] Ultimately, the system forms a high-speed closed loop through the collaborative work of the CPU and GPU. After completing parallel computation, the GPU writes the results back to shared memory; the CPU reads these results, quickly completes data association and trajectory decisions, and then issues precise real-time tracking and prediction commands to the external output module, namely the industrial robot. The robot executes sorting actions according to these commands. Simultaneously, the illustration also shows that the characteristics (contextual cues) and motion state of the target itself are continuously captured by the vision system, forming a continuously feedback and corrective closed loop, ensuring the robustness and real-time performance of the entire dynamic sorting process.

[0060] This embodiment introduces uncertainty quantization and attention fusion. Through a dynamic weighting mechanism, it solves the problem of feature failure in traditional fixed fusion under drastic changes or occlusion of the target's appearance, ensuring the continuous reliability of the target feature vector. Using a Gaussian Mixture Model (GMM) for state prediction effectively captures the nonlinear and multi-hypothesis characteristics of target motion, overcoming the prediction limitations of traditional linear models in complex scenes.

[0061] An adaptive matching cost function was designed, combining feature reliability weights and GMM probability prediction to achieve accurate and intelligent trajectory-detection association when the target is close or moving complexly, significantly reducing the occurrence of identity switching. Heterogeneous collaboration between CPU / GPU / NPU was implemented, offloading computationally intensive tasks to the GPU / NPU and retaining logical decision-making tasks for the CPU, maximizing hardware utilization efficiency and ensuring real-time processing even in complex scenarios.

[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A machine vision processor for a dynamic sorting system of industrial robots, characterized in that, include: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit, composed of a GPU or NPU, is configured with a first set of instructions to extract the trajectory, detect the appearance features and motion features of the target based on the input video image sequence, predict the possible position distribution of the next frame image based on the extracted features, and calculate the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory, and is used to characterize the matching cost between the trajectory and the detection. The logic control unit, composed of a CPU, is configured with a second set of instructions, including a matching association module and a trajectory management and output module. The matching association module is used to match and associate the target trajectory with the newly detected target based on the adaptive matching cost matrix. The trajectory management and output module is used to output tracking information and manage the trajectory according to the matching association result. And a shared memory unit, consisting of a dual-port RAM, for data sharing and interaction between the parallel processing unit and the logic control unit.

2. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 1, characterized in that, The parallel processing unit includes: The multimodal feature extraction module is used to extract trajectory, appearance features of the detected target, and motion features in parallel from the input video image sequence. The appearance features include size, shape, and texture, and the motion features include displacement, direction, and velocity. The attention fusion and uncertainty quantification module, which is connected to the multimodal feature extraction module, is used to generate trajectory fusion feature vectors for the extracted appearance features and motion features of the detection target; estimate the feature uncertainty of each feature; and generate the reliability weights of the corresponding trajectories. The probabilistic state prediction and data association module, connected to the attention fusion and uncertainty quantification module, is used to predict the probabilistic multi-hypothesis position distribution of the trajectory target in the next frame image based on the motion features, which consists of multiple weighted candidate regions. It also constructs an adaptive cost matrix for all trajectories and all detected targets, where the matching cost between any trajectory and the detected target is jointly determined by the trajectory and the detected target based on reliability weights.

3. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 2, characterized in that, The form of the trajectory fusion feature vector is: ， in, , These are mapping networks pre-trained for appearance features and motion features, respectively; , The first The first of the trajectories Frame appearance feature vector, motion feature vector; , and These represent the number of extracted appearance features, the number of motion features, and the total number of frames, respectively. , The first The appearance features of the trajectory are fused into feature vectors, and the motion features are fused into feature vectors.

4. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 3, characterized in that, The reliability weight is calculated as follows: ， in, For the first Reliability weights of the trajectories For the Sigmoid function; This is a mapping network pre-trained to address feature biases.

5. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 2, characterized in that, The probabilistic state prediction and data association module uses a Gaussian mixture model to characterize the probabilistic multi-hypothesis location distribution, for the first... The motion characteristics of the trajectory, and the probability density of the trajectory target at any position in the next frame are: : ， in, To detect the two-dimensional position vector of the target, , These are the GMM component number and the total number, respectively. For GMM The probability density function of each component. , and GMM No. The weights, mean vectors, and covariance matrices of each component are derived from a matrix with the first component as the first component. The motion features of the trajectory are generated by the input prediction network.

6. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 2, characterized in that, The matching cost is calculated as follows: ， in, For the first Reliability weights for each trajectory; For the first Trajectory and the first The distance to the appearance features of each detected target is calculated using the following formula: ， in, , The first Trajectory, Number The appearance feature vector of each detected target; To detect the number of targets, For the first Trajectory and the first The motion offset distance of each detected target is calculated using the following formula: ， in, , The first The expected position coordinates of the trajectory in the next frame, the first... The measured coordinates of each detection target The calculation formula is: ， in, In order to target the The motion characteristics of the trajectory, and the probability density of the trajectory target at any position in the next frame.

7. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 1, characterized in that, The matching and association module is specifically used for: Data association is completed by running a preset matching algorithm based on the adaptive matching cost matrix. The preset matching algorithm solves the association matrix. The solution method is as follows: ， in, To detect the number of targets, For the number of trajectories, It is a binary decision matrix; For binary decision variables, ,in, Indicates the first Trajectory and the first Each detection is associated with, This indicates that they are not related.

8. The machine vision processor for a dynamic sorting system of an industrial robot as described in claim 1, characterized in that, The trajectory management and output module is specifically used for: Based on the matching and association results, update, create, or delete the target trajectory and output tracking information.

9. A machine vision processing method for a dynamic sorting system of an industrial robot, characterized in that, include: The data acquisition module is used to acquire video image sequences in real time. The parallel processing unit extracts the trajectory, appearance features and motion features of the detected target based on the input video image sequence, and predicts the possible position distribution of the next frame image based on the extracted features. It also calculates the adaptive matching cost matrix between the trajectory and the detection. The adaptive matching cost matrix is ​​calculated based on the reliability weight of the features and the probability value of the detection falling in the probabilistic multi-hypothesis position distribution of the corresponding trajectory. It is used to characterize the matching cost between the trajectory and the detection. The parallel processing unit is composed of a GPU or NPU and is configured with a first set of instructions. The matching and association module of the logic control unit uses the adaptive matching cost matrix to match and associate the target trajectory with the newly detected target. The trajectory management and output module outputs tracking information and manages the trajectory according to the matching and association results. The logic control unit CPU is configured with a second set of instructions. Data sharing and interaction between the parallel processing unit and the logic control unit are achieved by using a shared memory unit, which consists of a dual-port RAM.

10. The machine vision processing method for a dynamic sorting system of an industrial robot as described in claim 9, characterized in that, Parallel processing units are used to extract trajectories, appearance features, and motion features of detected targets from the input video image sequence. Based on the extracted features, the possible location distribution of the next frame image is predicted, and the adaptive matching cost matrix between the trajectory and the detection is calculated, including: The multimodal feature extraction module extracts trajectory, appearance features of the detected target, and motion features in parallel from the input video image sequence. The appearance features include size, shape, and texture, and the motion features include displacement, direction, and velocity. The attention fusion and uncertainty quantization modules are used to generate trajectory fusion feature vectors based on the extracted appearance and motion features of the detected targets; the uncertainty of each feature is estimated, and the reliability weights of the corresponding trajectories are generated. Based on the motion features, the probabilistic state prediction and data association module predicts the probabilistic multi-hypothesis location distribution of the trajectory target in the next frame image, which consists of multiple weighted candidate regions. For all trajectories and all detected targets, an adaptive cost matrix is ​​constructed, where the matching cost between any trajectory and the detected target is jointly determined by the trajectory and the detected target based on reliability weights.

Images