Foreign matter sorting cooperative control method and system based on overhead track type double-arm robot arm

Abstract

The application discloses a foreign matter sorting cooperative control method and system based on a suspended rail type double-arm mechanical arm, and relates to the technical field of industrial control. The method comprises the following steps: collecting an electromagnetic induction signal and a surface image of a material flow, fusing and processing, identifying target foreign matters in the material flow, and generating foreign matter sensing information; according to the foreign matter sensing information and vibration motion parameters of a vibrating distributor, predicting a predicted motion trajectory of each target foreign matter, and on the basis of the attribute category of the target foreign matter and the corresponding predicted motion trajectory, planning non-interference sorting trajectories for a first mechanical arm and a second mechanical arm respectively and determining execution priorities; and according to the execution priorities and the sorting trajectories, controlling the first mechanical arm and the second mechanical arm. The application realizes high-precision identification of foreign matters, intelligent cooperative sorting and efficient conflict-free operation in a dynamic and complex environment.

Description

Technical Field

[0001] This invention relates to the field of industrial control technology, specifically to a collaborative control method and system for foreign object sorting based on a rail-mounted dual-arm robotic arm. Background Technology

[0002] In industrial automated production processes, for scenarios involving the handling of bulk materials such as coal preparation and ore sorting, the material flow is often mixed with metallic or non-metallic foreign objects. The presence of various foreign objects not only reduces the purity and quality of the final product, but may also cause wear or even serious damage to downstream key equipment such as crushing and conveying, thereby affecting the continuous and stable operation of the entire production line.

[0003] Currently, existing technologies for sorting foreign objects in material flows mostly rely on single types of sensors, such as visual sensors or metal detectors. These have significant limitations in complex and dynamic environments: visual solutions struggle to reliably distinguish between non-metallic foreign objects and normal materials that are similar in color and texture, and are susceptible to changes in lighting and dust interference; metal detection solutions cannot identify non-metallic foreign objects. Furthermore, existing robotic arm sorting systems typically use fixed sequences or simple rules for grasping, lacking accurate prediction of the dynamic movements of material flows, especially complex movements such as throwing and sliding caused by vibrating feeders, resulting in low grasping success rates. When multiple robotic arms are deployed collaboratively, existing methods often lack effective task allocation and collision avoidance trajectory planning mechanisms, easily leading to problems such as mutual interference between robotic arms and wasted waiting time, failing to achieve dynamic optimization and balance between sorting efficiency, operational safety, and system energy consumption. Summary of the Invention

[0004] This invention addresses the technical problems of existing technologies that rely on a single sensor, resulting in incomplete foreign object identification categories and insufficient reliability, as well as inaccurate prediction of dynamic material flow, low sorting efficiency, easy collisions, and high energy consumption. It provides a collaborative control method and system for foreign object sorting based on a rail-mounted dual-arm robotic arm.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm, comprising: Electromagnetic sensors and high-speed vision sensors arranged above the vibrating cloth synchronously acquire electromagnetic induction signals and material flow surface images of the material flow. The electromagnetic induction signal and the surface image of the material flow are fused together to identify the target foreign objects in the material flow and generate foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object. The attribute category includes metallic foreign objects or non-metallic foreign objects. Based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, the predicted motion trajectory of each target foreign object in the future time period is predicted. Based on the attribute category of the target foreign object and the corresponding predicted motion trajectory, the first robotic arm and the second robotic arm are planned to have non-interference sorting trajectories and the execution priority is determined. Based on the execution priority and sorting trajectory, the first robotic arm is controlled to attract and remove metallic foreign objects through the magnetic actuator at its end, and the second robotic arm is controlled to grab and remove non-metallic foreign objects through the gripper actuator at its end.

[0006] Secondly, the present invention provides a collaborative control system for foreign object sorting based on a rail-mounted dual-arm robotic arm, comprising: The sensor acquisition module is used to synchronously acquire electromagnetic induction signals and material flow surface images of the material flow through an electromagnetic sensor arranged above the vibrating cloth and a high-speed vision sensor. The information fusion and recognition module is used to fuse the electromagnetic induction signal and the surface image of the material flow, identify the target foreign objects in the material flow, and generate foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object, wherein the attribute category includes metallic foreign objects or non-metallic foreign objects. The trajectory prediction and planning module is used to predict the predicted motion trajectory of each target foreign object in the future time period based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, and to plan non-interference sorting trajectories for the first robotic arm and the second robotic arm and determine the execution priority based on the attribute category of the target foreign object and the corresponding predicted motion trajectory. The sorting execution control module is used to control the first robotic arm to attract and remove metallic foreign objects through the magnetic actuator at its end, and to control the second robotic arm to grab and remove non-metallic foreign objects through the gripper actuator at its end, according to the execution priority and sorting trajectory.

[0007] The beneficial effects of this invention are: Compared to existing technologies, this invention achieves comprehensive and reliable identification of both metallic and non-metallic foreign objects through synchronous acquisition and data fusion of electromagnetic sensors and high-speed vision sensors, overcoming the limitations of single-sensor technology. Secondly, this invention combines the motion parameters of the vibrating fabric feeder to perform high-precision prediction of the foreign object's trajectory, providing accurate timing and location information for subsequent grasping. Thirdly, based on the type of foreign object and the predicted trajectory, this invention dynamically plans interference-free motion paths for the two robotic arms and intelligently allocates execution priorities, effectively avoiding robotic arm conflicts and improving collaborative operation efficiency and safety. Finally, through differentiated configuration and collaborative control of the end effector, synchronous, efficient, and automatic sorting of the two types of foreign objects is achieved, improving the overall sorting system's adaptability, throughput, and energy efficiency. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the collaborative control method for foreign object sorting based on a suspended dual-arm robotic arm provided by the present invention. Figure 2 This is a schematic diagram of the structure of the collaborative control system for foreign object sorting based on a rail-mounted dual-arm robotic arm provided by the present invention.

[0009] Figure 3 The present invention provides a foreign object sorting device based on a rail-mounted dual-arm robotic arm.

[0010] In the attached diagram, the components represented by each number are as follows: Sensor acquisition module 11, information fusion and recognition module 12, trajectory prediction and planning module 13, and sorting execution control module 14. Detailed Implementation

[0011] Example 1, as Figure 1 As shown, this embodiment of the invention provides a collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm, including: S10: Electromagnetic induction signals and surface images of the material flow are simultaneously acquired by an electromagnetic sensor and a high-speed vision sensor arranged above the vibrating cloth. First, electromagnetic sensors and high-speed vision sensors are installed above the vibrating material distributor. The vibrating material distributor is a key piece of equipment used in bulk material handling processes such as coal preparation and ore sorting. Its main function is to uniformly and continuously transport the mixed materials to the downstream process through periodic vibration, while providing a dynamic working interface for online sorting during this process.

[0012] Electromagnetic sensors are devices that detect induced currents or magnetic field disturbances in materials due to changes in their electromagnetic properties. They are used for non-contact detection of the presence of metallic substances in material flows, such as sensitively responding to the passage of metallic foreign objects like iron and copper. High-speed vision sensors are industrial cameras equipped with high-speed image acquisition chips and processing units. They are used to clearly and continuously capture the appearance and texture details of material surfaces under conditions of strong vibration and rapid movement, such as acquiring image sequences containing shape, color, and surface structure information in real time.

[0013] Electromagnetic sensors and high-speed vision sensors arranged above the vibrating distributor can simultaneously acquire electromagnetic induction signals and surface images of the material flow. The material flow refers to the collection of bulk materials to be processed that continuously move and spread on the surface of the vibrating distributor bed, such as the mixture flow composed of raw coal, gangue, and debris in the coal preparation process.

[0014] Specifically, electromagnetic induction signals and surface images of the material flow are simultaneously acquired by an electromagnetic sensor and a high-speed vision sensor arranged above the vibrating material distributor, including: Based on the ambient lighting conditions above the vibrating fabric, the gain parameters, exposure time parameters, and supplementary light intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions. Under the image acquisition parameters, the high-speed vision sensor is controlled to acquire images of the material flow surface. The electromagnetic sensor is synchronously controlled to collect electromagnetic induction signals of the material flow, and the electromagnetic induction signals are aligned with the timestamp of the material flow surface image.

[0015] First, based on the ambient lighting conditions above the vibrating fabric distributor, the gain parameters, exposure time parameters, and supplementary lighting intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters suitable for the current ambient lighting conditions. Since the lighting environment in industrial settings can fluctuate due to day-night cycles, weather changes, or equipment obstruction, resulting in overexposure, underexposure, or insufficient contrast in the acquired material flow surface images, it is necessary to monitor the lighting conditions in real time and automatically adjust the optical parameters to ensure that the high-speed vision sensor can acquire clear and stable image data under different lighting conditions, providing a reliable foundation for subsequent visual analysis.

[0016] Specifically, based on the ambient lighting conditions above the vibrating fabric distribution device, the gain parameters, exposure time parameters, and supplementary lighting intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions, including: Obtain the average brightness and average contrast values ​​of the monitoring area above the vibrating fabric distributor; Calculate the brightness deviation between the average brightness value and the preset standard brightness value, and calculate the contrast deviation between the average contrast value and the preset standard contrast value; Based on the brightness deviation, the gain parameter and exposure time parameter of the high-speed vision sensor are adjusted, wherein the adjustment direction of the gain parameter and exposure time parameter is opposite to the direction of the brightness deviation; Based on the contrast deviation, the fill light intensity of the fill light is adjusted, wherein the direction of adjustment of the fill light intensity is opposite to the direction of the contrast deviation.

[0017] First, the average brightness and average contrast values ​​of the monitoring area above the vibrating fabric are acquired. Specifically, the average brightness value reflects the overall intensity of light in the monitoring area, while the average contrast value characterizes the degree of difference between bright and dark areas in the image. By calculating the average brightness and average contrast values ​​in real time, the lighting conditions of the current environment can be quantitatively assessed.

[0018] Secondly, the brightness deviation between the acquired average brightness value and the preset standard brightness value is calculated, and the contrast deviation between the average contrast value and the preset standard contrast value is also calculated. The preset standard brightness value and preset standard contrast value are ideal reference values ​​pre-set based on the optimal imaging quality of the high-speed vision sensor. They represent the light intensity and contrast level that, under specific material characteristics and working distances, allows for the clearest image details and the easiest feature extraction.

[0019] Brightness deviation is the algebraic difference between the average brightness of the current monitored area and the preset standard brightness. A positive value indicates that the environment is too bright, while a negative value indicates that the environment is too dark. Contrast deviation is the algebraic difference between the average contrast of the current image and the preset standard contrast. A positive value indicates that the contrast is too strong, while a negative value indicates that the contrast is insufficient. The purpose of calculating brightness and contrast deviations is to clarify the specific difference between the current lighting conditions and the ideal imaging conditions, thereby providing a precise quantitative basis for subsequent adaptive adjustment of parameters.

[0020] Furthermore, based on the calculated brightness deviation, the gain and exposure time parameters of the high-speed vision sensor are adjusted inversely. The sign and magnitude of the brightness deviation directly characterize the direction and degree of deviation of the ambient light intensity from the ideal state. The adjustment follows the principle that the adjustment direction of the gain and exposure time parameters must be opposite to the direction indicated by the brightness deviation. Specifically, when the monitoring data shows that the ambient light intensity is too high, i.e., there is a positive brightness deviation, the gain parameter value of the high-speed vision sensor and the set value of the exposure time parameter are reduced accordingly to suppress excessive charge accumulation in the sensor's photosensitive unit, thereby preventing overexposure of the acquired material flow surface image. Conversely, when the ambient light intensity is insufficient, i.e., there is a negative brightness deviation, the gain parameter value is increased and the set value of the exposure time parameter is increased to amplify the light signal intensity entering the sensor, compensate for insufficient illumination, and ensure that the image can acquire sufficient brightness information and signal-to-noise ratio.

[0021] Simultaneously, based on the calculated contrast deviation, the intensity of the supplementary lighting is adjusted in the opposite direction. The value and sign of the contrast deviation directly reflect the difference between the overall brightness and darkness of the acquired image and the preset ideal contrast level. The principle followed in the adjustment is that the direction of adjustment of the supplementary lighting intensity should be opposite to the direction indicated by the contrast deviation. Specifically, when the monitoring and analysis results indicate insufficient image contrast, i.e., a negative contrast deviation exists, this usually corresponds to an image that appears dark, flat, and with blurred material outlines and surface texture details. In this case, the intensity of the supplementary lighting will be increased to increase the amount of light incident on the surface of the material flow, thereby highlighting the geometric edges and micro-morphological features of the material and effectively improving the local and overall contrast of the image. Conversely, when the image contrast is detected to be too high, i.e., a positive contrast deviation exists, excessive brightness and darkness contrast may lead to loss of details in bright areas or significant noise in dark areas, or even cause local glare. In this case, the intensity of the supplementary lighting will be reduced to soften the lighting conditions, balance the brightness distribution in different areas of the image, and avoid adverse optical interference caused by excessive illumination.

[0022] Through the closed-loop adjustment process described above, the high-speed vision sensor can automatically adapt to changing ambient lighting and always operate under optimized parameters, thereby ensuring that the acquired material flow surface images have consistent high quality and rich analyzable features.

[0023] Furthermore, under defined image acquisition parameters, a high-speed vision sensor is controlled to acquire images of the moving material flow, thereby obtaining an image of the material flow surface. The acquired image of the material flow surface can realistically reflect the texture, shape, and color characteristics of the material surface, avoiding image distortion or information loss caused by improper parameters.

[0024] Simultaneously, electromagnetic sensors are controlled to scan the same segment of the material flow to collect corresponding electromagnetic induction signals. It is important to note that to ensure spatiotemporal consistency for subsequent multi-source data fusion and collaborative analysis, the timestamps of the electromagnetic induction signals and the material flow surface images must be precisely aligned. Specifically, through timestamp alignment, for identical physical locations or strictly corresponding time segments within the material flow, the electromagnetic characteristic data detected by the electromagnetic sensors and the visual feature data captured by the high-speed vision sensors are matched on a time coordinate. This provides a data foundation for the subsequent reliable correlation and fusion of the material's electromagnetic response intensity and its visual morphological characteristics.

[0025] S20: The electromagnetic induction signal and the surface image of the material flow are fused and processed to identify the target foreign objects in the material flow, and foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object is generated, wherein the attribute category includes metallic foreign objects or non-metallic foreign objects. Secondly, the acquired electromagnetic induction signals and material flow surface images are fused together. This fusion process integrates data sources from different physical sensing principles and performs feature-level or decision-level integration based on spatiotemporal correspondence. By combining the electrical and magnetic conductivity of the material reflected by the electromagnetic induction signals with the morphological, texture, and color visual features reflected by the material flow surface images, high-confidence identification and fine classification of target foreign objects in the material flow can be achieved. For each successfully identified target foreign object, structured foreign object perception information containing its attribute category, spatial location, and corresponding timestamp is generated to construct a complete, accurate, and temporally correlated dynamic list of foreign objects, providing core input for subsequent trajectory prediction and robotic arm collaborative sorting decisions.

[0026] Specifically, the electromagnetic induction signal and the surface image of the material flow are fused to identify target foreign objects in the material flow, and foreign object sensing information containing the attribute category, spatial location, and timestamp of each target foreign object is generated, including: Based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis. The electromagnetic induction signal is analyzed synchronously to obtain the electromagnetic response intensity of each material to be identified in space. The estimated three-dimensional volume, surface visual texture features, and electromagnetic response intensity of each material to be identified are input into a pre-trained multi-level classification model, wherein the multi-level classification model includes a first-level classification module and a second-level classification module connected in series. The first-level classification module classifies each material to be identified into normal material or target foreign object based on the estimated three-dimensional volume and surface visual texture features. The second-level classification module combines the electromagnetic response intensity of the target foreign object to classify it into a metallic or non-metallic foreign object and determines its corresponding spatial coordinates. Based on the classification results and spatial coordinates output by the second-level classification module, foreign object perception information is generated, which includes the attribute category, spatial location and collection timestamp of each target foreign object.

[0027] First, based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis methods.

[0028] The visual analysis method is an algorithmic process that integrates digital image processing, stereo vision, and machine learning techniques to extract and calculate the three-dimensional geometry and surface appearance information of a target object from a two-dimensional image. The three-dimensional volume estimation value is an approximate numerical value about the size of the material in three-dimensional space, obtained by analyzing and calculating the material's projected contour, motion parallax, and prior size model in the image. It represents the physical size of the material in space and can be used to distinguish between abnormally large or small foreign objects. Surface visual texture features are obtained by performing local statistical analysis and deep learning feature extraction on the material's surface image area. This yields a set of multi-dimensional feature vectors that quantify appearance attributes such as surface roughness, pattern regularity, and color distribution uniformity. These feature vectors represent the microstructure and optical reflection characteristics of the material surface and can be used to distinguish foreign objects that differ from normal materials in color, gloss, or texture.

[0029] In typical industrial scenarios such as coal preparation, the physical dimensions of normal materials, such as coal lumps and gangue, are constrained by upstream crushing and screening processes, typically exhibiting a stable and concentrated statistical distribution. By performing 3D reconstruction and point cloud segmentation on the material flow surface image, the estimated 3D volume of each independently segmented material can be calculated. For example, objects whose estimated 3D volume deviates significantly from the above-mentioned normal distribution range, such as excessively large blocks of wood or excessively small metal parts, will exhibit statistical anomalies in their volume parameters. These anomalies can serve as an important geometric basis for identifying target foreign objects.

[0030] Meanwhile, after undergoing a series of processes such as mining, crushing, and transportation, normal materials develop texture patterns on their surfaces with specific statistical regularities, such as characteristic surface roughness, gloss distribution, and granular structure. By applying deep learning methods such as convolutional neural networks, high-dimensional, quantified surface visual texture feature vectors can be extracted from the regions corresponding to the material surface images. For example, the surface visual texture features of an abnormally smooth plastic sheet, a patterned flexible packaging, or a wood product with regular fibrous texture are fundamentally different from the natural texture patterns of coal and gangue. This difference based on texture patterns can be effectively identified and distinguished by pre-trained classification models, thus constituting a key visual discrimination criterion for separating target foreign objects from the normal material background.

[0031] Specifically, based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis, including: Based on the material flow surface image, the three-dimensional point cloud data of the material flow surface is reconstructed using the grating deformation analysis method. The three-dimensional point cloud data is segmented, and based on the spatial aggregation characteristics of the three-dimensional point cloud data, a subset of the point cloud of each material to be identified in the material flow is separated. Based on the segmented point cloud subsets, calculate the estimated three-dimensional volume of each material to be identified; For each material to be identified, a two-dimensional image region matching the projection area of ​​the material to be identified is extracted from the surface image of the material flow; Using a convolutional neural network, surface visual texture features of each of the two-dimensional image regions are extracted, wherein the surface visual texture features include roughness, contrast and directionality values.

[0032] First, based on the acquired images of the material flow surface, a grating deformation analysis method is used to reconstruct the three-dimensional point cloud data of the material flow surface. Specifically, the grating deformation analysis method projects a specific structured grating pattern onto the material flow surface and uses a high-speed vision sensor to capture the deformation of the pattern caused by surface undulations. Based on the principle of optical triangulation, the three-dimensional coordinates of each point on the surface are calculated, thereby generating a dense and accurate set of three-dimensional point cloud data that characterizes the surface morphology of the material flow.

[0033] Secondly, the reconstructed 3D point cloud data is segmented. Specifically, this segmentation process is based on the spatial aggregation characteristics and continuity of the 3D point cloud data, such as clustering criteria based on the distance between points and the consistency of normal vectors, to separate point cloud clusters belonging to the same physical material from the background and other materials, thereby obtaining an independent point cloud subset corresponding to each material to be identified in the material flow.

[0034] Then, based on each segmented subset of the point cloud, a three-dimensional volume estimate of the corresponding material to be identified is calculated. Preferably, the calculation method typically involves voxelizing or reconstructing the 3D space occupied by the subset of point clouds to estimate the volume of the enclosed spatial region. This three-dimensional volume estimate quantifies the physical size of the material.

[0035] Furthermore, for each material to be identified that has undergone point cloud segmentation, a two-dimensional image region that perfectly matches the projection area of ​​the material's three-dimensional point cloud onto the image plane is extracted from the original material flow surface image from which it originates. This two-dimensional image region refers to an image sub-block recorded in the form of a pixel matrix, whose boundary contour is determined by projecting the material's three-dimensional point cloud onto the camera's imaging plane. It represents the complete surface visual information presented by the specific material at the time of acquisition, including all appearance features such as color, light reflection, and micro-texture structure.

[0036] Finally, a pre-trained convolutional neural network (CNN) is used to extract deep features from each extracted two-dimensional image region. The CNN is a deep learning model with multi-layer convolution and pooling structures, suitable for processing image data with grid-like topologies. This CNN has been trained on a large number of sample images containing various material surface textures, and its model parameters have been fully optimized. It can automatically learn and output feature vectors that provide a high-dimensional, non-linear quantitative description of the surface texture of the input image region. Specifically, the extracted surface visual texture features include roughness, representing the size and density distribution of texture particles; contrast, reflecting the strength of brightness differences and contrast in local image regions; and directional values, describing the regularity of the spatial orientation of texture lines or patterns. Roughness, contrast, and directional values ​​together constitute a comprehensive, structured, and digital description of the visual pattern of the material surface.

[0037] Furthermore, the electromagnetic induction signal is simultaneously analyzed to obtain the electromagnetic response intensity at the spatial location of each material to be identified. Specifically, the electromagnetic sensor can detect changes in the induced signal caused by the different electromagnetic properties of the material, and the intensity of this electromagnetic response directly reflects the material's conductivity and magnetism. For example, metallic substances usually elicit a strong electromagnetic response, while the response of non-metallic substances is relatively weak or exhibits a specific pattern.

[0038] Subsequently, the estimated 3D volume, surface visual texture features, and electromagnetic response intensity of each material to be identified are input into a pre-trained multi-level classification model. This multi-level classification model comprises a first-level classification module and a second-level classification module connected in series. The first-level classification module performs a preliminary judgment on each material to be identified based on the input estimated 3D volume and surface visual texture features, classifying it as normal material or a target foreign object. This first-level classification module primarily relies on significant deviations in the estimated 3D volume and statistical differences in the surface visual texture features to make the judgment.

[0039] For materials identified as target foreign objects by the first-level classification module, their data will be sent to the second-level classification module. Specifically, the second-level classification module combines the electromagnetic response intensity corresponding to the target foreign object to perform a refined classification, determining whether it is a metallic or non-metallic foreign object. At the same time, it determines the precise spatial coordinates of the target foreign object in the vibrating fabric distributor coordinate system.

[0040] Finally, based on the classification results and spatial coordinates output by the second-level classification module, a structured foreign object perception information is generated for each target foreign object. This foreign object perception information fully includes the target foreign object's attribute category, spatial location, and collection timestamp, constituting a dynamic and quantitative description of the foreign object's state in the material flow. It can be used as input data for subsequent motion trajectory prediction and sorting decisions.

[0041] S30: Based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, predict the predicted motion trajectory of each target foreign object in the future time period, and based on the attribute category of the target foreign object and the corresponding predicted motion trajectory, plan non-interference sorting trajectories for the first robotic arm and the second robotic arm and determine the execution priority. Furthermore, since the motion state of the target foreign object is directly affected by the periodic vibration of the vibrating material distributor, and the motion parameters of the vibrating material distributor determine the dynamic laws of material throwing and sliding, the predicted motion trajectory of each target foreign object in the future time period can be predicted based on the spatial position in the foreign object sensing information and the real-time vibration motion parameters of the vibrating material distributor. This predicted motion trajectory is a set of discrete or continuous data points describing the changes in the spatial coordinates of the target foreign object in a time series, characterizing the dynamic positional evolution of the foreign object relative to the vibrating material distributor and even the entire sorting workspace over a future period.

[0042] Based on this predicted motion trajectory, the time and space of the target object arriving at the optimal gripping position of the robotic arm can be accurately predicted. This provides a key basis for the advanced planning and efficient execution of the robotic arm's sorting action, ensuring that the sorting process can accurately capture the target in dynamic motion and maximize the success rate of gripping.

[0043] Specifically, based on the foreign object sensing information and the vibration motion parameters of the vibrating fabric, the predicted motion trajectory of each target foreign object in the future time period is predicted, including: The system acquires the real-time vibration phase signal of the vibratory fabric feeder drive motor, as well as the tilt angle, vibration frequency, and amplitude parameters of the vibratory fabric feeder bed. Based on the vibration frequency and amplitude parameters, the throwing index under the current vibration conditions is calculated. Based on the throwing index and the friction coefficient between the material and the vibrating cloth bed, the critical throwing phase angle at which the material begins to throw is determined. The current phase angle of the real-time vibration phase signal is compared with the critical throwing phase angle. If the current phase angle is less than the critical throwing phase angle, it is determined that the target foreign object is in a sliding state in contact with the bed surface. If the current phase angle is greater than or equal to the critical throwing phase angle, it is determined that the target foreign object is in a throwing state being thrown by the bed surface. If it is determined that the target foreign object is in a sliding state, then based on the Coulomb friction dynamics equation, according to the spatial position coordinates, real-time vibration phase signal, vibration frequency parameter, amplitude parameter and attribute category of the target foreign object in the foreign object sensing information, the predicted motion trajectory of the target foreign object is calculated in real time. If the target object is determined to be in a throwing state, the spatial coordinates, real-time vibration phase signal, tilt angle parameter, vibration frequency parameter, amplitude parameter, and target object attribute category from the object perception information are input into the pre-trained projectile trajectory prediction agent, and the predicted motion trajectory of the target object in the future time period is output.

[0044] First, the real-time vibration phase signal of the vibratory feeder drive motor, as well as the tilt angle, vibration frequency, and amplitude parameters of the vibratory feeder bed, are acquired in real time. The real-time vibration phase signal is a continuous quantity that changes periodically with time, expressed in angles or radians, used to accurately characterize the instantaneous angular position of the vibratory feeder drive motor within the current vibration cycle. The tilt angle parameter refers to the fixed angle formed between the vibratory feeder bed and the horizontal reference plane, used to determine the decomposition relationship of the gravitational component of the material in the normal and tangential directions of the bed. The vibration frequency parameter refers to the number of periodic movements completed per unit time by the vibratory feeder drive motor or bed, used to define the fast and slow periodic characteristics of the vibration motion. The amplitude parameter refers to the distance between the vibratory feeder bed and its maximum displacement position, used to quantify the magnitude of the bed's movement in a single vibration. These parameters together constitute the core physical quantities describing the motion of the vibratory feeder, completely defining the precise kinematic state of the vibratory feeder at the current moment.

[0045] Secondly, based on the obtained vibration frequency and amplitude parameters, the throwing index under the current vibration conditions is calculated. Specifically, the throwing index (Γ) can be calculated using the formula Γ=(A·(2πf)). 2 The formula is calculated as (·sinβ) / (g·cosα). Where A is the amplitude parameter, f is the vibration frequency parameter, β is the vibration direction angle (in this embodiment, it can be related to or equal to the tilt angle parameter α), g is the gravitational acceleration, and α is the tilt angle parameter. The throwing index is a dimensionless parameter whose value directly reflects the ratio of the component of vibration acceleration perpendicular to the bed surface to the gravitational acceleration, and is used to measure whether the vibration intensity is sufficient to cause the material to detach from the bed surface.

[0046] Furthermore, based on the calculated throwing index and the coefficient of friction between the material and the vibrating fabric bed, the critical throwing phase angle at which the material begins to throw is determined. The coefficient of friction can be dynamically obtained by looking up a table or by estimation based on the properties of the target foreign object. The critical throwing phase angle identifies the phase angle corresponding to the critical moment within one vibration cycle when the bed motion changes from pushing the material to sliding to throwing the material.

[0047] Furthermore, the current phase angle of the real-time vibration phase signal is compared with the calculated critical throwing phase angle. If the current phase angle is less than the critical throwing phase angle, it is determined that the target object is in a sliding state that is in contact with the bed surface; if the current phase angle is greater than or equal to the critical throwing phase angle, it is determined that the target object is in an air-throwing state that is being thrown up by the bed surface.

[0048] Specifically, if the target foreign object is determined to be in a sliding state, then based on the Coulomb friction dynamics equation, and combined with the spatial position coordinates, real-time vibration phase signal, amplitude parameters, vibration frequency parameters and target foreign object attribute categories in the foreign object sensing information, dynamic integral calculation is performed in real time to deduce the predicted motion trajectory of the target foreign object in the continuous sliding phase.

[0049] Specifically, the calculation process begins with the known current spatial coordinates of the target foreign object. Substituting the real-time vibration phase signal, amplitude parameters, and vibration frequency parameters into the simple harmonic motion equation, the instantaneous displacement, velocity, and acceleration of the vibrating fabric bed surface at any given moment under that phase can be accurately calculated. Combined with the bed surface tilt angle parameter, the acceleration components of the bed surface motion in the parallel and perpendicular directions to the bed surface can be further decomposed. The attribute category of the target foreign object implicitly contains key physical parameters such as its mass and friction coefficient. Based on Coulomb's law of friction—that the magnitude of sliding friction is equal to the product of the friction coefficient and the normal force, and its direction is opposite to the relative motion trend—a dynamic differential equation for the target foreign object on the bed surface can be established. This equation takes the driving acceleration of the bed surface, the gravitational component, and the sliding friction force as inputs, and uses numerical integration methods, such as the Runge-Kutta method, to progressively solve for the velocity and position changes of the target foreign object over future time periods, starting from the current moment. Through this real-time iterative calculation, the predicted trajectory of the target foreign object over future time periods can be derived under the assumption that it continues to slide in contact with the bed surface.

[0050] Furthermore, if the target object is determined to be in a throwing state, the spatial coordinates, real-time vibration phase signal, tilt angle parameter, vibration frequency parameter, amplitude parameter, and target object attribute category from the object's sensing information are used together as input feature vectors and input into a pre-trained projectile trajectory prediction agent. This projectile trajectory prediction agent, by learning from a large amount of projectile motion simulation or experimental data, can establish a nonlinear mapping relationship between the input parameters and complex projectile trajectories, and directly output the complete predicted motion trajectory of the target object in the future time period.

[0051] Specifically, the training process of the projectile trajectory prediction agent includes: Constructing an intelligent agent for predicting projectile trajectories based on machine learning; Based on the physical parameters of the vibrating fabric feeder and typical foreign objects, a parameterized digital twin simulation system is constructed. In the digital twin simulation system, by randomly setting the spatial coordinates of the sample, the vibration motion parameters of the sample, the properties of the foreign object and the initial state, a high-fidelity physics simulation is run to generate corresponding real projectile trajectories in batches, thus forming a sample dataset. The projectile trajectory prediction agent is trained using the sample dataset to obtain a trained projectile trajectory prediction agent.

[0052] First, a projectile trajectory prediction agent is constructed based on machine learning methods. Preferably, this projectile trajectory prediction agent can use a deep neural network model as its core architecture, and its design goal is to learn and establish a complex nonlinear mapping relationship between input state parameters and future projectile trajectories.

[0053] Secondly, based on the precise geometric dimensions and dynamic characteristics of the vibrating material distributor, as well as the physical parameters of typical foreign objects, a parametric digital twin simulation system is constructed. This digital twin simulation system is a high-fidelity computer model capable of accurately simulating the projectile motion of materials on the vibrating material distributor according to physical laws, such as Newtonian mechanics and contact dynamics.

[0054] Subsequently, in this digital twin simulation system, a series of sample conditions are programmed and randomly set for simulation, including the spatial coordinates of the foreign object, the vibration motion parameters of the sample, the properties of the foreign object, and its initial state. For each set of randomly set parameters, a high-fidelity multibody dynamics simulation is run to calculate the complete motion path of the target foreign object from being thrown up to falling back onto the bed surface or leaving the working area under that specific condition, i.e., a realistic projectile trajectory. By repeating this process extensively, projectile trajectory data covering various possible working conditions can be generated in batches, collectively forming a sample dataset for model training.

[0055] Finally, the pre-built projectile trajectory prediction agent is trained under supervised conditions using the generated sample dataset. During training, the state parameters in the sample data are used as input features, and the corresponding real projectile trajectories are used as supervision labels. The weight parameters in the agent's neural network are continuously optimized through backpropagation until the model performance reaches a preset convergence condition. The convergence condition is preset based on the training objective and evaluation metric. For example, it can be set to ensure that the average positional error between the predicted trajectory and the real trajectory on an independent validation dataset is less than 5 mm, and that this metric does not decrease further within 20 consecutive training epochs. After training, a projectile trajectory prediction agent with strong generalization ability, capable of quickly and accurately predicting projectile trajectories based on real-time input parameters, is obtained.

[0056] For example, the projectile trajectory prediction agent can employ an encoder-decoder architecture. The encoder is a multi-layer fully connected neural network. Its input layer receives a standardized feature vector containing encodings of spatial location coordinates, real-time vibration phase, tilt angle, vibration frequency, amplitude, and foreign object attribute category. The number of neurons in the hidden layer is configured according to the input dimension. Each layer uses the ReLU activation function to introduce non-linear modeling capability, and Dropout layers are embedded between layers with a dropout rate set to 0.3 to enhance the model's generalization ability. The decoder is also a fully connected network. The number of neurons in its output layer corresponds to the dimension of the trajectory point coordinates in the future time period. It uses a linear activation function and directly outputs the predicted trajectory sequence coordinates.

[0057] During training, the key hyperparameters were set as follows: the initial learning rate was 0.0005, dynamically adjusted using a cosine annealing scheduling strategy; the number of training epochs was 200; and the batch size was 128. The learning rate was set to balance training stability and convergence speed, the number of training epochs ensured that the model fully learned the complex patterns of projectile motion, and the batch size was chosen to balance training efficiency and gradient estimation stability.

[0058] The training process employs supervised learning. The sample dataset generated by the digital twin simulation system is randomly divided into training, validation, and test sets in a 7:2:1 ratio. During training, the feature vectors of samples from the training set are input into the agent, with their corresponding real launch trajectory coordinate sequences serving as supervision labels. The network weights are iteratively optimized using backpropagation and the Adam optimizer. A smoothed L1 loss function is used to comprehensively measure the positional deviation between the predicted and real trajectories. The training process is monitored using the validation set. When the validation set loss function value does not decrease for 15 consecutive training epochs, and the average error of the predicted trajectory endpoint positions is less than 10 mm, the model is considered converged, and training is terminated. After training, a converged launch trajectory prediction agent is obtained. This agent can effectively establish a complex mapping relationship from multi-dimensional state parameters to future launch trajectories, achieving high-precision trajectory prediction.

[0059] Finally, if it is determined that the target object is in a throwing state, the spatial coordinates, real-time vibration phase signal, tilt angle parameter, vibration frequency parameter, amplitude parameter, and attribute category of the target object in the object perception information are used together as input feature vectors and input into the trained projectile trajectory prediction agent to output the predicted motion trajectory of the target object in the future time period.

[0060] Furthermore, based on the attribute category of the target foreign object and its corresponding predicted motion trajectory, interference-free sorting trajectories are planned for the first robotic arm and the second robotic arm, and their execution priorities are determined, including: Based on the spatial position of each target foreign object on the vibrating cloth, the direction of movement of the vibrating cloth, and the corresponding predicted motion trajectory, the departure time of each target foreign object from the effective sorting work area of ​​the robotic arm is calculated. Based on the attribute category and corresponding departure time of each target foreign object, the execution priority is determined for the first robotic arm and the second robotic arm. The execution priority is used to specify the execution order of the first robotic arm and the second robotic arm, so that only one robotic arm is allowed to perform the sorting action at any given time. According to the determined execution priority, the first or second robotic arm is driven sequentially. Within the overhead rail space shared by the first and second robotic arms, a sorting trajectory from the current pose to the corresponding target object grasping point is planned for the currently driven robotic arm based on the spatiotemporal planning algorithm.

[0061] First, based on the spatial position of each target foreign object on the vibrating cloth, the inherent direction of motion of the vibrating cloth, and the predicted trajectory of the foreign object obtained in the previous steps, the departure time of each target foreign object from the effective sorting area of ​​the robotic arm is calculated. The effective sorting area refers to the spatial range within which the overhead rail robotic arm can safely and accurately perform grasping actions. The departure time is calculated based on the time point corresponding to the spatial intersection of the predicted trajectory and the effective sorting area.

[0062] Secondly, based on the attribute category of each target foreign object and the calculated departure time, the execution priority of the first and second robotic arms is determined. The first robotic arm, equipped with a magnetic actuator at its end, is specifically responsible for adsorbing and removing metallic foreign objects; the second robotic arm, equipped with a gripper actuator at its end, is specifically responsible for grasping and removing non-metallic foreign objects. Since the two robotic arms are mounted on the same overhead rail, sharing a limited three-dimensional workspace, and the high-speed sorting operation requires extremely high precision and timeliness of the motion trajectory, a clear execution sequence rule needs to be established for the collaborative operation of the two robotic arms to optimize overall sorting efficiency and ensure operational safety.

[0063] Specifically, execution priority is a set of decision rules whose core function is to specify the order in which the first and second robotic arms perform sorting actions. The design of this rule ensures that at any given moment, only one robotic arm is allowed to perform high-speed sorting actions within the shared overhead rail space, thereby fundamentally avoiding the risk of spatial interference or motion conflicts between the two robotic arms.

[0064] Specifically, based on the attribute category and corresponding departure time of each target foreign object, the execution priority of the first and second robotic arms is determined, including: Calculate the number of metallic foreign objects and the number of non-metallic foreign objects among all currently identified but not yet sorted target foreign objects. The first quantity and the second quantity are normalized to obtain the normalized value of the first quantity and the normalized value of the second quantity, respectively. The normalized value of the first quantity is equal to the second quantity divided by the sum of the first quantity and the second quantity, and the normalized value of the second quantity is equal to the first quantity divided by the sum of the first quantity and the second quantity. The first quantity normalized value is used as the basic priority value for the metal foreign object sorting task, and the second quantity normalized value is used as the basic priority value for the non-metal foreign object sorting task. According to the attribute category of the target foreign object, calculate the first average departure time of all metallic foreign objects and the second average departure time of all non-metallic foreign objects; The first average value and the second average value are normalized to obtain the first urgency parameter corresponding to the metal foreign object sorting task and the second urgency parameter corresponding to the non-metal foreign object sorting task. The basic priority value of the metal foreign object sorting task is weighted and fused with the first urgency parameter to obtain the comprehensive priority score of the metal foreign object sorting task. The basic priority value of the non-metal foreign object sorting task is weighted and fused with the second urgency parameter to obtain the comprehensive priority score of the non-metal foreign object sorting task. In the weighted fusion process, the weight of the urgency parameter is greater than the weight of the basic priority value. Compare the overall priority scores of metal foreign object sorting tasks with those of non-metal foreign object sorting tasks. The robotic arm corresponding to the sorting task with the higher overall priority score has priority in execution.

[0065] First, count the number of metallic foreign objects and the number of non-metallic foreign objects among all currently identified but not yet sorted target foreign objects.

[0066] Secondly, the first and second quantities obtained from the statistics are normalized, and normalized values ​​for the first and second quantities are calculated separately. The normalized value for the first quantity is equal to the second quantity divided by the sum of the first and second quantities, and the normalized value for the second quantity is equal to the first quantity divided by the sum of the first and second quantities. This normalization process allows for a higher base priority value for categories of foreign objects with smaller quantities, thus prioritizing the sorting of these types of foreign objects in a strategic manner, thereby balancing the sorting progress of the two types of foreign objects.

[0067] Furthermore, the calculated first normalized value is directly used as the base priority value for the metal foreign object sorting task, and the second normalized value is used as the base priority value for the non-metal foreign object sorting task. The base priority value is a scalar between 0 and 1, used to characterize the static tendency to prioritize sorting one type of foreign object when only considering the relative quantity of the two types of foreign objects.

[0068] Simultaneously, the foreign objects were grouped according to their attribute categories, and a first average departure time for all metallic foreign objects and a second average departure time for all non-metallic foreign objects were calculated. The first and second average values ​​were used to quantitatively assess the overall time urgency faced by the two types of foreign objects; the smaller the value, the faster, on average, the foreign object approached and left the effective working area of ​​the robotic arm.

[0069] Furthermore, the calculated first and second average values ​​are subjected to a similar normalization process. Specifically, the first and second average values ​​are substituted into the same calculation process as the aforementioned quantity normalization. That is, the first urgency parameter corresponding to the metal foreign object sorting task is equal to the second average value divided by the sum of the first and second average values, and the second urgency parameter corresponding to the non-metallic foreign object sorting task is equal to the first average value divided by the sum of the first and second average values. Through this normalization process, a higher urgency parameter can be assigned to foreign object categories with shorter average departure times, i.e., those that leave the effective working area faster, thereby reducing the risk of missing foreign objects due to processing delays.

[0070] Secondly, the basic priority value of the metal foreign object sorting task is weighted and fused with the first urgency parameter to calculate the comprehensive priority score for the metal foreign object sorting task. Similarly, the basic priority value of the non-metallic foreign object sorting task is weighted and fused with the second urgency parameter to calculate the comprehensive priority score for the non-metallic foreign object sorting task. During the weighted fusion process, the weight of the urgency parameter must be greater than the weight of the basic priority value to ensure that the execution priority determination focuses more on the urgency of time, prioritizing the processing of foreign objects that are about to leave, thereby effectively avoiding missing the sorting of quickly leaving foreign objects due to the pursuit of quantity balance. For example, the weight of the urgency parameter is set to 0.7, and the weight of the basic priority value is set to 0.3.

[0071] Finally, the overall priority scores of metal foreign object sorting tasks are compared with those of non-metal foreign object sorting tasks. The robotic arm corresponding to the sorting task with the higher overall priority score is given priority execution.

[0072] Finally, according to the execution priority determined in the above steps, the first or second robotic arm is sequentially driven to prepare for the sorting task. When planning the specific motion path for the robotic arm that currently has the right to execute, a collision-free sorting trajectory is planned for it in the three-dimensional space of the overhead rail shared by the first and second robotic arms, based on a spatiotemporal planning algorithm, from the current pose of the robotic arm to the expected grasping point of the corresponding target foreign object.

[0073] Among them, the spatiotemporal planning algorithm is a path planning method that uses time as an additional dimension to perform unified modeling and searching in three-dimensional space. This spatiotemporal planning algorithm can simultaneously consider spatial obstacles and temporal occupancy, ensuring that the planned trajectory not only avoids static obstacles and the predetermined position of another robotic arm in the spatial path, but also avoids conflicts with the movement process of another robotic arm in the temporal sequence, thereby achieving truly interference-free collaborative operation.

[0074] S40: Based on the execution priority and sorting trajectory, control the first robotic arm to attract and remove metallic foreign objects through the magnetic actuator at its end, and control the second robotic arm to grab and remove non-metallic foreign objects through the gripper actuator at its end.

[0075] Finally, based on the execution priority and sorting trajectory, the first and second robotic arms are controlled. Specifically, for the first robotic arm, when it gains execution authority, the control system drives it to move along the planned sorting trajectory, ensuring that the end effector's magnetic actuator accurately reaches the predicted gripping point of the target metal object. At the appropriate time, the magnetic actuator is activated to generate a strong magnetic field to firmly attract the metal object. Subsequently, the robotic arm is controlled to transport the object along the planned removal path to the designated waste or collection area. Finally, the magnetic field is deactivated to release the object, completing one metal object sorting task.

[0076] For the second robotic arm, when it gains execution authority, the control system drives it to move along its own sorting trajectory, so that the end effector gripper reaches the predicted gripping point of the target non-metallic foreign object. The control system then controls the gripper to perform the gripping action, ensuring reliable gripping of the non-metallic foreign object through force feedback or position closed loop. Subsequently, the control system moves the foreign object to the corresponding collection area and releases the gripper, completing one non-metallic foreign object sorting task.

[0077] In summary, the entire process follows a predetermined execution priority sequence and sorting trajectory, ensuring that at any given time only one robotic arm performs high-speed sorting actions within the shared overhead rail space, thereby achieving safe, efficient, and orderly collaborative operations until all identified target foreign objects are removed.

[0078] In summary, the embodiments of this application have at least the following technical effects: Compared to existing technologies, this invention first achieves comprehensive and reliable identification of both metallic and non-metallic foreign objects through the synchronous acquisition and data fusion of electromagnetic and visual sensors, overcoming the limitations of single-sensor technologies. Secondly, this invention combines the dynamic parameters of the vibrating fabric distributor to perform high-precision prediction of the foreign object's trajectory, providing accurate spatiotemporal basis for dynamic grasping. Thirdly, based on the foreign object's properties and predicted trajectory, this invention intelligently allocates task priorities to the dual robotic arms and plans interference-free motion paths, effectively avoiding robotic arm conflicts and improving the efficiency and safety of collaborative operations.

[0079] Finally, through differentiated configuration and coordinated control of the end effector, this invention achieves synchronous, efficient, and automated sorting of two types of foreign objects, thereby achieving multi-objective dynamic optimization and comprehensive performance improvement in sorting efficiency, operational safety, and system energy consumption under complex vibration conditions.

[0080] Example 2, as Figure 2 , Figure 3 As shown, based on the same inventive concept as the foreign object sorting collaborative control method based on a rail-mounted dual-arm robotic arm provided in Embodiment 1, this embodiment of the invention also provides a foreign object sorting collaborative control system based on a rail-mounted dual-arm robotic arm, including: The sensor acquisition module 11 is used to synchronously acquire electromagnetic induction signals and material flow surface images of the material flow through an electromagnetic sensor arranged above the vibrating cloth and a high-speed vision sensor. The information fusion and recognition module 12 is used to fuse the electromagnetic induction signal and the surface image of the material flow, identify the target foreign objects in the material flow, and generate foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object, wherein the attribute category includes metallic foreign objects or non-metallic foreign objects. The trajectory prediction and planning module 13 is used to predict the predicted motion trajectory of each target foreign object in the future time period based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, and to plan non-interference sorting trajectories for the first robotic arm and the second robotic arm and determine the execution priority based on the attribute category of the target foreign object and the corresponding predicted motion trajectory. The sorting execution control module 14 is used to control the first robotic arm to attract and remove metallic foreign objects through the magnetic actuator at its end, and to control the second robotic arm to grab and remove non-metallic foreign objects through the gripper actuator at its end, according to the execution priority and sorting trajectory.

[0081] Specifically, the sensor acquisition module 11 is used for: Specifically, electromagnetic induction signals and surface images of the material flow are simultaneously acquired by an electromagnetic sensor and a high-speed vision sensor arranged above the vibrating material distributor, including: Based on the ambient lighting conditions above the vibrating fabric, the gain parameters, exposure time parameters, and supplementary light intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions. Under the image acquisition parameters, the high-speed vision sensor is controlled to acquire images of the material flow surface. The electromagnetic sensor is synchronously controlled to collect electromagnetic induction signals of the material flow, and the electromagnetic induction signals are aligned with the timestamp of the material flow surface image.

[0082] Specifically, based on the ambient lighting conditions above the vibrating fabric distribution device, the gain parameters, exposure time parameters, and supplementary lighting intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions, including: Obtain the average brightness and average contrast values ​​of the monitoring area above the vibrating fabric distributor; Calculate the brightness deviation between the average brightness value and the preset standard brightness value, and calculate the contrast deviation between the average contrast value and the preset standard contrast value; Based on the brightness deviation, the gain parameter and exposure time parameter of the high-speed vision sensor are adjusted, wherein the adjustment direction of the gain parameter and exposure time parameter is opposite to the direction of the brightness deviation; Based on the contrast deviation, the fill light intensity of the fill light is adjusted, wherein the direction of adjustment of the fill light intensity is opposite to the direction of the contrast deviation.

[0083] The information fusion and recognition module 12 is specifically used for: Specifically, the electromagnetic induction signal and the surface image of the material flow are fused to identify target foreign objects in the material flow, and foreign object sensing information containing the attribute category, spatial location, and timestamp of each target foreign object is generated, including: Based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis. The electromagnetic induction signal is analyzed synchronously to obtain the electromagnetic response intensity of each material to be identified in space. The estimated three-dimensional volume, surface visual texture features, and electromagnetic response intensity of each material to be identified are input into a pre-trained multi-level classification model, wherein the multi-level classification model includes a first-level classification module and a second-level classification module connected in series. The first-level classification module classifies each material to be identified into normal material or target foreign object based on the estimated three-dimensional volume and surface visual texture features. The second-level classification module combines the electromagnetic response intensity of the target foreign object to classify it into a metallic or non-metallic foreign object and determines its corresponding spatial coordinates. Based on the classification results and spatial coordinates output by the second-level classification module, foreign object perception information is generated, which includes the attribute category, spatial location and collection timestamp of each target foreign object.

[0084] Specifically, based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis, including: Based on the material flow surface image, the three-dimensional point cloud data of the material flow surface is reconstructed using the grating deformation analysis method. The three-dimensional point cloud data is segmented, and based on the spatial aggregation characteristics of the three-dimensional point cloud data, a subset of the point cloud of each material to be identified in the material flow is separated. Based on the segmented point cloud subsets, calculate the estimated three-dimensional volume of each material to be identified; For each material to be identified, a two-dimensional image region matching the projection area of ​​the material to be identified is extracted from the surface image of the material flow; Using a convolutional neural network, surface visual texture features of each of the two-dimensional image regions are extracted, wherein the surface visual texture features include roughness, contrast and directionality values.

[0085] Specifically, the trajectory prediction and planning module 13 is used for: Specifically, based on the foreign object sensing information and the vibration motion parameters of the vibrating fabric, the predicted motion trajectory of each target foreign object in the future time period is predicted, including: The system acquires the real-time vibration phase signal of the vibratory fabric feeder drive motor, as well as the tilt angle, vibration frequency, and amplitude parameters of the vibratory fabric feeder bed. Based on the vibration frequency and amplitude parameters, the throwing index under the current vibration conditions is calculated. Based on the throwing index and the friction coefficient between the material and the vibrating cloth bed, the critical throwing phase angle at which the material begins to throw is determined. The current phase angle of the real-time vibration phase signal is compared with the critical throwing phase angle. If the current phase angle is less than the critical throwing phase angle, it is determined that the target foreign object is in a sliding state in contact with the bed surface. If the current phase angle is greater than or equal to the critical throwing phase angle, it is determined that the target foreign object is in a throwing state being thrown by the bed surface. If it is determined that the target foreign object is in a sliding state, then based on the Coulomb friction dynamics equation, according to the spatial position coordinates, real-time vibration phase signal, vibration frequency parameter, amplitude parameter and attribute category of the target foreign object in the foreign object sensing information, the predicted motion trajectory of the target foreign object is calculated in real time. If the target object is determined to be in a throwing state, the spatial coordinates, real-time vibration phase signal, tilt angle parameter, vibration frequency parameter, amplitude parameter, and target object attribute category from the object perception information are input into the pre-trained projectile trajectory prediction agent, and the predicted motion trajectory of the target object in the future time period is output.

[0086] Specifically, the training process of the projectile trajectory prediction agent includes: Constructing an intelligent agent for predicting projectile trajectories based on machine learning; Based on the physical parameters of the vibrating fabric feeder and typical foreign objects, a parameterized digital twin simulation system is constructed. In the digital twin simulation system, by randomly setting the spatial coordinates of the sample, the vibration motion parameters of the sample, the properties of the foreign object and the initial state, a high-fidelity physics simulation is run to generate corresponding real projectile trajectories in batches, thus forming a sample dataset. The projectile trajectory prediction agent is trained using the sample dataset to obtain a trained projectile trajectory prediction agent.

[0087] Furthermore, based on the attribute category of the target foreign object and its corresponding predicted motion trajectory, interference-free sorting trajectories are planned for the first robotic arm and the second robotic arm, and their execution priorities are determined, including: Based on the spatial position of each target foreign object on the vibrating cloth, the direction of movement of the vibrating cloth, and the corresponding predicted motion trajectory, the departure time of each target foreign object from the effective sorting work area of ​​the robotic arm is calculated. Based on the attribute category and corresponding departure time of each target foreign object, the execution priority is determined for the first robotic arm and the second robotic arm. The execution priority is used to specify the execution order of the first robotic arm and the second robotic arm, so that only one robotic arm is allowed to perform the sorting action at any given time. According to the determined execution priority, the first or second robotic arm is driven sequentially. Within the overhead rail space shared by the first and second robotic arms, a sorting trajectory from the current pose to the corresponding target object grasping point is planned for the currently driven robotic arm based on the spatiotemporal planning algorithm.

[0088] Specifically, based on the attribute category and corresponding departure time of each target foreign object, the execution priority of the first and second robotic arms is determined, including: Calculate the number of metallic foreign objects and the number of non-metallic foreign objects among all currently identified but not yet sorted target foreign objects. The first quantity and the second quantity are normalized to obtain the normalized value of the first quantity and the normalized value of the second quantity, respectively. The normalized value of the first quantity is equal to the second quantity divided by the sum of the first quantity and the second quantity, and the normalized value of the second quantity is equal to the first quantity divided by the sum of the first quantity and the second quantity. The first quantity normalized value is used as the basic priority value for the metal foreign object sorting task, and the second quantity normalized value is used as the basic priority value for the non-metal foreign object sorting task. According to the attribute category of the target foreign object, calculate the first average departure time of all metallic foreign objects and the second average departure time of all non-metallic foreign objects; The first average value and the second average value are normalized to obtain the first urgency parameter corresponding to the metal foreign object sorting task and the second urgency parameter corresponding to the non-metal foreign object sorting task. The basic priority value of the metal foreign object sorting task is weighted and fused with the first urgency parameter to obtain the comprehensive priority score of the metal foreign object sorting task. The basic priority value of the non-metal foreign object sorting task is weighted and fused with the second urgency parameter to obtain the comprehensive priority score of the non-metal foreign object sorting task. In the weighted fusion process, the weight of the urgency parameter is greater than the weight of the basic priority value. Compare the overall priority scores of metal foreign object sorting tasks with those of non-metal foreign object sorting tasks. The robotic arm corresponding to the sorting task with the higher overall priority score has priority in execution.

[0089] The sorting execution control module 14 is specifically used for: Based on the execution priority and sorting trajectory, the first robotic arm is controlled to attract and remove metallic foreign objects through the magnetic actuator at its end, and the second robotic arm is controlled to grab and remove non-metallic foreign objects through the gripper actuator at its end.

Claims

1. A collaborative control method for foreign object sorting based on a suspended dual-arm robotic arm, characterized in that, The method includes: Electromagnetic sensors and high-speed vision sensors arranged above the vibrating cloth synchronously acquire electromagnetic induction signals and material flow surface images of the material flow. The electromagnetic induction signal and the surface image of the material flow are fused together to identify the target foreign objects in the material flow and generate foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object. The attribute category includes metallic foreign objects or non-metallic foreign objects. Based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, the predicted motion trajectory of each target foreign object in the future time period is predicted. Based on the attribute category of the target foreign object and the corresponding predicted motion trajectory, the first robotic arm and the second robotic arm are planned to have non-interference sorting trajectories and the execution priority is determined. Based on the execution priority and sorting trajectory, the first robotic arm is controlled to attract and remove metallic foreign objects through the magnetic actuator at its end, and the second robotic arm is controlled to grab and remove non-metallic foreign objects through the gripper actuator at its end. Specifically, based on the foreign object sensing information and the vibration motion parameters of the vibrating fabric, the predicted motion trajectory of each target foreign object in a future time period is predicted, including: The system acquires the real-time vibration phase signal of the vibratory fabric feeder drive motor, as well as the tilt angle, vibration frequency, and amplitude parameters of the vibratory fabric feeder bed. Based on the vibration frequency and amplitude parameters, the throwing index under the current vibration conditions is calculated. Based on the throwing index and the friction coefficient between the material and the vibrating cloth bed, the critical throwing phase angle at which the material begins to throw is determined. The current phase angle of the real-time vibration phase signal is compared with the critical throwing phase angle. If the current phase angle is less than the critical throwing phase angle, it is determined that the target foreign object is in a sliding state in contact with the bed surface. If the current phase angle is greater than or equal to the critical throwing phase angle, it is determined that the target foreign object is in a throwing state being thrown by the bed surface. If it is determined that the target foreign object is in a sliding state, then based on the Coulomb friction dynamics equation, according to the spatial position coordinates, real-time vibration phase signal, vibration frequency parameter, amplitude parameter and attribute category of the target foreign object in the foreign object sensing information, the predicted motion trajectory of the target foreign object is calculated in real time. If the target object is determined to be in a throwing state, the spatial coordinates, real-time vibration phase signal, tilt angle parameter, vibration frequency parameter, amplitude parameter, and target object attribute category from the object perception information are input into the pre-trained projectile trajectory prediction agent, and the predicted motion trajectory of the target object in the future time period is output.

2. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 1, characterized in that, Electromagnetic sensors and high-speed vision sensors, arranged above the vibrating material distributor, synchronously acquire electromagnetic induction signals and surface images of the material flow, including: Based on the ambient lighting conditions above the vibrating fabric, the gain parameters, exposure time parameters, and supplementary light intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions. Under the image acquisition parameters, the high-speed vision sensor is controlled to acquire images of the material flow surface. The electromagnetic sensor is synchronously controlled to collect electromagnetic induction signals of the material flow, and the electromagnetic induction signals are aligned with the timestamp of the material flow surface image.

3. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 2, characterized in that, Based on the ambient lighting conditions above the vibrating fabric fabricator, the gain parameters, exposure time parameters, and supplementary lighting intensity of the high-speed vision sensor are dynamically adjusted to obtain image acquisition parameters adapted to the current ambient lighting conditions, including: Obtain the average brightness and average contrast values ​​of the monitoring area above the vibrating fabric distributor; Calculate the brightness deviation between the average brightness value and the preset standard brightness value, and calculate the contrast deviation between the average contrast value and the preset standard contrast value; Based on the brightness deviation, the gain parameter and exposure time parameter of the high-speed vision sensor are adjusted, wherein the adjustment direction of the gain parameter and exposure time parameter is opposite to the direction of the brightness deviation; Based on the contrast deviation, the fill light intensity of the fill light is adjusted, wherein the direction of adjustment of the fill light intensity is opposite to the direction of the contrast deviation.

4. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 1, characterized in that, The electromagnetic induction signal and the surface image of the material flow are fused together to identify target foreign objects in the material flow, and foreign object sensing information including the attribute category, spatial location, and timestamp of each target foreign object is generated, including: Based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis. The electromagnetic induction signal is analyzed synchronously to obtain the electromagnetic response intensity of each material to be identified in space. The estimated three-dimensional volume, surface visual texture features, and electromagnetic response intensity of each material to be identified are input into a pre-trained multi-level classification model, wherein the multi-level classification model includes a first-level classification module and a second-level classification module connected in series. The first-level classification module classifies each material to be identified into normal material or target foreign object based on the estimated three-dimensional volume and surface visual texture features. The second-level classification module combines the electromagnetic response intensity of the target foreign object to classify it into a metallic or non-metallic foreign object and determines its corresponding spatial coordinates. Based on the classification results and spatial coordinates output by the second-level classification module, foreign object perception information is generated, which includes the attribute category, spatial location and collection timestamp of each target foreign object.

5. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 4, characterized in that, Based on the surface image of the material flow, the three-dimensional volume estimate and surface visual texture features of each material to be identified in the material flow are obtained through visual analysis, including: Based on the material flow surface image, the three-dimensional point cloud data of the material flow surface is reconstructed using the grating deformation analysis method. The three-dimensional point cloud data is segmented, and based on the spatial aggregation characteristics of the three-dimensional point cloud data, a subset of the point cloud of each material to be identified in the material flow is separated. Based on the segmented point cloud subsets, calculate the estimated three-dimensional volume of each material to be identified; For each material to be identified, a two-dimensional image region matching the projection area of ​​the material to be identified is extracted from the surface image of the material flow; Using a convolutional neural network, surface visual texture features of each of the two-dimensional image regions are extracted, wherein the surface visual texture features include roughness, contrast and directionality values.

6. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 1, characterized in that, The training process of the projectile trajectory prediction agent includes: Constructing an intelligent agent for predicting projectile trajectories based on machine learning; Based on the physical parameters of the vibrating fabric feeder and typical foreign objects, a parameterized digital twin simulation system is constructed. In the digital twin simulation system, by randomly setting the spatial coordinates of the sample, the vibration motion parameters of the sample, the properties of the foreign object and the initial state, a high-fidelity physics simulation is run to generate corresponding real projectile trajectories in batches, thus forming a sample dataset. The projectile trajectory prediction agent is trained using the sample dataset to obtain a trained projectile trajectory prediction agent.

7. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 1, characterized in that, Based on the attribute category of the target foreign object and its corresponding predicted motion trajectory, interference-free sorting trajectories are planned for the first robotic arm and the second robotic arm, and their execution priorities are determined, including: Based on the spatial position of each target foreign object on the vibrating cloth, the direction of movement of the vibrating cloth, and the corresponding predicted motion trajectory, the departure time of each target foreign object from the effective sorting work area of ​​the robotic arm is calculated. Based on the attribute category and corresponding departure time of each target foreign object, the execution priority is determined for the first robotic arm and the second robotic arm. The execution priority is used to specify the execution order of the first robotic arm and the second robotic arm, so that only one robotic arm is allowed to perform the sorting action at any given time. According to the determined execution priority, the first or second robotic arm is driven sequentially. Within the overhead rail space shared by the first and second robotic arms, a sorting trajectory from the current pose to the corresponding target object grasping point is planned for the currently driven robotic arm based on the spatiotemporal planning algorithm.

8. The collaborative control method for foreign object sorting based on a rail-mounted dual-arm robotic arm according to claim 7, characterized in that, Based on the attribute category and corresponding departure time of each target foreign object, the execution priority is determined for the first and second robotic arms, including: Calculate the number of metallic foreign objects and the number of non-metallic foreign objects among all currently identified but not yet sorted target foreign objects. The first quantity and the second quantity are normalized to obtain the normalized value of the first quantity and the normalized value of the second quantity, respectively. The normalized value of the first quantity is equal to the second quantity divided by the sum of the first quantity and the second quantity, and the normalized value of the second quantity is equal to the first quantity divided by the sum of the first quantity and the second quantity. The first quantity normalized value is used as the basic priority value for the metal foreign object sorting task, and the second quantity normalized value is used as the basic priority value for the non-metal foreign object sorting task. According to the attribute category of the target foreign object, calculate the first average departure time of all metallic foreign objects and the second average departure time of all non-metallic foreign objects; The first average value and the second average value are normalized to obtain the first urgency parameter corresponding to the metal foreign object sorting task and the second urgency parameter corresponding to the non-metal foreign object sorting task. The basic priority value of the metal foreign object sorting task is weighted and fused with the first urgency parameter to obtain the comprehensive priority score of the metal foreign object sorting task. The basic priority value of the non-metal foreign object sorting task is weighted and fused with the second urgency parameter to obtain the comprehensive priority score of the non-metal foreign object sorting task. In the weighted fusion process, the weight of the urgency parameter is greater than the weight of the basic priority value. Compare the overall priority scores of metal foreign object sorting tasks with those of non-metal foreign object sorting tasks. The robotic arm corresponding to the sorting task with the higher overall priority score has priority in execution.

9. A collaborative control system for foreign object sorting based on a rail-mounted dual-arm robotic arm, characterized in that, The method for collaborative control of foreign object sorting based on a rail-mounted dual-arm robotic arm as described in any one of claims 1-8 includes: The sensor acquisition module is used to synchronously acquire electromagnetic induction signals and material flow surface images of the material flow through an electromagnetic sensor arranged above the vibrating cloth and a high-speed vision sensor. The information fusion and recognition module is used to fuse the electromagnetic induction signal and the surface image of the material flow, identify the target foreign objects in the material flow, and generate foreign object perception information containing the attribute category, spatial location and timestamp of each target foreign object, wherein the attribute category includes metallic foreign objects or non-metallic foreign objects. The trajectory prediction and planning module is used to predict the predicted motion trajectory of each target foreign object in the future time period based on the foreign object sensing information and the vibration motion parameters of the vibrating cloth, and to plan non-interference sorting trajectories for the first robotic arm and the second robotic arm and determine the execution priority based on the attribute category of the target foreign object and the corresponding predicted motion trajectory. The sorting execution control module is used to control the first robotic arm to attract and remove metallic foreign objects through the magnetic actuator at its end, and to control the second robotic arm to grab and remove non-metallic foreign objects through the gripper actuator at its end, according to the execution priority and sorting trajectory.

Images