Non-standard special-shaped workpiece sorting method and device based on cross-view joint understanding

Abstract

This invention provides a method and apparatus for sorting non-standard irregularly shaped workpieces based on cross-view joint understanding. The method includes: inputting an image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by the reclamation of recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operating table and the recycling bin in the input image sequence based on an attention mechanism, and generates a sorting action prediction sequence in combination with the task instructions; according to the action prediction sequence, the corresponding action is executed using Cartesian space path planning. This invention utilizes the intelligent understanding and generalization capabilities of an end-to-end model for complex scenarios to accurately identify the targets to be sorted and their corresponding destinations in a cluttered operating table, achieving a leap from perception to cognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for sorting non-standard irregular workpieces based on cross-view joint understanding. Background Technology

[0002] With the deep integration of artificial intelligence and robotics, embodied intelligence is demonstrating enormous application potential in industrial automation, particularly in the automated dismantling, sorting, and recycling of electronic waste (E-Waste). E-Waste involves handling workpieces with highly uncertain shapes, such as discarded motherboards, tangled cables, and damaged casings. This not only requires robots with high-precision motion control capabilities but also relies heavily on vision systems for accurate perception and understanding of complex environments. In recent years, visual language models, with their powerful multimodal understanding and generalization capabilities, have been gradually introduced into the field of robot task planning. This aims to address the lack of flexibility in traditional automated production lines through end-to-end learning, driving the resource recycling industry towards intelligent and flexible transformation.

[0003] Currently, electronic waste processing often adopts a cascaded vision system architecture. This architecture typically breaks down complex dismantling and sorting tasks into independent sub-task modules. Specifically, a dedicated target detection model (such as the YOLO series) is used to identify the specific electronic components or irregularly shaped workpieces on each workbench in the workbench area, and another set of vision algorithms is used to perform overflow detection and status analysis on the recycling bin images in the classified storage area. Then, the output results of the above two steps are correlated through rule-based hard-coded logic to drive the robotic arm to perform the corresponding grasping and delivery actions.

[0004] However, cascaded architectures separate recognition from planning, making it difficult for models to understand fuzzy instructions based on global semantics, lacking end-to-end joint reasoning capabilities, and struggling to adapt to complex and ever-changing business logic. Furthermore, dedicated template detection models heavily rely on large amounts of labeled data, resulting in extremely poor generalization ability for irregularly shaped workpieces with diverse forms, such as twisted copper wires or broken cables. Additionally, image augmentation techniques such as random cropping and flipping easily destroy crucial subtle features in images or sever logical dependencies across scenes, leading to scarce and low-quality training data. Summary of the Invention

[0005] This invention provides a sorting method and apparatus for non-standard irregular workpieces based on cross-view joint understanding, which solves the defects of existing technologies that lack end-to-end joint reasoning ability due to the separation of recognition and planning. By utilizing the intelligent understanding and generalization ability of end-to-end models for complex scenarios, it can accurately identify the target to be sorted and its corresponding destination in a cluttered operating table, and realize the leap from perception to cognition.

[0006] This invention provides a sorting method for non-standard irregular workpieces based on cross-view joint understanding, comprising: acquiring an image sequence containing multiple operation table images and recycling bin images; inputting the image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operation table and the recycling bin in the input image sequence based on an attention mechanism, and to generate a sorting action prediction sequence in combination with the task instructions; according to the action prediction sequence, using Cartesian space path planning, executing the corresponding actions to sort the workpieces to be sorted on the corresponding operation table to the corresponding recycling bin.

[0007] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding is provided. The method inputs image sequences and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the model. The method includes: inputting image sequences and task instructions into the sorting action prediction model to perform visual encoding on the input image sequences to obtain image sequence encoding, and performing text lexicalization on the input task instructions to obtain an instruction text lexical sequence; based on the image sequence encoding and the instruction text lexical sequence, using a cross-attention mechanism to obtain a multimodal joint feature sequence, and progressively predicting and outputting a sorting action prediction sequence containing grasping and placement poses through autoregressive decoding.

[0008] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding includes the following steps before inputting image sequences and task instructions into a sorting action prediction model: semantic atomization decomposition of sorting demonstration data using a large model to obtain fine-grained atomic facts; atomic facts are used to characterize object attributes, states, and coordinate information; based on the dismantling station atomic facts and recycling bin atomic facts, the data is recombined according to recycling business logic to generate synthetic task samples, and combined with the sorting demonstration data to obtain training samples; the training samples include image input, instruction text, and action sequences; the image input and instruction text are used as input data for training, and the action sequences are used as labels for training to train the sorting action prediction model to obtain a sorting action prediction model for predicting sorting actions.

[0009] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding is provided. The method trains a sorting action prediction model, including: for each iteration step, inputting an image and instruction text into the sorting action prediction model to obtain a training action sequence output by the model; when the image input length is less than a preset length threshold, constructing a total loss function based on the training action sequence and the corresponding action sequence; when the image input length is greater than or equal to the preset length threshold, constructing a total loss function based on the training action sequence and the corresponding action sequence, combined with a global contrastive loss constraint; updating the model parameters based on the total loss function and iterating again until the total loss function converges or the maximum number of iterations is reached, thus ending the training.

[0010] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding is provided. The method constructs a total loss function, including: constructing a cross-entropy loss function based on the training action sequence and the corresponding action sequence; constructing an indicator function based on whether the image input length is greater than a preset length threshold; constructing a global contrastive loss function based on the instruction intent extracted from the image input and instruction text input to the sorting action prediction model to be trained, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input; and obtaining the total loss function by combining the global contrastive loss function and the indicator function with the cross-entropy loss function.

[0011] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding is provided. The method constructs a global contrastive loss function, comprising: extracting the instruction intent vector, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input from the image input and instruction text input to the sorting action prediction model to be trained; performing gradient blocking on the local vector to obtain the blocked local vector; determining the first similarity between the blocked local vector and the instruction intent vector, and determining the second similarity between the global vector and the instruction intent vector; and obtaining the global contrastive loss function based on the first and second similarities, combined with a preset boundary parameter threshold.

[0012] According to the present invention, a sorting method for non-standard irregular-shaped workpieces based on cross-view joint understanding is provided. The sorting demonstration data includes a sequence of operation table images and a sequence of recycling bin images. Based on the atomic facts of the dismantling station and the recycling bin in the atomic facts, the method recombines them according to the recycling business logic to generate a synthetic task sample. This includes: constructing a program generation engine and embedding the recycling business logic; the recycling business logic includes material classification rules, hazardous waste priority treatment logic, and / or load balancing logic; calling the atomic facts of the dismantling station and the recycling bin in the atomic facts through the program generation engine, and recombining them according to the recycling business logic based on the attributes of the workpiece to be sorted on the dismantling station and the container status of the recycling bin, generating corresponding instruction text and a synthetic action sequence; using the operation table image and recycling bin image corresponding to the instruction text as image input, and combining the corresponding instruction text and the synthetic action sequence, a synthetic task sample is obtained.

[0013] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding includes, after obtaining a synthetic task sample, performing data augmentation processing on the image input in the synthetic task sample; the data augmentation processing includes random shuffling of image order and / or random injection of interference images; wherein, random shuffling of image order is used to break the inherent positional arrangement of the input images; random injection of interference images is used to randomly insert interference images unrelated to the current task into the input image sequence.

[0014] According to the present invention, a sorting method for non-standard irregular workpieces based on cross-view joint understanding is provided. Based on the action prediction sequence, Cartesian space path planning is used to execute corresponding actions, including: parsing the action prediction sequence to obtain movement instructions, grasping instructions, and placement instructions; extracting the workpiece pixel coordinates and the recycling bin pixel coordinates; mapping the workpiece pixel coordinates and recycling bin pixel coordinates to physical space three-dimensional coordinates in the corresponding robotic arm base coordinate system according to the camera intrinsic parameter matrix and the pre-calibrated hand-eye calibration matrix; performing Cartesian space path planning based on the corresponding physical space coordinates to generate a collision-free motion trajectory, and controlling the robotic arm to perform corresponding grasping and placement actions according to the motion trajectory.

[0015] This invention also provides a sorting device for non-standard irregular-shaped workpieces based on cross-view joint understanding, comprising: an image acquisition module for acquiring an image sequence containing multiple operation table images and recycling bin images; an action prediction module for inputting the image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using a sorting demonstration dataset and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration dataset, and a synthetic task sample generated by the recycling business logic constraint reorganization; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operation table and the recycling bin in the input image sequence based on an attention mechanism, and to generate a sorting action prediction sequence in combination with the task instructions; and a workpiece sorting module, according to the action prediction sequence, using Cartesian space path planning to execute the corresponding actions to sort the workpieces to be sorted on the corresponding operation table to the corresponding recycling bin.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sorting method for non-standard irregular workpieces based on cross-view joint understanding as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sorting method for non-standard irregular workpieces based on cross-view joint understanding as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sorting method for non-standard irregular workpieces based on cross-view joint understanding as described above.

[0019] The present invention provides a sorting method and apparatus for non-standard irregular workpieces based on cross-view joint understanding. By decomposing complex demonstration data into fine-grained atomic facts and then recombining them according to the recycling business logic, a massive number of logically reasonable synthetic training samples are derived from limited real data. This reduces the dependence on manual demonstration data and ensures that the training data covers various possible sorting combinations and edge cases, significantly improving the adaptability and robustness of the model when facing unknown sorting tasks.

[0020] Furthermore, by inputting image sequences and task instructions into the sorting action prediction model, an attention mechanism is used to guide the model to automatically filter out background noise and irrelevant interference, accurately focus on the decisive features of the workpiece, establish a deep semantic mapping between it and the text task instructions, and perform semantic confirmation in conjunction with the task instructions. This implicitly integrates the prediction of the feasibility of subsequent planning, ensuring that the model can accurately locate the target to be sorted in chaotic scenes, especially irregularly shaped items, and visually lock onto the parts suitable for grasping. This provides a precise perceptual basis for the generation of subsequent actions, realizing the leap from perception to cognition. The action prediction sequence is used as a macro guide, combined with Cartesian space path planning for specific execution, forming a hierarchical architecture of decision-making and execution. This retains the end-to-end model's intelligent understanding and generalization ability in complex scenes, while introducing the stability and safety guarantees of classic planning algorithms. This effectively avoids obstacles and ensures the smoothness and accuracy of the motion trajectory, fundamentally solving the problem of the disconnect between perception and execution, and ensuring the intelligence of the sorting action and the stability of the physical execution process. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the sorting method for non-standard irregular workpieces based on cross-view joint understanding provided by the present invention. Figure 2 This is a schematic diagram of the structure of the non-standard irregular workpiece sorting device based on cross-view joint understanding provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 This is a flowchart illustrating the non-standard irregular workpiece sorting method based on cross-view joint understanding provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain an image sequence containing multiple workbench images and recycle bin images; Step 102: Input the image sequence and task instructions into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of sorting demonstration data, and synthesized task samples generated by the recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpiece to be sorted on the operating table and the recycling bin in the input image sequence based on the attention mechanism, and generate the sorting action prediction sequence in combination with the task instructions; Step 103: Based on the action prediction sequence, Cartesian space path planning is used to execute the corresponding actions to sort the workpieces to be sorted on the corresponding operating table into the corresponding recycling bins.

[0025] It should be noted that the following describes in detail the sorting method for non-standard irregular workpieces based on cross-view joint understanding of the present invention.

[0026] Step 101: Obtain an image sequence containing multiple workbench images and recycle bin images.

[0027] It should be noted that the acquired image sequence can be obtained by capturing single-frame images of multiple operating tables and recycling bins using a camera; or by capturing video streams of the operating tables and recycling bins using a camera and acquiring corresponding images of the operating tables and recycling bins from the video stream; or by acquiring three-dimensional point cloud data of the operating tables and recycling bins using a high-precision 3D camera, and processing the three-dimensional point cloud data with reconstruction and geometric segmentation algorithms to identify the three-dimensional contours of the operating tables and recycling bins, so as to generate or extract corresponding images based on the three-dimensional contours, thereby obtaining the image sequence.

[0028] In addition, the shooting equipment can be selected according to the actual shooting accuracy requirements, such as a high-resolution camera, without further restrictions here.

[0029] In an optional embodiment, before inputting the image sequence and task instructions into the sorting action prediction model, the method includes: using a perceiver resampler or a query transformer Q-Former to fuse multiple images in the image sequence into a feature map, and inputting the feature map into the sorting action prediction model for sorting action prediction. The specific prediction principle is described below and will not be repeated here.

[0030] Step 102: Input the image sequence and task instructions into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained by semantic atomization decomposition of sorting demonstration data, and synthetic task samples generated by recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpiece to be sorted on the operating table and the recycling bin in the input image sequence based on the attention mechanism, and generate the sorting action prediction sequence in combination with the task instructions.

[0031] In this embodiment, the image sequence and task instructions are input into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model. This includes: inputting the image sequence and task instructions into the sorting action prediction model to perform visual encoding on the input image sequence to obtain image sequence encoding, and performing text lexicalization on the input task instructions to obtain instruction text lexical sequence; based on the image sequence encoding and instruction text lexical sequence, using a cross-attention mechanism to obtain a multimodal joint feature sequence, and using an autoregressive decoding method to progressively predict and output the sorting action prediction sequence containing the grasping pose and the placement pose.

[0032] It should be noted that the model visually encodes the input image sequence and textualizes the task instructions to provide a computable and unified representation for subsequent cross-modal fusion. This ensures, from the data level, the model's comprehensive and accurate perception of the visual state of non-standard, irregularly shaped workpieces and the intent of sorting instructions, reducing information omissions or semantic distortion. Furthermore, through a cross-attention mechanism, the image sequence encoding and the instruction text lexical sequence are fused into a multimodal joint feature sequence, thereby defining precise visual and linguistic boundaries for action prediction. This ensures that the generated action logic conforms to the instructions and scene constraints, reducing false positives and violations. Using an autoregressive decoding approach, the model's dynamic generation capability flexibly adapts to complex scenes and progressively predicts sorting action sequences including grasping and placement poses. The multimodal joint features are transformed into mandatory guidance for the generation probability at each step, ensuring that the generated pose parameters strictly adhere to spatial and physical constraints, reducing illegal collisions and unreachable actions, and guaranteeing the legality and executability of the sorting action sequence in continuous space.

[0033] It should be added that the sorting action prediction model can adopt a visual language model (VLM) fine-tuned with global contrast loss (GCR Loss), such as the Tongyi Qianwen visual language model Qwen-VL or an improved version of the large language and visual assistant model LLaVA. For details, please refer to the following text, which will not be repeated here.

[0034] In addition, by using the trained sorting action prediction model as a high-level planner to handle multiple visual inputs that are physically isolated in space but logically interdependent, and through the attention mechanism, the workpieces to be sorted on the disassembly table and the storage labels on the recycling bin are semantically aligned across image boundaries. This ensures that the system can perform cross-scene reasoning and output precise operation instructions based on fuzzy natural language instructions (such as "find the most suitable place to put this broken part") in workpieces with complex backgrounds and uncertain shapes.

[0035] In an optional embodiment, before inputting the image sequence and task instructions into the sorting action prediction model, the method includes: semantically atomizing the sorting demonstration data using a large model to obtain fine-grained atomic facts; the atomic facts are used to characterize object attributes, states, and coordinate information; based on the dismantling station atomic facts and recycling bin atomic facts in the atomic facts, the data is recombined according to recycling business logic to generate synthetic task samples, and combined with the sorting demonstration data to obtain training samples; the training samples include image input, instruction text, and action sequences; the image input and instruction text are used as input data for training, and the action sequences are used as labels for training to train the sorting action prediction model to obtain a sorting action prediction model for predicting sorting actions.

[0036] It should be noted that by leveraging the semantic understanding capabilities of the large model, complex sorting demonstration data is deconstructed into standardized atomic facts such as object attributes, states, and coordinates. This process removes redundant actions and irrelevant backgrounds from the demonstration, accurately retaining the core logical units of task execution. Furthermore, the atomic facts are recombined based on the recycling business logic, breaking the spatiotemporal limitations of the original demonstration data. By freely combining different dismantling table objects and recycling bin destinations, a massive number of synthetic task samples are exponentially derived from the limited demonstrations, filling the gaps in the long-tail distribution of training data and significantly improving the model's generalization boundary. This allows the model to not only learn the semantic logic represented by the atomic facts but also learn how to transform this logic into specific, executable action steps. This end-to-end training method ensures that when faced with new synthetic tasks, the model can generate accurate action sequences that conform to physical constraints and business logic, achieving an effective transformation from logical cognition to physical execution.

[0037] It should be added that images of non-standard industrial scenarios are decomposed into atomic facts, and pre-defined recycling business logic, such as material classification rules and container load balancing rules, are used to programmatically reorganize the atomic facts to synthesize multi-image, instruction, and action triples with strong logical dependencies, so as to ensure the absolute correctness and diversity of data logic and avoid the loss of logic caused by traditional cropping and augmentation.

[0038] For example, if sorting demonstration data is used as the original sample, the original sample consists of two disassembly tables and two material bins. The task is "which disassembly table's workpiece should be placed in which recycling bin?" The answer is "move to disassembly table 1, grab the part selected by the bbox on disassembly table 1, move it to recycling bin 2, and place the part."

[0039] Therefore, the atomic facts can be broken down into three atomic facts: "A. For dismantling stations 1 and 2, which grid should be selected under business constraints?", "B. For the image of dismantling station 1, which part should be selected?", and "C. For recycling bins 1 and 2, which recycling bin should be selected?". Correspondingly, during reassembly, facts A and B from the original sample 1 can be combined with fact C from the original sample 2 to synthesize a new sample, i.e., a synthesized task sample. The corresponding generated instruction is: "Grab a specific workpiece to be sorted and place it into the corresponding recycling bin." Specifically, the sorting demonstration data includes a sequence of images of the workbench and a sequence of images of the recycling bins. Based on the atomic facts of the dismantling station and the recycling bins in the atomic facts, the data is recombined according to the recycling business logic to generate a synthetic task sample. This includes: building a program generation engine and embedding the recycling business logic; the recycling business logic includes material classification rules, hazardous waste priority treatment logic, and / or load balancing logic; calling the atomic facts of the dismantling station and the recycling bins in the atomic facts through the program generation engine, and recombining them according to the recycling business logic based on the attributes of the workpieces to be sorted on the dismantling station and the container status of the recycling bins to generate corresponding instruction text and synthetic action sequences; using the corresponding workbench images and recycling bin images as image inputs, and combining the corresponding instruction text and synthetic action sequences, a synthetic task sample is obtained.

[0040] In an optional embodiment, after obtaining the synthetic task sample, the method includes: performing data augmentation processing on the image input in the synthetic task sample; the data augmentation processing includes random shuffling of image order and / or random injection of interfering images; wherein, random shuffling of image order is used to break the inherent positional arrangement of the input images, forcing the model to establish cross-modal associations based on image content semantics; random injection of interfering images is used to randomly insert interfering images unrelated to the current task into the input image sequence, so as to force the model to establish the connection between objects and boxes based on image content rather than positional order, eliminate the model's overfitting to image positions, and improve the model's anti-interference ability.

[0041] It should be added that the interference images can be selected according to the actual design requirements, such as a photo of an empty conveyor belt or a photo of a processed workpiece. The number of interference images can be set according to prior experience and actual training requirements, such as 1-2 images. No further limitation is made here.

[0042] In addition, training the sorting action prediction model to be trained includes: for each iteration step, inputting the image input and instruction text into the sorting action prediction model to be trained to obtain the training action sequence output by the sorting action prediction model to be trained; when the image input length is less than a preset length threshold, constructing a total loss function based on the training action sequence and the corresponding action sequence; when the image input length is greater than or equal to the preset length threshold, constructing a total loss function based on the training action sequence and the corresponding action sequence, combined with global contrastive loss constraints; updating the model parameters based on the total loss function and iterating again until the total loss function converges or the maximum number of iterations is reached, and then ending the training.

[0043] Furthermore, the total loss function is constructed, including: constructing a cross-entropy loss function based on the training action sequence and the corresponding action sequence; constructing an indicator function based on whether the image input length is greater than a preset length threshold; constructing a global contrastive loss function based on the instruction intent extracted from the image input and instruction text input to the sorting action prediction model to be trained, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input; and obtaining the total loss function by combining the global contrastive loss function and the indicator function with the cross-entropy loss function.

[0044] It should be added that the total loss function is expressed as: in, Represents the total loss function; Represents the cross-entropy loss function; Indicates characteristic functions; Represents the global contrastive loss function; This represents the weighting parameter.

[0045] Furthermore, a global contrastive loss function is constructed, including: extracting the instruction intent vector, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input from the sorting action prediction model to be trained based on inputting the image input and instruction text; performing gradient blocking operation on the local vector to obtain the blocked local vector; determining the first similarity between the blocked local vector and the instruction intent vector, and determining the second similarity between the global vector and the instruction intent vector; and obtaining the global contrastive loss function based on the first similarity and the second similarity, combined with a preset boundary parameter threshold.

[0046] It should be added that the gradient blocking operation is configured to keep the parameters of the local vectors fixed and only update the parameters of the global vectors during the computation of the global contrastive regularization loss, so as to force the model to focus on global information when understanding the intent.

[0047] In addition, the global contrastive loss function is expressed as: in, Indicates the first similarity; Indicates the command intent vector; Represents a local vector; Gradient blocking, or gradient blocking, is used to prevent the model from modifying local features by adding gradient blocking to local vectors, thus forcing the model to boost global vectors instead. The quality of the model is improved, thereby effectively correcting recency bias and ensuring that the model utilizes information from both the dismantling station and the recycling bin. Indicates the second similarity; Represents a global vector; This represents the preset boundary parameter threshold, which can be selected according to the actual experiment, for example, 0.1.

[0048] Forced by the above formula q With global vector The second similarity must be greater than that of the second similarity. q With local vectors The first similarity is higher than a threshold α This approach strongly correlates the instruction intent with the global vector, forcing the model to learn to transmit and compress key information from all preceding images (such as "which control panel has a faulty PCB") into the features of the final image, making it the carrier of the global state. The model recognizes that local information alone (such as seeing only the control panel but not the box's state) is insufficient for decision-making. Therefore, during inference, the model actively reduces the weight of local vectors to prevent premature target locking.

[0049] Step 103: Based on the action prediction sequence, Cartesian space path planning is used to execute the corresponding actions to sort the workpieces to be sorted on the corresponding operating table into the corresponding recycling bins.

[0050] In this embodiment, based on the action prediction sequence, Cartesian space path planning is used to execute the corresponding actions, including: parsing the action prediction sequence to obtain movement instructions, grasping instructions, and placement instructions; extracting the pixel coordinates of the workpiece to be sorted and the pixel coordinates of the recycling bin to be collected; mapping the pixel coordinates of the workpiece and the recycling bin to physical space three-dimensional coordinates in the corresponding robotic arm base coordinate system according to the camera intrinsic parameter matrix and the pre-calibrated hand-eye calibration matrix; performing Cartesian space path planning based on the corresponding physical space coordinates to generate a collision-free motion trajectory, and controlling the robotic arm to perform the corresponding grasping and placement actions according to the motion trajectory.

[0051] It should be added that after completing the corresponding grasping and placement actions, the robotic arm resets and triggers the acquisition of the next image sequence. Furthermore, the grasping instruction includes the workpiece pixel coordinates to be sorted, determined based on the image pixel coordinate system, i.e., the grasping target position; the placement instruction includes the pixel coordinates of the recycling bin to be collected, i.e., the working area position determined based on the target recycling bin.

[0052] In summary, the embodiments of the present invention decompose complex demonstration data into fine-grained atomic facts and then reassemble them according to the recycling business logic to derive a massive amount of logically reasonable synthetic training samples from limited real data. This reduces the dependence on manual demonstration data and ensures that the training data covers all possible sorting combinations and edge cases, significantly improving the adaptability and robustness of the model when facing unknown sorting tasks.

[0053] Furthermore, by inputting image sequences and task instructions into the sorting action prediction model, an attention mechanism is used to guide the model to automatically filter out background noise and irrelevant interference, accurately focus on the decisive features of the workpiece, establish a deep semantic mapping between it and the text task instructions, and perform semantic confirmation in conjunction with the task instructions. This implicitly integrates the prediction of the feasibility of subsequent planning, ensuring that the model can accurately locate the target to be sorted in chaotic scenes, especially irregularly shaped items, and visually lock onto the parts suitable for grasping. This provides a precise perceptual basis for the generation of subsequent actions, realizing the leap from perception to cognition. The action prediction sequence is used as a macro guide, combined with Cartesian space path planning for specific execution, forming a hierarchical architecture of decision-making and execution. This retains the end-to-end model's intelligent understanding and generalization ability in complex scenes, while introducing the stability and safety guarantees of classic planning algorithms. This effectively avoids obstacles and ensures the smoothness and accuracy of the motion trajectory, fundamentally solving the problem of the disconnect between perception and execution, and ensuring the intelligence of the sorting action and the stability of the physical execution process.

[0054] The following describes the non-standard irregular workpiece sorting device based on cross-view joint understanding provided by the present invention. The non-standard irregular workpiece sorting device based on cross-view joint understanding described below and the non-standard irregular workpiece sorting method based on cross-view joint understanding described above can be referred to in correspondence.

[0055] Figure 2 A schematic diagram of a non-standard, irregularly shaped workpiece sorting device based on cross-view joint understanding is shown. The device includes: Image acquisition module 21 acquires an image sequence containing multiple images of the workbench and the recycling bin; The action prediction module 22 inputs the image sequence and task instructions into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model. The action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by the recycling business logic constraints. The sorting action prediction model is used to determine the cross-modal semantic association between the workpiece to be sorted on the operating table and the recycling bin in the input image sequence based on the attention mechanism, and to generate the sorting action prediction sequence in combination with the task instructions. The workpiece sorting module 23, based on the action prediction sequence, uses Cartesian space path planning to execute the corresponding actions to sort the workpieces to be sorted on the corresponding operating table into the corresponding recycling bins.

[0056] It should be noted that the specific principles of the embodiments of the present invention are the same as those of the method embodiments described above. For details, please refer to the method embodiments above. More detailed explanations will not be repeated here.

[0057] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logic instructions in the memory 330 to execute a sorting method for non-standard irregular workpieces based on cross-view joint understanding. The method includes: acquiring an image sequence containing multiple operation table images and recycling bin images; inputting the image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operation table and the recycling bin in the input image sequence based on an attention mechanism, and generate a sorting action prediction sequence in combination with the task instructions; according to the action prediction sequence, the corresponding action is executed using Cartesian space path planning to sort the workpieces to be sorted on the corresponding operation table to the corresponding recycling bin.

[0058] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the sorting method for non-standard irregular workpieces based on cross-view joint understanding provided by the above methods. The method includes: acquiring an image sequence containing multiple operation table images and recycling bin images; inputting the image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the sorting action prediction model; wherein, the action prediction model is trained using sorting demonstration data and atomic facts obtained by semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operation table and the recycling bin in the input image sequence based on an attention mechanism, and generate a sorting action prediction sequence in combination with the task instructions; according to the action prediction sequence, using Cartesian space path planning, executing the corresponding action to sort the workpieces to be sorted on the corresponding operation table to the corresponding recycling bin.

[0060] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the sorting method for non-standard irregular workpieces based on cross-view joint understanding provided by the above methods. The method includes: acquiring an image sequence containing multiple workstation images and recycling bin images; inputting the image sequence and task instructions into a sorting action prediction model to obtain a sorting action prediction sequence output by the sorting action prediction model; wherein the action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and synthesized task samples generated by recycling business logic constraints; the sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the workstations and the recycling bins in the input image sequence based on an attention mechanism, and to generate a sorting action prediction sequence in combination with the task instructions; according to the action prediction sequence, using Cartesian space path planning, executing the corresponding actions to sort the workpieces to be sorted on the corresponding workstations to the corresponding recycling bins.

[0061] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding, characterized in that, include: Acquire an image sequence containing multiple console images and recycle bin images; The image sequence and task instructions are input into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model. The action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and then reassembled using recycling business logic constraints to generate synthetic task samples. The sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operating table and the recycling bin in the input image sequence based on an attention mechanism, and combines this with the task instructions to generate the sorting action prediction sequence. Based on the predicted action sequence, Cartesian space path planning is used to execute the corresponding actions to sort the workpieces to be sorted on the corresponding worktable into the corresponding recycling bin.

2. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 1, characterized in that, The image sequence and task instructions are input into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model, including: The image sequence and task instructions are input into the sorting action prediction model to perform visual encoding on the input image sequence to obtain image sequence encoding, and to perform text lexicalization on the input task instructions to obtain instruction text lexical sequence. Based on the image sequence encoding and the instruction text word sequence, a multimodal joint feature sequence is obtained using a cross-attention mechanism. Then, through an autoregressive decoding method, a sorting action prediction sequence containing the grasping pose and the placement pose is gradually predicted and output.

3. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 1, characterized in that, Before inputting the image sequence and task instructions into the sorting action prediction model, the following steps are included: The sorting demonstration data is semantically atomized using a large model to obtain fine-grained atomic facts; these atomic facts are used to characterize object attributes, states, and coordinate information. Based on the dismantling station atomic facts and recycling bin atomic facts in the atomic facts, the data are recombined according to the recycling business logic to generate synthetic task samples, and combined with the sorting demonstration data to obtain training samples; the training samples include image input, instruction text, and action sequences. The image input and instruction text are used as input data for training, and the action sequence is used as labels for training. The sorting action prediction model to be trained is trained to obtain a sorting action prediction model for predicting sorting actions.

4. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 3, characterized in that, The training of the sorting action prediction model includes: For each iteration step, the image input and instruction text are input into the sorting action prediction model to be trained, and the training action sequence output by the sorting action prediction model to be trained is obtained. When the image input length is determined to be less than a preset length threshold, a total loss function is constructed based on the training action sequence and the corresponding action sequence. When the image input length is determined to be greater than or equal to the preset length threshold, a total loss function is constructed based on the training action sequence and the corresponding action sequence, combined with the global contrastive loss constraint. The model parameters are updated based on the total loss function and the iteration continues until the total loss function converges or the maximum number of iterations is reached, at which point the training ends.

5. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 4, characterized in that, Construct the total loss function, including: Based on the training action sequence and the corresponding action sequence, construct the cross-entropy loss function; A characteristic function is constructed based on whether the image input length is greater than a preset length threshold. Based on the instruction intent extracted from the image input and instruction text input into the sorting action prediction model to be trained, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input, a global contrast loss function is constructed. Based on the global contrastive loss function and the indicator function, combined with the cross-entropy loss function, the total loss function is obtained.

6. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 5, characterized in that, Construct a global contrastive loss function, including: Based on inputting the image input and instruction text into the sorting action prediction model to be trained, the instruction intent vector, the global vector of the last image in the corresponding image input, and the local vector of the preceding image in the corresponding image input are extracted. Perform a gradient blocking operation on the local vector to obtain the blocked local vector; Determine the first similarity between the local vector after blocking and the instruction intent vector, and determine the second similarity between the global vector and the instruction intent vector; Based on the first similarity and the second similarity, and combined with a preset boundary parameter threshold, a global contrast loss function is obtained.

7. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 3, characterized in that, The sorting demonstration data includes a sequence of images of the control panel and a sequence of images of the recycling bins; Based on the dismantling platform atomic facts and recycling bin atomic facts in the atomic facts, the data is recombined according to the recycling business logic to generate a synthetic task sample, including: Build a program generation engine and embed recycling business logic; The recycling business logic includes material classification rules, hazardous waste priority treatment logic, and / or load balancing logic; The program generation engine calls the dismantling station atomic facts and recycling bin atomic facts in the atomic facts. Based on the attributes of the workpieces to be sorted on the dismantling station and the container status of the recycling bin, the data is recombined according to the recycling business logic to generate corresponding instruction text and synthetic action sequence. The corresponding console image and recycling bin image are used as image inputs, and combined with the corresponding instruction text and the synthesized action sequence, a synthesized task sample is obtained.

8. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 7, characterized in that, After obtaining the synthetic task sample, the following is included: The image input in the synthetic task samples is subjected to data augmentation processing; the data augmentation processing includes randomly shuffling the image order and / or randomly injecting interference images; The random shuffling of the image order is used to disrupt the inherent positional arrangement of the input images; the random injection of interference images is used to randomly insert interference images unrelated to the current task into the input image sequence.

9. The sorting method for non-standard irregularly shaped workpieces based on cross-view joint understanding according to claim 1, characterized in that, Based on the predicted action sequence, Cartesian space path planning is used to execute the corresponding actions, including: The action prediction sequence is parsed to obtain the movement command, grab command and drop command, and the pixel coordinates of the workpiece to be sorted and the pixel coordinates of the recycling bin to be collected are extracted. Based on the camera intrinsic parameter matrix and the pre-calibrated hand-eye calibration matrix, the pixel coordinates of the workpiece and the pixel coordinates of the recycling bin are respectively mapped to the physical space three-dimensional coordinates under the corresponding robotic arm base coordinate system; Cartesian path planning is performed based on the corresponding physical space coordinates to generate a collision-free motion trajectory, and the robotic arm is controlled to perform corresponding grasping and dropping actions according to the motion trajectory.

10. A sorting device for non-standard irregularly shaped workpieces based on cross-view joint understanding, characterized in that, include: The image acquisition module acquires an image sequence containing multiple images of the workbench and the recycling bin; The action prediction module inputs the image sequence and task instructions into the sorting action prediction model to obtain the sorting action prediction sequence output by the sorting action prediction model. The action prediction model is trained using sorting demonstration data and atomic facts obtained from the semantic atomization decomposition of the sorting demonstration data, and then reassembled using recycling business logic constraints to generate synthetic task samples. The sorting action prediction model is used to determine the cross-modal semantic association between the workpieces to be sorted on the operating table and the recycling bin in the input image sequence based on an attention mechanism, and then combines this with the task instructions to generate the sorting action prediction sequence. The workpiece sorting module executes corresponding actions based on the predicted action sequence and Cartesian space path planning to sort the workpieces to be sorted on the corresponding operating table into the corresponding recycling bins.

Images