Visual task dynamic scheduling method and system oriented to multi-station collaboration

Abstract

The invention discloses a visual task dynamic scheduling method and system oriented to multi-station collaboration, and the method comprises the following steps: S1, collecting visual data corresponding to a plurality of operation stations, and generating a multi-station visual perception data set in combination with the pose information of a mobile chassis; s2, constructing a multi-station operation state description result; s3, forming a visual task set; s4, constructing a task station relation graph, inputting the task station relation graph into an improved Grapporter model, introducing an adjustable station offset coding mechanism, and outputting a visual task execution sequence; s5, executing an operation action corresponding to the visual task; s6, updating a multi-station operation state description result; and S7, executing the visual tasks according to the updated visual task execution sequence. According to the method, the visual task execution sequence can be dynamically adjusted according to the multi-station operation state change, the moving path cost is reduced, and the overall efficiency and scheduling reliability of multi-station collaborative operation are improved.

Description

Technical Field

[0001] This invention relates to the field of robot vision perception and intelligent scheduling control technology, and in particular to a method and system for dynamic scheduling of vision tasks for multi-station collaboration. Background Technology

[0002] With the widespread application of mobile robots, flexible manufacturing, and intelligent work units in industrial production, warehousing and logistics, and agricultural operations, visual perception, task planning, and collaborative scheduling technologies in multi-workstation environments have gradually become a focus of research and application. In existing technologies, when a mobile chassis carries a robotic arm and vision system to perform multi-workstation tasks, it typically relies on preset rules or static sequences to arrange visual tasks, or makes local decisions based solely on the state of a single workstation. When faced with complex scenarios involving a large number of workstations, dynamically changing task distribution, and frequent fluctuations in path costs, overall scheduling efficiency and system stability are difficult to guarantee.

[0003] In practical applications, existing multi-station vision task scheduling schemes generally suffer from the following problems: The correlation between visual data and mobile chassis pose information is weak, making it difficult to reflect changes in the spatial position of the work target in a timely manner in task scheduling decisions, resulting in a mismatch between the task execution order and the actual work status; the description of multi-station work status is relatively crude, lacking unified modeling of key factors such as the number of work targets, spatial distribution, and path costs between workstations, easily leading to local optima or repeated movements; in the task sorting and scheduling decision-making process, existing methods mostly use simple weights or heuristic rules, making it difficult to characterize the complex relationships between task nodes and workstation nodes, especially failing to effectively introduce the impact of inter-workstation movement costs on the scheduling results; when the environmental state changes during work execution, the task reordering conditions are not clearly defined or the triggering mechanism is lagging, resulting in untimely system response and affecting overall work efficiency and collaboration capabilities.

[0004] Therefore, how to provide a method and system for dynamic scheduling of visual tasks for multi-workstation collaboration is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a dynamic scheduling method and system for visual tasks in multi-workstation collaboration. Based on multi-workstation visual perception data and mobile chassis pose information, this invention constructs a description of the multi-workstation operation status. By constructing a task-workstation relationship graph and introducing an improved Graphormer model and a calibrable workstation bias coding mechanism, the execution order of visual tasks is dynamically modeled and scheduled, realizing adaptive scheduling and collaborative execution of visual tasks among multiple workstations. This invention has the advantages of high scheduling flexibility, low operation path cost, strong task response capability, and high multi-workstation collaboration efficiency.

[0006] A method for dynamic scheduling of visual tasks for multi-station collaboration according to an embodiment of the present invention includes the following steps:

[0007] S1. Collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set.

[0008] S2. Based on the multi-station visual perception data set, construct the multi-station operation status description result;

[0009] S3. Based on the work objectives in the multi-station visual perception data set, generate visual tasks corresponding to the work objectives to form a visual task set;

[0010] S4. Based on the multi-station operation status description results and the visual task set, construct a task station relationship graph. Input the task station relationship graph into the improved Graphormer model. The improved Graphormer model introduces a calibrable station bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order.

[0011] S5. Based on the execution sequence of the vision task, control the mobile chassis to navigate and switch between work stations, and after arriving at the corresponding work station, control the robotic arm to perform the work action corresponding to the vision task.

[0012] S6. Collect the execution results of the vision task and update the multi-station operation status description results based on the execution results;

[0013] S7. If the updated multi-station job status description result meets the task rearrangement condition, then the updated visual task execution order is generated, and the visual tasks are executed according to the updated visual task execution order.

[0014] Optionally, S1 specifically includes:

[0015] Collect visual data corresponding to multiple workstations. The visual data includes image data and depth data. Associate each set of visual data with a workstation identifier and a collection timestamp.

[0016] The pose information of the mobile chassis is collected, including position coordinates and attitude angles, and a collection timestamp is associated with the pose information.

[0017] Based on the acquisition timestamp, the visual data and the pose information are time-aligned to form aligned data.

[0018] Based on the alignment data, spatial coordinate mapping processing is performed on the visual data to generate a multi-station visual perception data set with associated workstation identifiers and pose information.

[0019] Optionally, S2 specifically includes:

[0020] Based on a multi-workstation visual perception dataset, the work targets associated with workstation identifiers in the visual data are parsed and processed to obtain the quantity information of the work targets;

[0021] Based on the spatial location data corresponding to the task objectives, generate spatial distribution information of the task objectives;

[0022] Based on the pose information of the mobile chassis, calculate the path cost information for the mobile chassis to reach the work station;

[0023] Based on the information on the number of work targets, the spatial distribution of work targets, and the path cost information, a multi-station work status description result is constructed.

[0024] Optionally, S3 specifically includes:

[0025] Based on a multi-station visual perception dataset, the task target in the visual data is identified and processed to obtain the spatial location data corresponding to the task target.

[0026] Based on the work station identifiers and spatial location data associated with the work objectives, a corresponding visual task is generated for each work objective.

[0027] Visual tasks are associated with workstation identifiers and spatial location data to form a set of visual tasks.

[0028] Optionally, the construction of the task-workstation relationship diagram based on the multi-workstation operation status description results and the visual task set specifically includes:

[0029] Visual tasks are used as task nodes, and task nodes are associated with task target identifiers, workstation identifiers, and spatial location data.

[0030] The workstation is used as a workstation node, and the workstation node is associated with the quantity information of the work target, the spatial distribution information of the work target, and the path cost information.

[0031] Based on the correspondence between visual tasks and workstation identifiers, generate association edges between task nodes and workstation nodes;

[0032] Generate connection edges between workstation nodes based on the path cost information between workstations;

[0033] Combine task nodes, workstation nodes, and associated edges to form a task-workstation relationship diagram.

[0034] Optionally, the improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order, specifically:

[0035] Based on the task-workstation relationship diagram, the operation target identifier, operation station identifier and spatial location data associated with the task node are encoded, and the operation target quantity information, operation target spatial distribution information and path cost information associated with the workstation node are encoded to generate task node feature vectors and workstation node feature vectors.

[0036] The calibrable workstation bias coding mechanism includes:

[0037] Based on the path cost information between workstations, numerical mapping processing is performed on the path cost between workstation nodes to generate workstation bias code values ​​between workstation nodes. The workstation bias code values ​​are then associated with the connection edges between workstation nodes to form a workstation bias code set.

[0038] The task node feature vector, workstation node feature vector, the associated edges between task nodes and workstation nodes, the connection edges between workstation nodes, and the workstation bias encoding set are input into the improved Graphormer model. The workstation bias encoding value is superimposed during the calculation of attention weights between nodes to generate a node relationship representation.

[0039] Based on the node relationship representation, a ranking calculation is performed on the visual task to generate a ranking score, and the execution order of the visual task is generated according to the ranking score.

[0040] Optionally, S5 specifically includes:

[0041] Based on the execution order of visual tasks, read the workstation identifiers and spatial location data associated with the visual tasks to determine the target workstation;

[0042] Based on the pose information of the target work station and the mobile chassis, a navigation path for the mobile chassis is generated, and the mobile chassis is controlled to travel along the navigation path to the target work station.

[0043] After the mobile chassis arrives at the target work station, the robot arm's work trajectory is generated based on the spatial position data associated with the vision task, and the robot arm is controlled to perform work actions according to the work trajectory.

[0044] Optionally, S6 specifically includes:

[0045] Collect the results of visual task execution. The results include visual task identifier, workstation identifier, task completion status, and task end timestamp.

[0046] Based on the job execution results and the multi-station visual perception data set, update processing is performed on the job target quantity information and job target spatial distribution information to generate updated job target quantity information and updated job target spatial distribution information;

[0047] Based on the job execution results and the pose information of the mobile chassis, the path cost information of the mobile chassis to the work station is updated to generate updated path cost information.

[0048] Based on the updated information on the number of job targets, the updated spatial distribution information of job targets, and the updated path cost information, the multi-station job status description results are updated.

[0049] Optionally, S7 specifically includes:

[0050] Based on the updated multi-station operation status description results, the target quantity information of each operation station is statistically analyzed; the difference between the updated target quantity information and the previous target quantity information is calculated to obtain the change value of the target quantity; if the change value of the target quantity is greater than the first preset threshold, it is determined that the multi-station operation status description results meet the task rearrangement conditions.

[0051] Based on the updated spatial distribution information of the task targets, calculate the distance between any task target spatial location and the center location of the corresponding task station; calculate the difference between the distance value and the corresponding distance value before the update to obtain the task target spatial offset; if any task target spatial offset is greater than a second preset threshold, determine that the multi-station task status description result meets the task rearrangement condition.

[0052] Based on the updated path cost information, calculate the total path cost of the mobile chassis to the work station; compare the updated total path cost with the total path cost before the update; if the total path cost is greater than the third preset threshold, determine that the multi-workstation operation status description result meets the task rearrangement condition.

[0053] If any judgment result satisfies the task rearrangement condition, the task station relationship graph is reconstructed based on the updated multi-station operation status description result and the visual task set; the reconstructed task station relationship graph is input into the improved Graphormer model, and a calibrable station bias encoding mechanism is introduced to perform graph structure modeling processing to generate the updated visual task execution order; the visual task is executed based on the updated visual task execution order.

[0054] Optionally, a dynamic scheduling system for vision tasks in a multi-workstation collaborative manner includes the following modules:

[0055] The perception and acquisition module is used to collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set.

[0056] The status construction module is used to construct a multi-workstation operation status description result based on the multi-workstation visual perception data set.

[0057] The task generation module is used to generate visual tasks corresponding to the work objectives based on the work objectives in the multi-station visual perception data set, thereby forming a visual task set;

[0058] The scheduling decision module is used to construct a task-workstation relationship graph based on the multi-workstation operation status description results and the visual task set. The task-workstation relationship graph is input into the improved Graphormer model. The improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order.

[0059] The execution update module is used to control the mobile chassis to navigate and switch between workstations based on the execution order of the vision tasks, and after arriving at the corresponding workstation, control the robotic arm to perform the work actions corresponding to the vision tasks; collect the work execution results of the vision tasks, and update the multi-workstation work status description results based on the work execution results; if the updated multi-workstation work status description results meet the task rearrangement conditions, then generate an updated vision task execution order, and execute the vision tasks according to the updated vision task execution order.

[0060] The beneficial effects of this invention are:

[0061] (1) Based on the multi-station visual perception data, construct the multi-station operation status description results, and integrate the number of operation targets, spatial distribution and movement path cost into the scheduling decision process, so that the execution order of visual tasks can be dynamically updated with the change of operation status, which significantly improves the flexibility and adaptability of scheduling.

[0062] (2) By constructing a task station relationship graph, visual tasks and work stations are jointly modeled in the same graph structure. In the process of graph structure modeling, an calibrable station bias coding mechanism is introduced through an improved Graphormer model, so that the path cost difference between different work stations can directly participate in the visual task ranking calculation, overcoming the problem that it is difficult to accurately reflect the spatial movement cost in the existing technology, thereby reducing invalid movement and path redundancy.

[0063] (3) This invention proposes a multi-condition task reordering judgment method based on changes in the number of task targets, spatial location offset, and path cost. When the task status changes significantly, visual task reordering is automatically triggered, enabling the system to have continuous adaptive adjustment capabilities, avoiding the failure of scheduling results due to environmental changes, and improving the overall efficiency and operational stability of multi-workstation collaborative operations. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a dynamic scheduling method for visual tasks oriented towards multi-station collaboration proposed in this invention;

[0066] Figure 2 This is a schematic diagram of the structure of the improved Graphormer model proposed in this invention;

[0067] Figure 3 This is a schematic diagram of a vision task dynamic scheduling system for multi-station collaboration proposed in this invention. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0069] refer to Figure 1 and Figure 2 A dynamic scheduling method for visual tasks in multi-station collaboration includes the following steps:

[0070] S1. Collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set.

[0071] S2. Based on the multi-station visual perception data set, construct the multi-station operation status description result;

[0072] S3. Based on the work objectives in the multi-station visual perception data set, generate visual tasks corresponding to the work objectives to form a visual task set;

[0073] S4. Based on the multi-station operation status description results and the visual task set, construct a task station relationship graph. Input the task station relationship graph into the improved Graphormer model. The improved Graphormer model introduces a calibrable station bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order.

[0074] S5. Based on the execution sequence of the vision task, control the mobile chassis to navigate and switch between work stations, and after arriving at the corresponding work station, control the robotic arm to perform the work action corresponding to the vision task.

[0075] S6. Collect the execution results of the vision task and update the multi-station operation status description results based on the execution results;

[0076] S7. If the updated multi-station job status description result meets the task rearrangement condition, then the updated visual task execution order is generated, and the visual tasks are executed according to the updated visual task execution order.

[0077] This implementation method collects visual data corresponding to multiple workstations and combines it with the pose information of the mobile chassis to form a multi-workstation visual perception data set, enabling a unified representation of the spatial state of the multi-workstation work environment. Furthermore, based on the multi-workstation visual perception data set, a multi-workstation work state description result is constructed, allowing for a structured characterization of state information such as the number of work targets, spatial distribution, and path costs. Simultaneously, a visual task set is generated according to the work targets, establishing a clear correspondence between work requirements and scheduling objects. Further, by constructing a task-workstation relationship diagram and introducing an improved Gradient Graph with a calibrable workstation bias coding mechanism... The Phormer model performs graph structure modeling, which effectively expresses the relationship between tasks and workstations during scheduling, thereby outputting a reasonable visual task execution order. Simultaneously, based on the visual task execution order, the mobile chassis and robotic arm are controlled to perform operational actions, allowing the scheduling results to directly impact the operational process. Furthermore, by collecting operational execution results and updating the multi-workstation operational status descriptions, the operational status is dynamically adjusted during execution. Even further, when changes in operational status meet the task rearrangement conditions, an updated visual task execution order is generated, enabling dynamic scheduling capabilities for multi-workstation collaborative operations.

[0078] In this embodiment, S1 specifically refers to:

[0079] When collecting visual data corresponding to multiple workstations, image data and depth data are simultaneously acquired by visual acquisition devices set at the front end of the mobile chassis and the end of the robotic arm. The image data is a two-dimensional color image, and the depth data is a distance information matrix that corresponds one-to-one with the image pixels. The acquisition frequency of the visual acquisition device is set to 10fps to 30fps, preferably 20fps, to ensure the continuity and integrity of the workstation scene changes.

[0080] During the visual data acquisition process, a work station identifier and an acquisition timestamp are written for each frame of image data and depth data. The work station identifier is a pre-set integer code used to distinguish work stations in different spatial locations. The acquisition timestamp uses millisecond-level time stamps to describe the acquisition sequence of visual data.

[0081] When collecting the pose information of the mobile chassis, the position coordinates and attitude angles are obtained through the positioning and attitude sensing device inside the mobile chassis. The position coordinates are represented by two-dimensional or three-dimensional rectangular coordinates, and the attitude angles include heading angle and pitch angle information. The frequency of pose information collection is consistent with the frequency of visual data collection, and each set of pose information is associated with a collection timestamp of the same format.

[0082] When performing time alignment processing on visual data and pose information based on the acquisition timestamp, visual data with a timestamp difference of less than 0.05s is matched with pose information, and visual data exceeding this time difference threshold is not included in the alignment processing. The 0.05s is used as the first preset time alignment threshold to control the data synchronization accuracy.

[0083] Based on the visual data and pose information aligned with the completion time, when performing spatial coordinate mapping processing on the visual data, the image coordinates are converted into spatial coordinates according to the distance value corresponding to each pixel in the depth data. Combined with the position coordinates and attitude angle of the moving chassis at the corresponding time, the spatial coordinates are transformed into a unified coordinate system so that the position of the work target in the visual data is mapped to a unified spatial coordinate system with the work station as the reference.

[0084] Through spatial coordinate mapping processing, a multi-station visual perception data set is generated that simultaneously associates the workstation identifier, spatial coordinate information, pose information, and acquisition timestamp. The multi-station visual perception data set is stored in the form of structured data.

[0085] In this embodiment, S2 specifically refers to:

[0086] Based on a multi-workstation visual perception dataset, the visual data of associated workstation identifiers is processed frame by frame. Under a unified spatial coordinate system, the work targets presented in the visual data are parsed. By performing aggregation judgment on the target positions with a spatial coordinate distance of less than 0.1m under the same workstation identifier, targets with overlapping spatial positions or a height difference of less than 0.05m in consecutive frames are judged as the same work target. The judged work targets are counted to obtain the work target quantity information corresponding to each workstation. The work target quantity information is represented in the form of positive integers.

[0087] Based on the spatial location data corresponding to the work targets obtained by analysis, the spatial distribution of the work targets under the same work station identifier is sorted out. By recording the three-dimensional coordinate vectors of the work targets in a unified spatial coordinate system, the spatial distribution information of the work targets is formed. The spatial distribution information of the work targets includes the relative positional relationship of the work targets in the work station working plane and the set of spatial distances between the targets. Preferably, when the spatial distance between any two work targets is less than 0.2m, it is determined that the work targets are in a densely distributed state.

[0088] By combining the mobile chassis pose information associated with the multi-station visual perception data set, the spatial distance between the current position of the mobile chassis and the reference position of each work station is calculated. In addition, the path cost information of the mobile chassis to reach the work station is quantitatively described by combining the change in steering angle in the path. The path cost information is obtained by weighted summation of travel distance and cumulative steering angle. Preferably, the travel distance weight is set to 0.7 and the steering angle weight is set to 0.3.

[0089] Based on the information on the number of work targets, the spatial distribution of work targets, and the path cost information, the information is summarized and organized according to the work station identifier to form a comprehensive description of the scale of work targets, spatial distribution characteristics, and movement cost characteristics of each work station. This results in a multi-work station work status description, which is stored in the form of structured data.

[0090] In this embodiment, S3 specifically refers to:

[0091] Based on a multi-station visual perception dataset, image data and depth data are jointly analyzed to extract the contours and divide the regions of the work objects presented in the images. Combined with the distance information in the depth data that corresponds one-to-one with the image pixels, the actual position of the work objects in space is confirmed. Work objects that meet the area threshold and height threshold conditions are determined as work targets. The area threshold is preferably set to no less than 500 pixels, and the height threshold is preferably set to no less than 0.02m to eliminate background noise and non-work objects.

[0092] Based on the determined task target, the spatial coordinate information corresponding to the task target is read under a unified spatial coordinate system. The spatial coordinate information includes the position values ​​of the task target in the X-axis, Y-axis and Z-axis directions. The spatial coordinate information is then bound to the task station identifier to form the spatial position data of the task target.

[0093] Based on the work station identifier and spatial location data associated with the work target, a visual task description is constructed for each work target. The visual task description includes at least the work target identifier, work station identifier, work target spatial coordinates, and operation point information corresponding to the work target. The operation point information is obtained by applying a fixed offset to the work target spatial coordinates. The offset is preferably set to 0.05m to limit the work approach position of the end of the robotic arm.

[0094] The generated visual task descriptions are associated with and stored with the corresponding workstation identifiers and spatial location data to form a visual task set containing multiple visual tasks. The visual task set records the one-to-one correspondence between each visual task and the workstation in the form of structured data, which is used for calculating and scheduling the execution order of visual tasks.

[0095] In this embodiment, the construction of a task-workstation relationship diagram based on the multi-workstation operation status description results and the visual task set specifically includes:

[0096] When constructing task nodes, each visual task in the visual task set is treated as an independent task node, and a unique task node identifier is established accordingly. The task target identifier, workstation identifier, and spatial location data associated with the visual task are written into the task node attribute field. The task target identifier is an integer code, the workstation identifier is an integer code, and the spatial location data is in three-dimensional rectangular coordinate form, including x, y, and z values, which are used to describe the position status of the task target in a unified spatial coordinate system.

[0097] When constructing workstation nodes, each workstation in the multi-workstation operation status description result is treated as an independent workstation node, and a unique workstation node identifier is established accordingly. The workstation is associated with the work target quantity information, work target spatial distribution information, and path cost information, which are written into the workstation node attribute fields. The work target quantity information is a non-negative integer used to represent the number of work targets to be processed within the workstation. The work target spatial distribution information is a set of work target spatial location data used to describe the spatial discrete state of the work targets within the workstation range. The path cost information is numerical data used to represent the cost level of the mobile chassis to reach the workstation from the current position. The numerical range of the path cost information is preferably set to 0 to 1000.

[0098] When generating the association edge between the task node and the workstation node, the workstation identifier recorded in the task node is matched with the workstation node identifier. When the workstation identifier of the task node and the workstation node identifier are consistent, an association edge is established between the corresponding task node and the workstation node. The association edge is used to represent the membership relationship between the visual task and the workstation.

[0099] When generating connection edges between workstation nodes, a connection edge is established between any two workstation nodes based on the path cost information in the multi-workstation operation status description results, and the corresponding path cost value is recorded in the connection edge; when the path cost value is less than the preset path reachable threshold, a connection edge is established. Preferably, the path reachable threshold is set to 300, which is used to restrict the establishment of connection relationships only between workstations where the mobile chassis has feasible passage conditions.

[0100] After completing the construction of task nodes, workstation nodes, the associated edges between task nodes and workstation nodes, and the connecting edges between workstation nodes, the nodes and edges are combined according to a unified data structure to form a task-workstation relationship graph containing a set of nodes and a set of edges. The task-workstation relationship graph is stored in the form of graph structure data.

[0101] In this embodiment, the improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order, specifically:

[0102] When constructing node features based on the task-workstation relationship diagram, the task target identifiers and workstation identifiers associated with the task nodes are encoded in integer form. The three-dimensional coordinate values ​​in the spatial location data are then normalized and concatenated to form a continuous numerical feature vector. This normalization process uses the maximum range of spatial coordinate values ​​in the work scene as a reference scale, ensuring that the spatial location features of different task nodes are distributed within a unified numerical range. Simultaneously, the number of task targets associated with workstation nodes is mapped using integer values. The spatial distribution information of task targets is obtained by statistically processing the spatial location data of task targets within the workstation range to obtain the spatial center position value and spatial dispersion value. Path cost information is then normalized, using the maximum value of the multi-workstation path cost as a reference scale, ensuring that the path cost features of different workstation nodes have consistent dimensions. The aforementioned processing generates task node feature vectors and workstation node feature vectors.

[0103] When constructing workstation bias codes based on path cost information between workstations, a numerical mapping process is performed on the path cost values ​​recorded in the connecting edges between workstation nodes. The path cost values ​​are mapped to bias code values. Smaller path cost values ​​correspond to larger bias code values, and larger path cost values ​​correspond to smaller bias code values. The mapping relationship is completed in a linear proportional manner. The mapping range is preferably set to -10 to 0 to characterize the difference in the difficulty of passage between workstations. The generated bias code values ​​are then associated one-to-one with the connecting edges between workstation nodes to form a workstation bias code set consistent with the set of connecting edges.

[0104] The feature vectors of task nodes, feature vectors of workstation nodes, the associated edges between task nodes and workstation nodes, the connecting edges between workstation nodes, and the workstation bias encoding set are input into the improved Graphormer model as graph structure input data. During the calculation of attention weights between nodes, the workstation bias encoding values ​​associated with the corresponding connecting edges are numerically superimposed with the attention calculation results between nodes, so that workstation nodes with lower path costs receive higher weights in attention allocation. This introduces the influence of workstation access cost differences on the node relationship modeling results during graph structure modeling, and generates a node relationship representation to describe the comprehensive association strength between task nodes and workstation nodes.

[0105] When performing sorting calculations for visual tasks based on node relationship representation, the node relationship representation corresponding to the task node is mapped to a sorting score. The sorting score is in the form of a continuous numerical value. The larger the sorting score, the higher the priority of the visual task in the scheduling process. Based on the size relationship of the sorting scores of all visual tasks, the visual tasks are arranged in an orderly manner to form the visual task execution order, which is used for task scheduling control in multi-workstation collaborative operation scenarios.

[0106] In this embodiment, S5 specifically refers to:

[0107] When determining the work station based on the execution order of visual tasks, the visual task information is read one by one according to the execution order of visual tasks, and the associated work station identifier and spatial location data are extracted from the visual tasks. The work station identifier is in integer encoding form, and the spatial location data is a three-dimensional rectangular coordinate value, which is used to clarify the target work station location corresponding to the current visual task.

[0108] When generating a navigation path based on the pose information of the target workstation and the mobile chassis, the current position coordinates and attitude angle of the mobile chassis are read. The current position coordinates and the spatial coordinates of the target workstation are used as the starting point and ending point of the path planning. The passable area within the work area is discretized, and a feasible path from the starting point to the ending point is calculated in the discretized space. The feasible path is represented in the form of a continuous path point sequence. The generated path point sequence is smoothed to limit the turning angle and displacement variation between adjacent path points. Preferably, the turning angle of adjacent path points is no greater than 30°, and the displacement distance between adjacent path points is no greater than 0.5m, to ensure the stability of the mobile chassis during driving. Based on the generated navigation path, a driving control command is sent to the mobile chassis to control the mobile chassis to drive along the navigation path, passing through the path points in sequence to the target workstation.

[0109] When generating the robotic arm's work motion after the mobile chassis arrives at the target work station, the spatial position data associated with the vision task is read and used as the coordinates of the target point of the robotic arm's work. Based on the robotic arm's installation pose parameters and joint structure parameters, the coordinates of the target point are converted into the target pose of the robotic arm's end effector, and a continuous work motion trajectory from the initial pose to the target pose is generated in the robotic arm's workspace. The work motion trajectory consists of a set of continuous joint angle sequences. The rate of change of joint angles in the work motion trajectory is constrained. Preferably, the joint angular velocity is no greater than 60° / s to avoid impact on the robotic arm's movement. Based on the generated work motion trajectory, control commands are sent to the robotic arm to control the robotic arm to complete the work motion corresponding to the vision task according to the work motion trajectory.

[0110] In this embodiment, S6 specifically refers to:

[0111] When collecting the execution results of vision tasks, after the robotic arm completes the corresponding task, it records the execution information that corresponds one-to-one with the vision task. The execution information includes vision task identifier, work station identifier, task completion status and task end timestamp. The vision task identifier and work station identifier are in integer encoding form. The task completion status is represented by a binary status, with a value of 1 indicating that the task is completed and a value of 0 indicating that the task is not completed. The task end timestamp is a millisecond-level time stamp used to describe the specific time when the task execution ends.

[0112] When updating the quantity and spatial distribution information of work targets based on the job execution results and the multi-station visual perception data set, the processing status of the work targets is determined according to the completion status of the work actions. When the completion status of the work action is 1, the quantity of work targets in the corresponding work station is decremented by 1. When the completion status of the work action is 0, the quantity of work targets remains unchanged. At the same time, the spatial location data corresponding to the completed work targets are removed from the multi-station visual perception data set, and the spatial location data of the remaining work targets are re-statistically analyzed to generate updated spatial distribution information of work targets. The spatial distribution information of work targets is still represented in the form of a spatial location data set of work targets, which is used to reflect the spatial status of the remaining work targets in the work station.

[0113] When updating the path cost information based on the job execution results and the pose information of the mobile chassis, the position coordinates and attitude angle of the mobile chassis at the time corresponding to the end timestamp of the job action are read, and the actual travel distance and travel time of the mobile chassis from the previous work station to the current work station are combined to correct the original path cost information. The path cost information is obtained by weighting the travel distance and travel time. The sum of the travel distance weight and the travel time weight is 1. Preferably, the travel distance weight is set to 0.6 and the travel time weight is set to 0.4 to reflect both spatial distance factors and time consumption factors.

[0114] Based on the updated information on the number of work targets, the spatial distribution of work targets, and the path cost, the overall description results of the multi-workstation work status are updated, and the updated information is written into the status description data structure of the corresponding workstation, so that the multi-workstation work status description results continuously reflect the current work execution progress and changes in the work environment.

[0115] In this embodiment, S7 specifically refers to:

[0116] Based on the updated multi-workstation operation status description results, the target quantity information of each workstation is statistically processed to obtain the updated target quantity value; at the same time, the target quantity value of the same workstation before the update is read, and the difference between the target quantity values ​​before and after the update is calculated. The difference is calculated by subtracting the target quantity value before the update from the target quantity value after the update to obtain the change value of the target quantity; when the change value of the target quantity corresponding to any workstation is greater than a first preset threshold, it is determined that the multi-workstation operation status description results meet the task rearrangement condition. Preferably, the first preset threshold is set to 2 to identify situations where the target quantity of workstations changes significantly.

[0117] Based on the updated spatial distribution information of the work targets, a distance calculation is performed between the spatial coordinates of each work target and the center coordinates of the work station to which the work target belongs. The distance calculation is the Euclidean distance between the spatial coordinates of the work target and the center coordinates of the work station in a three-dimensional rectangular coordinate system, resulting in an updated target distance value. Simultaneously, the target distance value of the corresponding work target before the update is read, and a difference calculation is performed between the target distance values ​​before and after the update to obtain the spatial offset of the work target. When the spatial offset of any work target is greater than a second preset threshold, it is determined that the multi-work station operation status description result meets the task rearrangement condition. Preferably, the second preset threshold is set to 0.3m to identify situations where the work target undergoes significant spatial migration within the work station area.

[0118] Based on the updated path cost information, the path cost values ​​of the mobile chassis to all work stations are accumulated to obtain the updated total path cost value; at the same time, the corresponding total path cost value before the update is read and the total path cost value before and after the update is compared; when the updated total path cost value is greater than the third preset threshold, it is determined that the multi-work station operation status description result meets the task reordering condition. Preferably, the third preset threshold is set to 1500 to identify the situation where the overall travel cost has increased significantly.

[0119] When any one of the following conditions is met: the result of determining the change in the number of task targets, the result of determining the spatial offset of task targets, or the result of determining the total path cost, the task nodes, workstation nodes, the associated edges between task nodes and workstation nodes, and the connection edges between workstation nodes are regenerated based on the updated multi-workstation operation status description and the visual task set, forming an updated task-workstation relationship graph. The updated task-workstation relationship graph is then input into the improved Graphormer model, and a calibrable workstation bias encoding mechanism is introduced during the node relationship modeling process to generate an updated visual task execution order. Based on the updated visual task execution order, the execution of visual tasks in the multi-workstation collaborative operation process continues to be controlled.

[0120] refer to Figure 3 In this embodiment, a dynamic scheduling system for visual tasks oriented towards multi-station collaboration includes the following modules:

[0121] The perception and acquisition module is used to collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set.

[0122] The status construction module is used to construct a multi-workstation operation status description result based on the multi-workstation visual perception data set.

[0123] The task generation module is used to generate visual tasks corresponding to the work objectives based on the work objectives in the multi-station visual perception data set, thereby forming a visual task set;

[0124] The scheduling decision module is used to construct a task-workstation relationship graph based on the multi-workstation operation status description results and the visual task set. The task-workstation relationship graph is input into the improved Graphormer model. The improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order.

[0125] The execution update module is used to control the mobile chassis to navigate and switch between workstations based on the execution order of the vision tasks, and after arriving at the corresponding workstation, control the robotic arm to perform the work actions corresponding to the vision tasks; collect the work execution results of the vision tasks, and update the multi-workstation work status description results based on the work execution results; if the updated multi-workstation work status description results meet the task rearrangement conditions, then generate an updated vision task execution order, and execute the vision tasks according to the updated vision task execution order.

[0126] Example 1:

[0127] To verify the feasibility of this invention in practice, it was applied to a pecan greenhouse harvesting line. Multiple workstations were arranged along a passageway within the greenhouse, each corresponding to a fixed harvesting area and crating position. A mobile chassis moved back and forth within the passageway, driving a robotic arm to grasp and place the pecans. A vision acquisition device continuously output image and depth data, while a positioning and attitude sensing device continuously outputs pose information. An industrial control computer, operating within a ROS environment, performed navigation control, robotic arm control, and dynamic scheduling of vision tasks. Common problems in this scenario included uneven target density among workstations, path cost fluctuations due to passageway congestion, and spatial displacement of the target position due to branch swaying. Traditional strategies of switching between workstations in a fixed order or greedily based on the nearest distance often resulted in frequent backtracking, amplified local congestion, and increased robotic arm waiting time, diluting the effective harvesting time per day. Furthermore, when target displacement was significant, grasping failures and repeated attempts occurred. The hardware configuration and operating environment of the mobile harvesting robot referenced a typical configuration of a mobile chassis, a 6-axis robotic arm, a depth camera, and a ROS control system.

[0128] When applying this invention in this scenario, after the operation begins, the mobile chassis completes initial positioning at the channel entrance and enters inspection mode. The vision acquisition device synchronously acquires visual data at the front end of the mobile chassis and the end of the robotic arm. The pose information and visual data are aligned in time and spatial coordinate mapping is completed to form a multi-station visual perception data set. The multi-station visual perception data set enters the operation state construction process. The quantity information of the operation target is obtained by parsing the operation target, the spatial distribution information of the operation target is obtained by aggregating the spatial location data, and the path cost information is obtained by navigation estimation under the pose and channel topology. The combination of these three types of information forms the multi-station operation state description result. The visual task set is generated by binding the spatial location data obtained from the operation target identification with the operation station identifier. Each operation target corresponds to one visual task and carries the operation target identifier, the operation station identifier, and the spatial location data. The task station relationship diagram consists of task nodes, station nodes, associated edges, and connecting edges. The task nodes carry visual task side information, and the station nodes carry operation target quantity information, etc. The system generates task execution order by inputting spatial distribution information and path cost information for the task targets, with connecting edges carrying path costs between workstations. The task workstation relationship graph is input into the improved Graphormer model, and workstation bias encoding values ​​are superimposed during the attention weight calculation. The mobile chassis navigates and switches between workstations according to the visual task execution order. After the mobile chassis reaches the target workstation, the robotic arm generates the operation trajectory based on the spatial position data associated with the visual task and completes the picking and placement. The task execution result is written back and triggers a state update. The task target quantity information, task target spatial distribution information, and path cost information are updated synchronously to obtain the updated multi-workstation operation state description result. The task reordering condition is determined by thresholds of the change value of the task target quantity, the task target spatial offset, and the sum of path costs. If any of the conditions is met, the task workstation relationship graph is reconstructed and the updated visual task execution order is output again, thereby maintaining stable scheduling efficiency and execution continuity when the target density changes, the target position shifts, and the channel cost fluctuates.

[0129] To demonstrate the beneficial effects, the same greenhouse, the same picking passage, and the same work group's working hours were selected for comparison. The baseline scheme was "fixed workstation sequence + single task execution upon arrival," while the control scheme was the "task workstation relationship diagram + improved Graphormer model scheduling + task rearrangement" of this invention. The scheme was run continuously for three months, with 20 effective working days counted each month, and the monthly average was taken as the indicator. The results are as follows.

[0130] Table 1. Comparison of Multi-Workstation Collaborative Scheduling Efficiency

[0131]

[0132] Table 1 shows that this invention continuously improves scheduling efficiency and execution consistency in the same scenario. The number of visual tasks completed per day increased by approximately 84, 96, and 111 respectively, and the improvement trend is consistent with the dynamic scheduling capability under changes in the density of work targets. The average scheduling waiting time per task decreased from 8.6s to 5.1s and further to 4.5s, reflecting that the sorting output driven by the task station relationship graph reduced the equality between the robotic arm and the mobile chassis. The daily mileage was lower than the baseline solution for three months, indicating that the execution order of visual tasks suppressed the turnaround path and maintained a shorter channel occupancy even when the path cost fluctuated. The decrease in the number of station switching times indicates that the scheduling does not rely on frequent switching to pursue the local shortest path, but obtains a more stable station access sequence through global relationship modeling. The decrease in the average completion time per task reflects that the scheduling order and the work actions after arrival are more closely connected, and the overall effective work time ratio is increased.

[0133] Table 2 Comparison of Harvesting Quality and Safety Indicators

[0134]

[0135] Table 2 shows that the present invention also consistently benefits both quality and safety. The grasping failure rate decreased by approximately 2.3, 2.4, and 2.4 percentage points in three months, respectively, with a corresponding decrease in the number of repeated attempts. This indicates that after the visual task set is bound to spatial location data, the matching degree between navigation arrival and the generated robotic arm operation trajectory is higher. Furthermore, after the task reordering condition is triggered, targets with significant spatial offsets can be reintegrated into a more suitable execution order, reducing failures due to "target offset after arrival." The decrease in the number of minor collisions is consistent with the decrease in mileage, reflecting smoother navigation under reduced workstation switching and path cost constraints, and a reduction in contact risks caused by channel congestion and detours. The decrease in target miss rate indicates that the visual task execution order is more in line with changes in target spatial distribution, avoiding long periods of inaccessibility to local areas caused by a fixed order. The decrease in daily energy consumption comes from the reduction in invalid mileage and repeated actions. Energy consumption remains advantageous in the third month, indicating that energy expenditure can still be controlled even with increased target density and task volume. Combining the data in the two tables, the present invention can simultaneously improve efficiency, quality, and safety indicators in multi-workstation collaborative scenarios, solving the problems of traditional scheduling being prone to backtracking, waiting, and repeated attempts under conditions of uneven target density and fluctuating path costs.

[0136] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dynamic scheduling method for visual tasks in multi-station collaboration, characterized in that, Includes the following steps: S1. Collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set. S2. Based on the multi-station visual perception data set, construct the multi-station operation status description result; S3. Based on the work objectives in the multi-station visual perception data set, generate visual tasks corresponding to the work objectives to form a visual task set; S4. Based on the multi-station operation status description results and the visual task set, construct a task station relationship graph. Input the task station relationship graph into the improved Graphormer model. The improved Graphormer model introduces a calibrable station bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order. S5. Based on the execution sequence of the vision task, control the mobile chassis to navigate and switch between work stations, and after arriving at the corresponding work station, control the robotic arm to perform the work action corresponding to the vision task. S6. Collect the execution results of the vision task and update the multi-station operation status description results based on the execution results; S7. If the updated multi-station job status description result meets the task rearrangement condition, then the updated visual task execution order is generated, and the visual tasks are executed according to the updated visual task execution order.

2. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S1 is: Collect visual data corresponding to multiple workstations. The visual data includes image data and depth data. Associate each set of visual data with a workstation identifier and a collection timestamp. The pose information of the mobile chassis is collected, including position coordinates and attitude angles, and a collection timestamp is associated with the pose information. Based on the acquisition timestamp, the visual data and the pose information are time-aligned to form aligned data. Based on the alignment data, spatial coordinate mapping processing is performed on the visual data to generate a multi-station visual perception data set with associated workstation identifiers and pose information.

3. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S2 is: Based on a multi-workstation visual perception dataset, the work targets associated with workstation identifiers in the visual data are parsed and processed to obtain the quantity information of the work targets; Based on the spatial location data corresponding to the task objectives, generate spatial distribution information of the task objectives; Based on the pose information of the mobile chassis, calculate the path cost information for the mobile chassis to reach the work station; Based on the information on the number of work targets, the spatial distribution of work targets, and the path cost information, a multi-station work status description result is constructed.

4. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S3 is: Based on a multi-station visual perception dataset, the task target in the visual data is identified and processed to obtain the spatial location data corresponding to the task target. Based on the work station identifiers and spatial location data associated with the work objectives, a corresponding visual task is generated for each work objective. Visual tasks are associated with workstation identifiers and spatial location data to form a set of visual tasks.

5. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, The construction of the task-workstation relationship diagram based on the multi-workstation operation status description results and the visual task set is as follows: Visual tasks are used as task nodes, and task nodes are associated with task target identifiers, workstation identifiers, and spatial location data. The workstation is used as a workstation node, and the workstation node is associated with the work target quantity information, work target spatial distribution information and path cost information; Based on the correspondence between visual tasks and workstation identifiers, generate association edges between task nodes and workstation nodes; Generate connection edges between workstation nodes based on the path cost information between workstations; Combine task nodes, workstation nodes, and associated edges to form a task-workstation relationship diagram.

6. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, The improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the execution order of visual tasks, specifically: Based on the task-workstation relationship diagram, the operation target identifier, operation station identifier and spatial location data associated with the task node are encoded, and the operation target quantity information, operation target spatial distribution information and path cost information associated with the workstation node are encoded to generate task node feature vectors and workstation node feature vectors. The calibrable workstation bias coding mechanism includes: Based on the path cost information between workstations, numerical mapping processing is performed on the path cost between workstation nodes to generate workstation bias code values ​​between workstation nodes. The workstation bias code values ​​are then associated with the connection edges between workstation nodes to form a workstation bias code set. The task node feature vector, workstation node feature vector, the associated edges between task nodes and workstation nodes, the connection edges between workstation nodes, and the workstation bias encoding set are input into the improved Graphormer model. The workstation bias encoding value is superimposed during the calculation of attention weights between nodes to generate a node relationship representation. Based on the node relationship representation, a ranking calculation is performed on the visual task to generate a ranking score, and the execution order of the visual task is generated according to the ranking score.

7. The method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S5 is: Based on the execution order of visual tasks, read the workstation identifiers and spatial location data associated with the visual tasks to determine the target workstation; Based on the pose information of the target work station and the mobile chassis, a navigation path for the mobile chassis is generated, and the mobile chassis is controlled to travel along the navigation path to the target work station. After the mobile chassis arrives at the target work station, the robot arm's work trajectory is generated based on the spatial position data associated with the vision task, and the robot arm is controlled to perform work actions according to the work trajectory.

8. A method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S6 is: Collect the results of visual task execution. The results include visual task identifier, workstation identifier, task completion status, and task end timestamp. Based on the job execution results and the multi-station visual perception data set, update processing is performed on the job target quantity information and job target spatial distribution information to generate updated job target quantity information and updated job target spatial distribution information; Based on the job execution results and the pose information of the mobile chassis, the path cost information of the mobile chassis to the work station is updated to generate updated path cost information. Based on the updated information on the number of job targets, the updated spatial distribution information of job targets, and the updated path cost information, the multi-station job status description results are updated.

9. A method for dynamic scheduling of visual tasks for multi-station collaboration according to claim 1, characterized in that, Specifically, S7 is: Based on the updated multi-station operation status description results, the target quantity information of each operation station is statistically analyzed; the difference between the updated target quantity information and the previous target quantity information is calculated to obtain the change value of the target quantity; if the change value of the target quantity is greater than the first preset threshold, it is determined that the multi-station operation status description results meet the task rearrangement conditions. Based on the updated spatial distribution information of the task targets, calculate the distance between any spatial location of a task target and the center location of the corresponding task station; The difference between the distance value and the corresponding distance value before the update is calculated to obtain the task target space offset; if any task target space offset is greater than the second preset threshold, it is determined that the multi-station operation status description result meets the task rearrangement condition. Based on the updated path cost information, calculate the total path cost of the mobile chassis to the work station; compare the updated total path cost with the total path cost before the update. If the total path cost is greater than the third preset threshold, the result of the multi-station operation status description is determined to meet the task rearrangement condition. If any judgment result satisfies the task rearrangement condition, the task station relationship graph is reconstructed based on the updated multi-station operation status description result and the visual task set; the reconstructed task station relationship graph is input into the improved Graphormer model, and a calibrable station bias encoding mechanism is introduced to perform graph structure modeling processing to generate the updated visual task execution order; the visual task is executed based on the updated visual task execution order.

10. A dynamic scheduling system for visual tasks in multi-station collaboration, comprising executing the dynamic scheduling method for visual tasks in multi-station collaboration as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The perception and acquisition module is used to collect visual data corresponding to multiple workstations and combine it with the pose information of the mobile chassis to generate a multi-workstation visual perception data set. The status construction module is used to construct a multi-workstation operation status description result based on the multi-workstation visual perception data set. The task generation module is used to generate visual tasks corresponding to the work objectives based on the work objectives in the multi-station visual perception data set, thereby forming a visual task set; The scheduling decision module is used to construct a task-workstation relationship graph based on the multi-workstation operation status description results and the visual task set. The task-workstation relationship graph is input into the improved Graphormer model. The improved Graphormer model introduces a calibrable workstation bias encoding mechanism to perform graph structure modeling processing and outputs the visual task execution order. The execution update module is used to control the mobile chassis to navigate and switch between workstations based on the execution order of the vision tasks, and after arriving at the corresponding workstation, control the robotic arm to perform the work actions corresponding to the vision tasks; collect the work execution results of the vision tasks, and update the multi-workstation work status description results based on the work execution results; if the updated multi-workstation work status description results meet the task rearrangement conditions, then generate an updated vision task execution order, and execute the vision tasks according to the updated vision task execution order.

Images