Multi-mechanical-arm flexible production line scheduling method and system based on visual language model

Abstract

The invention discloses a multi-mechanical-arm flexible production line scheduling method and system based on a visual language model, and belongs to the technical field of production line scheduling. The method comprises the steps of obtaining a task target image and a task description text, and converting the task target image and the task description text into a task state joint embedding vector through a visual language model; the task urgency degree, the semantic matching degree between the task state joint embedding vector and the mechanical arm state embedding vector and the future state of the mechanical arm are introduced into a space-time weighted cost function, task allocation is conducted through the space-time weighted cost function, and a mechanical arm task is determined and executed; and the task completion state of the mechanical arm is determined, and the task of the mechanical arm is dynamically adjusted with the minimization of the space-time weighting cost and the feedback cost as the target. The multi-mechanical-arm cooperative work efficiency, the task scheduling precision and the production takt time stability can be improved in the dynamic production environment, and the problem that an existing multi-mechanical-arm flexible production line faces double bottlenecks of efficiency and reliability is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of production line scheduling, and in particular to a multi-robotic arm flexible production line scheduling method and system based on a visual language model. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] In motor stator production, flexible multi-robot production lines have become a crucial tool for achieving efficient manufacturing. However, existing production systems face numerous operational challenges, severely limiting improvements in production line efficiency and adaptability. The entire production process encompasses multiple steps, including winding, insulation, wire embedding, and end forming. Each step requires extremely high precision and efficiency. Even the slightest delay or error can trigger a chain reaction, reducing overall production line efficiency.

[0004] Currently, the operation of robotic arms in flexible production lines relies on pre-set, static scheduling logic. However, this model struggles to cope with complex dynamic scenarios, such as shifting task priorities or unexpected failures. For example, if a robotic arm fails to complete a task on time, subsequent robotic arms may experience a backlog of tasks or become idle, disrupting the production cycle. Summary of the Invention

[0005] In order to address the shortcomings of the existing technology, the present invention provides a multi-robotic arm flexible production line scheduling method, system, electronic device, computer-readable storage medium and computer program product based on a visual language model, which uses the visual language model to provide intelligent semantic understanding, combines global optimization to respond to changes in the production process, and improves the adaptability and efficiency of the production line.

[0006] In a first aspect, the present invention provides a multi-robotic arm flexible production line scheduling method based on a visual language model; A multi-robot flexible production line scheduling method based on a visual language model includes: Obtain the task target image and task description text, and convert them into a task state joint embedding vector through a visual language model; The task urgency, the semantic matching degree between the joint embedding vector of the task state and the embedding vector of the robot state, and the future state of the robot are introduced into the spatiotemporal weighted cost function. The spatiotemporal weighted cost function is used to allocate tasks, determine the robot arm tasks, and execute them. Determine the task completion status of the robotic arm and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

[0007] In some implementations, the task urgency is determined based on the remaining time and current delay of the task.

[0008] In some embodiments, the future state of the robotic arm is determined by processing historical task data and the historical operation trajectory of the robotic arm through an improved time series prediction model.

[0009] In some embodiments, processing historical task data and historical operation trajectories of the robotic arm using the improved time series prediction model includes: The encoder is used to process historical task data and the historical trajectory of the robotic arm to obtain the time series feature vector; The task embedding vector of the current moment is fused with the time series feature vector of the previous moment to form a feature prompter; the feature prompter corresponding to the previous moment is fused with the task embedding vector corresponding to the current moment to form the feature input of the current moment; The feature input at the current moment is processed by the convolutional gated recursive unit to obtain the time series feature output, and the load change and future position are predicted through the detection head.

[0010] In some embodiments, the feedback cost is determined based on the task completion rate of the robotic arm, the execution quality score of the task, and the actual completion time of the task.

[0011] In some implementations, the feedback cost is expressed as: ; Where, Represent the weight parameters of the feedback cost, represents the task completion rate of the robotic arm, represents the execution quality score of the task, represents the execution target score of the task, Indicates the actual completion time of the task. Indicates the estimated completion time of the task.

[0012] In a second aspect, the present invention provides a multi-robotic arm flexible production line scheduling system based on a visual language model; A multi-robot flexible production line scheduling system based on a visual language model, including: The joint embedding module is configured to: obtain the task target image and task description text, and convert the task target image and task description text into a task state joint embedding vector through a visual language model; The task assignment module is configured to: introduce the task urgency, the semantic match between the joint embedding vector of the task state and the embedding vector of the robot state, and the future state of the robot into the spatiotemporal weighted cost function, perform task assignment based on the spatiotemporal weighted cost function, determine the robot arm task, and execute it; The dynamic adjustment module is configured to: determine the task completion status of the robotic arm, and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

[0013] In a third aspect, the present invention provides an electronic device; An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium; A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model.

[0015] In a fifth aspect, the present invention provides a computer program product; A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The technical solution provided by the present invention not only relies on physical state information for task allocation among multiple robotic arms, but also introduces task descriptions and target image information based on visual language models. This task allocation combined with semantic information can effectively avoid task conflicts or resource waste that may exist in traditional methods, and optimize the accuracy of task allocation, especially when the task description and execution target are highly correlated.

[0017] 2. The technical solution provided by the present invention dynamically models task urgency, which helps to prioritize resource allocation to high-priority tasks in a multi-tasking environment and reduce the impact of critical task delays on production line rhythm; introduces a timing prediction model to enhance the system's ability to predict dynamic changes, making task allocation strategies more intelligent and efficient.

[0018] 3. The technical solution provided by this invention integrates the feedback mechanism with the cost function, quantifying the direct impact of feedback on the optimization objective, avoiding the need for separate processing of feedback information, and making the system more responsive. By dynamically adjusting feedback weights through learning, the system can flexibly adjust the scope of feedback influence based on production status, improving optimization effectiveness. The feedback learning mechanism is fully integrated with task allocation logic, forming a closed loop of "allocation-execution-feedback-optimization," enhancing the system's adaptability to dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 A flowchart of a multi-robotic arm flexible production line scheduling method based on a visual language model provided by an embodiment of the present invention; Figure 2 A node diagram of a multi-robotic arm flexible production line scheduling method based on a visual language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0022] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0024] Example 1 Due to the temporal and spatial non-uniformity of task distribution, the robotic arms of the flexible production line in existing motor stator production may face problems such as uneven load, path conflict and rhythm confusion; therefore, this embodiment provides a multi-robotic arm flexible production line scheduling method based on a visual language model, which combines global dynamic optimization and visual language models to improve the collaborative work efficiency of multiple robotic arms, task scheduling accuracy and production rhythm stability in a dynamic production environment.

[0025] Next, combine Figure 1-Figure 2 , a multi-robotic arm flexible production line scheduling method based on a visual language model disclosed in this embodiment is described in detail. The multi-robotic arm flexible production line scheduling method based on a visual language model includes: S1. Obtain the estimated completion time, location coordinates, task target image, and task description text for each task. Use the visual language model to convert the task target image and task description text into a joint task state embedding vector to determine the task set.

[0026] In this embodiment, visual-language information is added to further describe the requirements of the task. By using a visual language model (such as CLIP), the task description text (such as "Please quickly complete the winding work of the motor stator") and the task target image (such as an image of the motor stator) are converted into a joint embedding vector of the task state. Based on this, the task not only has time-space characteristics, but also incorporates semantic descriptions to form richer task features, which can help the system understand the semantics of the task and enhance the intelligence of task allocation.

[0027] For example, the task set Expressed as: ; ; ; Where, Indicates the i tasks, Indicates the i The estimated completion time of each task, Indicates the i The location coordinates of the tasks, , Indicates the i The position of a task along the X-axis in the world space coordinate system, Indicates the i The position of a task along the Y axis in the world space coordinate system, Indicates a task i The position along the Z axis in the world space coordinate system; represents the joint embedding vector of the i-th task state, Represents a visual-language model that maps task target images and task description text to a joint task state embedding vector; represents the task target image, Represents the task description text.

[0028] In this embodiment, the vision-language model may be a pre-trained model such as CLIP, BLIP, or ALIGN.

[0029] S2. Determine the current load, current position coordinates, current movement speed and current state of each robotic arm, and describe the state of each robotic arm based on this.

[0030] For example, jRobotic arms Status Described as: ; Where, Indicates the j The current load of each robot arm, Indicates the j The current position coordinates of the robot arm, Indicates the j The movement speed of the robot arm, Indicates the j The current status of each robot arm is idle / working / faulted).

[0031] Based on this, the status information of multiple robotic arms is aggregated, and the distribution of tasks can be dynamically adjusted according to factors such as the load capacity, position, and speed of the robotic arms.

[0032] S2. The semantic matching degree between the task urgency, the joint embedding vector of the task state and the embedding vector of the robot state, as well as the future state of the robot arm, is introduced into the spatiotemporal weighted cost function. Tasks are allocated through the spatiotemporal weighted cost function, and the robot arm tasks are determined and executed.

[0033] On a flexible production line, the urgency of different tasks may change dynamically over time, for example, a task may be delayed or a critical task may have a higher priority. Therefore, in this embodiment, dynamic modeling of task urgency and the introduction of dynamic task allocation help to prioritize resource allocation to high-priority tasks in a multi-task environment, thereby reducing the impact of critical task delays on the production line rhythm.

[0034] In order to more intelligently match appropriate tasks and robotic arms and reduce resource waste and execution conflicts, in this embodiment, the semantic difference between the task and the robotic arm is calculated to achieve semantic matching between the task and the robotic arm.

[0035] Most existing robot arm task allocations only use the current robot arm state for task allocation. First, there is a lack of dynamic changes in the robot arm, and it is impossible to foresee possible changes in the robot arm state in the future, such as position offset, load adjustment or path conflict, which may cause resource competition and task delays; second, ignoring the correlation between task chains can easily lead to increased waiting time for subsequent tasks, thereby affecting the overall production rhythm; at the same time, it is difficult to respond to emergencies in a dynamic environment in a timely manner, such as emergency task insertion or robot arm failure, which may cause the scheduling strategy to fail; finally, decisions based only on the current state lack support for global optimization, which may result in local optimality rather than overall optimality, making it difficult to meet the requirements of efficient collaboration and flexible scheduling in complex production environments.

[0036] Therefore, in this embodiment, the future state of the robotic arm is introduced into the task allocation decision.

[0037] As an implementation manner, before executing S2, the method further includes: (1) Obtain the remaining time and current delay of the task, and determine the urgency of the task based on the remaining time and current delay of the task.

[0038] For example, the urgency of a task is expressed as: ； Where, Indicates the remaining time of the task. k represents the adjustment parameter, Indicates the planned start time of the i-th task.

[0039] (2) Obtain the historical mission data and historical operation trajectory of the robotic arm, process the historical mission data and historical operation trajectory of the robotic arm through the time series prediction model, and determine the future state of the robotic arm .

[0040] For example, the future state of the robotic arm is represented as: ; Where, represents the predicted load change of the J-th robot arm, represents the predicted position of the J-th robotic arm, Represents historical task data, Indicates the trajectory of the robot arm.

[0041] Furthermore, in order to fuse the task embedding vector with the historical task data and trajectory of the robotic arm, in this embodiment, an innovative cascade modal complementary prompter design is performed to more accurately predict the load changes and future positions of the robotic arm.

[0042] As an implementation method, the specific process of processing historical task data and historical operation trajectories of the robotic arm through the improved time series prediction model is as follows: (201) The encoder processes the historical task data and the historical trajectory of the robot arm to obtain the time series feature vector. Expressed as: ; Where, Represents historical task data, Indicates the historical operation trajectory of the robotic arm.

[0043] In this embodiment, the encoder is a trained Transformer encoder, the input is historical task data and the historical operation trajectory of the robotic arm, and the output is a time series feature vector; here, no improvement is made to the Transformer encoder. During training, the historical task data and the historical operation trajectory of the robotic arm are used to construct a training set and the corresponding time series feature vectors are labeled to train the Transformer encoder.

[0044] In this step, the dynamic change information of the historical task status and the historical operation trajectory of the robot arm is represented by the time series feature vector, including the posture change of the robot arm, the control instruction sequence, the evolution of the task stage, etc., which has time dependence and context modeling capabilities to serve as the task embedding vector at the current moment.

[0045] (202) The task embedding vector at the current moment is fused with the temporal feature vector at the previous moment to form the feature prompter corresponding to the previous moment, which is expressed as: ; Where, Indicates the fusion operation, which can be splicing, weighted summation, etc. In this embodiment, splicing is used; the output is the prompter feature corresponding to the previous moment .

[0046] In this step, the prompter is used to provide the information of the previous moment and the fusion features of the current moment to enhance the network's ability to understand time series.

[0047] (203) The feature prompter corresponding to the previous moment and the task embedding vector corresponding to the current moment are fused to form the feature input of the current moment , expressed as: .

[0048] Based on this, the features of the current moment are enhanced through the prompter information of the previous moment.

[0049] (204) The feature input at the current moment is processed by the convolutional gated recurrent unit (ConvGRU) to obtain the time series feature output, and the load change and future position are predicted through the detection head.

[0050] ConvGRU (Convolutional Gated Recurrent Unit) has strong spatiotemporal modeling capabilities and can effectively learn the relationship between time and space; the temporal feature output Expressed as: ; Where, Output of the time series features at each moment.

[0051] The load change at the future moment is expressed as: ; The position at a future time is expressed as: ; Where, represents a fully connected layer.

[0052] In summary, the above steps improve the accuracy and adaptability of the time series prediction model by fusing the task embedding vector with the robot's historical task data and trajectory through an innovative cascaded modal complementary prompt. By enhancing the current moment's features with prompts from the previous moment and combining them with the spatiotemporal modeling capabilities of the ConvGRU, the network can more accurately predict the robot's payload changes and future position. This approach not only enhances understanding of complex tasks but also allows for flexible adjustments in multi-task environments, avoiding the limitations of traditional models when processing time series data. In this way, the model achieves efficient computation and more stable predictions, making it particularly suitable for dynamic and real-time task prediction.

[0053] (3) The images and texts of the robot arm performing the task are embedded through the visual language model to determine the robot arm state embedding vector.

[0054] Furthermore, the task urgency, the semantic matching between the joint embedding vector of the task state and the embedding vector of the robot state, and the spatiotemporal weighted cost function of the future state of the robot are introduced and expressed as: ; Where, represents the time weight coefficient, represents the spatial weight coefficient, represents the load weight coefficient, represents the semantic matching weight coefficient, Indicates a task Estimated completion time, Indicates the timestamp when the current j-th task starts. It is expressed as a time normalization factor, which is used to normalize the time intervals between different tasks to a uniform scale to avoid the time difference from occupying too high a weight in the task matching cost. Indicates the task location and the position of the robotic arm at the future moment The Euclidean distance of Represents a robotic arm The future load change, Indicates the i The task state joint embedding vector, Indicates the j The joint embedding vector of the task states.

[0055] S3. Determine the task completion status of the robotic arm and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

[0056] In this embodiment, with the goal of minimizing the spatiotemporal weighted cost and feedback cost, an adaptive reinforcement learning algorithm is used to adaptively adjust the optimization strategy according to the dynamic environment and task requirements. The specific process is as follows: S301. Define state, action, and reward functions.

[0057] Specifically, the task state of the manipulator, the task urgency and the current state of the manipulator are jointly embedded into a vector as state S , expressed as: ; Where, Indicates the task location, Indicates load, Indicates the i The task state joint embedding vector, Indicates the urgency of the task. Indicates the estimated completion time of the task.

[0058] action It refers to the adjustments made by the robotic arm during the execution of the task, including speed, path planning, load distribution, etc. The action space is discrete or continuous space and can be designed according to the specific task.

[0059] The reward function Designed as the negative of the spatiotemporal weighted cost and the feedback cost, the goal is to optimize task adjustment by minimizing these costs; the reward function Expressed as:

[0060] S302: Update the strategy through DeepQ-Network (DQN). The model dynamically adjusts the task execution strategy based on the current state and reward value.

[0061] Here, as training progresses, the strategy gradually converges towards minimizing the cost.

[0062] Furthermore, the feedback mechanism is introduced into the core of the optimization goal. By dynamically updating the parameters in the evaluation function, the optimization goal is adjusted in real time according to the feedback information to adapt to changes in the production line status.

[0063] Furthermore, the objective function is expressed as:

[0064] Where, Indicates the task Is it assigned to the robot arm? , represents the time weight coefficient, represents the spatial weight coefficient, represents the load weight coefficient, represents the semantic matching weight coefficient, represents the feedback weight coefficient, Indicates a task Estimated completion time, Indicates the timestamp when the current j-th task starts. It is expressed as a time normalization factor, which is used to normalize the time intervals between different tasks to a uniform scale to avoid the time difference from occupying too high a weight in the task matching cost. Indicates the task location and the position of the robotic arm at the future moment The Euclidean distance of Represents a robotic arm The future load change, represents the joint embedding vector of the i-th task state, represents the j-th task state joint embedding vector, represents the feedback cost term, Indicates the task number, Indicates the robot arm number, Represents the task urgency weight item.

[0065] The feedback cost is expressed as: ; Where, Represent the weight parameters of the feedback cost, Indicates the task completion rate of the robot arm. The lower it is, the worse the performance of the robot arm is, which increases the cost. represents the execution quality score of the task, Indicates the execution target score of the task and the deviation from the target score The degree of reflects the instability of task execution; Indicates the actual completion time of the task and the expected completion time of the task The deviation indicates the degree of task delay; Indicates the estimated completion time of the task.

[0066] The task execution quality score is an abstract indicator used to quantify the degree of match between the actual completion of the task and the expected goal, reflecting the accuracy, stability, and completion of the task execution. It comprehensively considers the accuracy, completeness, and consistency of the task results and numerically describes the task execution quality through predefined evaluation criteria. In some embodiments, the task execution quality score is calculated according to a preset formula, which is expressed as: ; Where, To adjust the parameters and control the sensitivity of score changes; Indicates the deviation of the actual result from the target value.

[0067] If the task quality includes multiple dimensions, a weighted average method can be used to determine the final execution quality score of the task, which can be expressed as: ; Where, represents the weight of the kth evaluation dimension, Represents the task execution quality score corresponding to the kth evaluation dimension.

[0068] Example 2 This embodiment discloses a multi-robot arm flexible production line scheduling system based on a visual language model, including: The joint embedding module is configured to: obtain the task target image and task description text, and convert the task target image and task description text into a task state joint embedding vector through a visual language model; The task assignment module is configured to: introduce the task urgency, the semantic match between the joint embedding vector of the task state and the embedding vector of the robot state, and the future state of the robot into the spatiotemporal weighted cost function, perform task assignment based on the spatiotemporal weighted cost function, determine the robot arm task, and execute it; The dynamic adjustment module is configured to: determine the task completion status of the robotic arm, and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

[0069] It should be noted that the aforementioned joint embedding module, task allocation module, and dynamic adjustment module correspond to the steps in Example 1. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the aforementioned modules, as part of a system, can be executed in a computer system, such as a set of computer-executable instructions.

[0070] Example 3 Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned multi-robotic arm flexible production line scheduling method based on the visual language model are completed.

[0071] Example 4 A fourth embodiment of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the multi-robotic arm flexible production line scheduling method based on the visual language model are completed.

[0072] Example 5 A fifth embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model.

[0073] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0076] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0077] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-robot flexible production line scheduling method based on a visual language model, characterized by: include: Obtain the task target image and task description text, and convert them into a task state joint embedding vector through a visual language model; The task urgency, the semantic matching degree between the joint embedding vector of the task state and the embedding vector of the robot state, and the future state of the robot are introduced into the spatiotemporal weighted cost function. The spatiotemporal weighted cost function is used to allocate tasks, determine the robot arm tasks, and execute them. Determine the task completion status of the robotic arm and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

2. The multi-robot flexible production line scheduling method based on a visual language model according to claim 1, characterized in that: The task urgency is determined based on the remaining time and current delay of the task.

3. The multi-robot flexible production line scheduling method based on a visual language model according to claim 1, characterized in that: The future state of the robotic arm is determined by processing historical task data and the historical operation trajectory of the robotic arm through an improved time series prediction model.

4. The multi-robot flexible production line scheduling method based on a visual language model according to claim 3, characterized in that: The improved time series prediction model is used to process historical task data and the robot's historical operation trajectory, including: The encoder is used to process historical task data and the historical trajectory of the robotic arm to obtain the time series feature vector; The task embedding vector of the current moment is fused with the time series feature vector of the previous moment to form a feature prompter; the feature prompter corresponding to the previous moment is fused with the task embedding vector corresponding to the current moment to form the feature input of the current moment; The feature input at the current moment is processed by the convolutional gated recursive unit to obtain the time series feature output, and the load change and future position are predicted through the detection head.

5. The multi-robot flexible production line scheduling method based on a visual language model according to claim 1, characterized in that: The feedback cost is determined based on the task completion rate of the robot arm, the execution quality score of the task and the actual completion time of the task.

6. The multi-robot flexible production line scheduling method based on a visual language model according to claim 1, characterized in that: The feedback cost is expressed as: ； Where, Represent the weight parameters of the feedback cost, represents the task completion rate of the robotic arm, represents the execution quality score of the task, represents the execution target score of the task, Indicates the actual completion time of the task. Indicates the estimated completion time of the task.

7. A multi-robot flexible production line scheduling system based on a visual language model, characterized by: include: The joint embedding module is configured to: obtain the task target image and task description text, and convert the task target image and task description text into a task state joint embedding vector through a visual language model; The task assignment module is configured to: introduce the task urgency, the semantic match between the joint embedding vector of the task state and the embedding vector of the robot state, and the future state of the robot into the spatiotemporal weighted cost function, perform task assignment based on the spatiotemporal weighted cost function, determine the robot arm task, and execute it; The dynamic adjustment module is configured to: determine the task completion status of the robotic arm, and dynamically adjust the robotic arm task with the goal of minimizing the spatiotemporal weighted cost and feedback cost.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the multi-robotic arm flexible production line scheduling method based on a visual language model according to any one of claims 1 to 6 are implemented.

Images