Intelligent task allocation and priority management system and method for multimodal interactive robotic arm

Abstract

The present invention discloses a multimodal interactive robotic arm intelligent task allocation and priority management system and method. First, through deeply integrated voice and gesture multimodal interaction, real-time and natural operation of the robotic arm grinding task is achieved, which greatly simplifies the operation complexity compared to the traditional single control mode. Secondly, the dynamic priority management mechanism flexibly adjusts the task sequence according to the urgency, importance and real-time feedback of the task, breaking through the limitations of the traditional fixed priority method, ensuring immediate response to high-priority tasks, and improving system adaptability. At the same time, the present invention has an intelligent conflict detection and mediation mechanism, which monitors task dependencies and resource occupancy in real time, automatically adjusts low-priority tasks, avoids delays caused by resource competition, and makes multi-task management smoother. Finally, the system dynamically optimizes the grinding path and force based on real-time feedback to adapt to the precision grinding of complex surfaces and heterogeneous materials, significantly improving the grinding quality and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of robotic arms, and in particular to a multimodal interactive robotic arm intelligent task allocation and priority management system and method. Background Art

[0002] With the advancement of industrial automation, robotic arms are widely used in high-precision machining applications such as automotive, aerospace, and electronics. In particular, robotic arms are often used to achieve high-quality surface finishes due to their precise control and flexibility in polishing complex curved surfaces. However, existing robotic arm polishing systems often struggle to achieve intelligent management in multi-tasking environments, resulting in efficiency and accuracy that fail to meet the requirements of complex machining scenarios. Existing technologies often rely on pre-programmed or manual control, unable to dynamically respond to real-time task changes and multi-task coordination requirements, leading to a series of problems.

[0003] In current polishing scenarios, to meet the polishing needs of different surface areas, the industry generally adopts a pre-set single task approach, that is, assigning fixed polishing tasks and priorities to the robot arm through programming or manual control. However, this approach has many limitations in practical applications:

[0004] 1. Poor operational flexibility and lack of dynamic scheduling capabilities. In complex industrial scenarios, the polishing requirements of different surface areas vary significantly. For example, different surfaces of automobile shells, engine components, or aviation parts often require different adjustments to the polishing force, angle, and speed. Because traditional control methods cannot respond flexibly, they can usually only be controlled by pre-setting a single task priority. Once the task requirements change or new tasks are added, the system is difficult to adjust in time, resulting in complex operations and frequent manual intervention, affecting overall efficiency.

[0005] 2. Lack of intelligent multi-task management can easily lead to task conflicts and resource waste. In multi-task grinding scenarios, multiple grinding tasks in different areas often need to be performed simultaneously. Existing robotic arm systems lack the ability to intelligently manage task priorities and dependencies. They typically execute tasks in a fixed order and lack intelligent resource allocation. Once multiple tasks compete for the same grinding head or resources, the system is prone to task conflicts, resulting in increased waiting times and reduced resource utilization efficiency, affecting overall grinding quality and operational continuity.

[0006] 3. Insufficient feedback and adjustment capabilities make it difficult to meet high-precision polishing requirements. Traditional systems mostly use fixed paths and forces to complete polishing, lacking dynamic adjustment functions based on surface quality. When polishing complex surfaces or heterogeneous materials, fixed force and path settings can easily lead to uneven surface quality, making it difficult to meet the requirements of precision machining. In scenarios requiring higher polishing precision, such as aerospace and high-end manufacturing, existing systems are unable to meet the polishing quality of different materials and shapes through real-time adjustments, and cannot guarantee product consistency.

[0007] To address these issues, the industry has experimented with improvements, such as allocating more resources or resolving task conflicts through human assistance. However, these approaches increase operational costs and labor burden, and still fail to fundamentally address the system's inadequate intelligent scheduling and feedback control. Therefore, intelligently implementing multi-task allocation and priority management, dynamic conflict resolution, and adaptive control based on real-time feedback have become core challenges that currently require solutions.

[0008] Therefore, how to achieve intelligent multi-task allocation and priority management, dynamic conflict mediation, and adaptive control based on real-time feedback has become a technical problem to be solved by the present invention. Summary of the Invention

[0009] The technical problem solved by the present invention is to address the defects existing in the above-mentioned prior art and provide a multimodal interactive robotic arm intelligent task allocation and priority management system and method to solve the problems of insufficient multi-task scheduling flexibility, frequent task conflicts, and low feedback adjustment capability raised in the above-mentioned background technology.

[0010] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0011] A multimodal interactive robotic arm intelligent task allocation and priority management system, comprising:

[0012] The robotic arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure precise grinding on different materials and complex surfaces.

[0013] A multimodal input device for receiving user input of voice commands and gestures, the multimodal input device comprising:

[0014] A microphone is used to receive user voice commands to create, schedule, and manage tasks;

[0015] The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm;

[0016] a multimodal interaction algorithm module, configured to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm;

[0017] The task management unit is used to create, adjust, and manage polishing tasks, including task priority setting, conflict detection, and task scheduling. The task management unit includes:

[0018] The task creation module is used to receive voice or gesture input to generate new polishing tasks;

[0019] Priority management module, used to dynamically adjust the priority of tasks based on their urgency, importance and dependencies;

[0020] Task scheduling module, used to dynamically adjust the order of task execution based on priority management results;

[0021] A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes:

[0022] Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time;

[0023] A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur;

[0024] Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

[0025] As a further solution of the present invention, the multimodal interaction algorithm module uses natural language processing (NLP) and deep learning technology to perform speech and gesture recognition, which is achieved through convolutional neural networks (CNN) and long short-term memory networks (LSTM), wherein:

[0026] The recognition of voice commands is based on the following model formula:

[0027]

[0028] in, Represents a given input When the output probability; is the hidden state obtained by LSTM; is the weight matrix;

[0029] Noise filtering is achieved through short-time Fourier transform STFT, the formula is:

[0030]

[0031] in is the time-frequency diagram, is the input signal, is the window function.

[0032] As a further solution of the present invention, the multimodal input device includes a microphone with a noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

[0033] As a further solution of the present invention, the multimodal interaction algorithm module supports user-defined voice and gesture instructions to enhance the flexibility and adaptability of task allocation and priority management.

[0034] As a further solution of the present invention, the following steps are included:

[0035] Step 1, multimodal input reception: receiving a user's voice commands and gestures through a multimodal input device, wherein the multimodal input device includes a microphone and a camera for receiving voice and capturing gestures, respectively;

[0036] Step 2, multimodal input parsing: Use the multimodal interaction algorithm module to parse the voice and gesture input, including:

[0037] Use natural language processing technology to parse voice commands and identify semantic information;

[0038] Use deep learning models to identify gesture image features and convert the parsed voice and gesture inputs into robotic arm operation commands;

[0039] Step 3, Task Management: Create, adjust, and manage polishing tasks in the task management unit according to the parsed user instructions, including the following steps:

[0040] Assign an initial priority to each task based on its urgency, importance, and dependencies;

[0041] During task execution, task priorities are dynamically adjusted based on task conditions and real-time feedback;

[0042] Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks;

[0043] Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource usage in the task queue are monitored in real time, including the following steps:

[0044] Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task will be executed first;

[0045] Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts;

[0046] Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts;

[0047] Step 5, Task execution and feedback adjustment: Control the robotic arm to perform the polishing task based on the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on the feedback to ensure polishing accuracy and quality.

[0048] As a further solution of the present invention, the following steps are further included:

[0049] User-defined command input: Users set custom voice and gesture commands through multimodal input devices. The custom commands are used to adjust task priorities, create or cancel tasks;

[0050] Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands;

[0051] Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. At the same time, mediate or postpone the execution of low-priority tasks to enhance the flexibility and adaptability of the system.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. Intelligent Dynamic Task Priority Management: This system features real-time dynamic task priority adjustment, automatically optimizing priority settings based on factors such as task urgency, importance, and dependencies. Compared to traditional fixed-priority or manual scheduling systems, this system is more adaptable to changing tasks and multitasking, ensuring timely completion of high-priority tasks, thereby improving task execution efficiency and system responsiveness.

[0054] 2. Efficient conflict detection and real-time mediation: In a multi-tasking environment, this system uses a priority decision unit to monitor task dependencies and resource usage in real time. When resource conflicts arise, it automatically mediates, prioritizing high-priority tasks and deferring low-priority ones. This feature effectively avoids task delays and resource waste caused by resource contention in traditional systems, ensuring smoother and more consistent polishing execution, and ensuring polishing quality and overall production efficiency.

[0055] 3. Natural and efficient multimodal human-machine interaction: This system combines multimodal interactive control with voice and gesture input, enabling users to efficiently manage robotic arm polishing tasks through natural voice commands and intuitive gestures. This natural interaction not only reduces the need for complex programming but also significantly improves user convenience and flexibility, making it particularly suitable for industrial automation environments.

[0056] 4. Real-time adaptive grinding path and force control: Based on real-time feedback, the system dynamically adjusts the robot arm's grinding path and force to meet the grinding requirements of varying surface qualities, achieving precise grinding results even on complex curved surfaces and heterogeneous materials. This adaptive control capability, unprecedented in existing technologies, significantly improves the accuracy and efficiency of the grinding process, making it suitable for high-precision industrial applications such as aerospace and automotive manufacturing.

[0057] 5. Support for user-defined commands, enhancing system adaptability: The system supports user-defined voice and gesture commands and automatically interprets these commands in real-time task management, further enhancing the system's adaptability in diverse scenarios. This flexible task allocation and priority management allows users to flexibly adjust polishing tasks according to actual needs, achieving more personalized and diverse operation plans and expanding the system's application scenarios.

[0058] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 This is a structural diagram of the multimodal interactive robotic arm intelligent task allocation and priority management system of the present invention. DETAILED DESCRIPTION

[0061] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0062] See also Figure 1 In an embodiment of the present invention, a multimodal interactive robotic arm intelligent task allocation and priority management system includes:

[0063] The robotic arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure precise grinding on different materials and complex surfaces.

[0064] A multimodal input device for receiving user input of voice commands and gestures, the multimodal input device comprising:

[0065] A microphone is used to receive user voice commands to create, schedule, and manage tasks;

[0066] The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm;

[0067] a multimodal interaction algorithm module, configured to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm;

[0068] The task management unit is used to create, adjust, and manage polishing tasks, including task priority setting, conflict detection, and task scheduling. The task management unit includes:

[0069] The task creation module is used to receive voice or gesture input to generate new polishing tasks;

[0070] Priority management module, used to dynamically adjust the priority of tasks based on their urgency, importance and dependencies;

[0071] Task scheduling module, used to dynamically adjust the order of task execution based on priority management results;

[0072] A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes:

[0073] Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time;

[0074] A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur;

[0075] Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

[0076] As a further solution of the present invention, the multimodal interaction algorithm module uses natural language processing (NLP) and deep learning technology to perform speech and gesture recognition, which is achieved through convolutional neural networks (CNN) and long short-term memory networks (LSTM), wherein:

[0077] The recognition of voice commands is based on the following model formula:

[0078]

[0079] in, Represents a given input When the output probability; is the hidden state obtained by LSTM; is the weight matrix;

[0080] The parsing process is as follows:

[0081] 1. Noise filtering: Noise filtering is achieved through short-time Fourier transform (STFT), and the formula is:

[0082]

[0083] in is the time-frequency diagram, is the input signal, is the window function.

[0084] 2. Feature extraction, using Mel-frequency cepstral coefficients (MFCC) to extract features.

[0085]

[0086] 3. Model parsing: Input the features into the speech recognition model for parsing.

[0087] Example code (Python):

[0088] # Read audio file and extract MFCC features

[0089] def extract_features(file_path):

[0090] y,sr = librosa.load(file_path, sr=None)

[0091] mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13)

[0092] return np.mean(mfcc.T, axis=0)

[0093] Speech Recognition

[0094] def recognize_command(audio_file, model):

[0095] features = extract_features(audio_file)

[0096] features = features.reshape(1, -1) # Adjust the shape

[0097] predicted_label = model.predict(features)

[0098] return predicted_label

[0099] # Load the model

[0100] model = load_model('speech_recognition_model.h5')

[0101] command = recognize_command('user_command.wav', model)

[0102] As a further solution of the present invention, the priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task. The priority calculation formula is:

[0103]

[0104] in, and is the weight coefficient, which is dynamically adjusted according to task requirements.

[0105] As a further solution of the present invention, the task scheduling module sorts the task queues according to priority, and the task scheduling formula is:

[0106]

[0107] T represents the task queue, and Sort represents sorting by priority.

[0108] As a further solution of the present invention, the conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module handles task conflicts based on a priority adjustment algorithm and postpones the execution time of the low-priority task by tlowpriority. The calculation formula is:

[0109]

[0110] Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

[0111] As a further solution of the present invention, the task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback. The control strategy formula is:

[0112]

[0113] Among them F打磨 is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

[0114] Furthermore, the gesture recognition module is used to convert the user's gesture movements into operating instructions for the robotic arm, especially for adjusting task parameters.

[0115] This technology is implemented through image processing and gesture recognition using deep learning models (such as ResNet or YOLO). This module uses a convolutional neural network (CNN) to process captured hand images and uses a posture estimation algorithm to identify the shape, direction, trajectory, and other information of the gesture.

[0116] The formula is:

[0117]

[0118] in, Represents a given image When the class probability; is the feature output by the CNN model.

[0119] The system pre-defines a library of common gestures (such as "left", "right", "raise priority", "cancel task", etc.), and users can also expand the gesture library by training new gestures.

[0120] The parsing process is as follows:

[0121] Image preprocessing: De-noising and edge detection are performed on the images captured by the camera.

[0122] Feature extraction: Use CNN to extract features from the hand area.

[0123] def preprocess_image(image):

[0124] # Preprocess the image

[0125] image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

[0126] image = cv2.GaussianBlur(image, (5, 5), 0)

[0127] return image

[0128] Gesture recognition uses a pose estimation algorithm to identify gestures and match them with a gesture library.

[0129] Example code:

[0130] Gesture Recognition

[0131] def recognize_gesture(image, model):

[0132] processed_image = preprocess_image(image)

[0133] # Assume the input is a 64x64 grayscale image

[0134] features = model.predict(processed_image.reshape(1, 64, 64, 1))

[0135] return np.argmax(features)

[0136] # Load the gesture recognition model

[0137] gesture_model = load_model('gesture_recognition_model.h5')

[0138] frame = cv2.imread('hand_gesture.jpg')

[0139] gesture = recognize_gesture(frame, gesture_model)

[0140] The task management unit is responsible for managing and scheduling all polishing tasks, including task creation, priority adjustment, conflict detection, and scheduling. Its functional module design ensures that the system operates efficiently in a multi-tasking environment.

[0141] 1. Task creation module.

[0142] Responsible for receiving user voice or gesture input, creating new polishing tasks and assigning initial priorities.

[0143] formula:

[0144]

[0145] in, Indicates a task Priority, Indicates task requirements, Indicates the task type.

[0146] 2. Priority management module.

[0147] Dynamically adjust task priorities based on factors such as the importance of the polishing area, polishing time, and task dependencies.

[0148] formula:

[0149]

[0150] in, Indicates the adjusted priority, is the degree of task dependency, is the urgency of the task, and α, β, and γ are weight coefficients.

[0151] 3. Task Scheduling Module

[0152] Based on the priority management results and real-time polishing status, the execution order of polishing tasks is dynamically scheduled.

[0153] formula:

[0154]

[0155] Among them, T is the task queue, and Sort means sorting by priority.

[0156] Example code:

[0157] def create_task(self, description, initial_priority):

[0158] new_task = Task(description, initial_priority)

[0159] self.tasks.append(new_task)

[0160] print(f"Create task: {description}, priority: {initial_priority}")

[0161] def adjust_priority(self, task, dependency_degree, urgency_level):

[0162] # Dynamically adjust task priority

[0163] task.priority = (0.5 * task.priority) + (0.3 * dependency_degree) + (0.2 * urgency_level)

[0164] print(f"Adjust task: {task.description}'s priority is: {task.priority}")

[0165] def schedule_tasks(self):

[0166] # Sort tasks by priority

[0167] self.tasks.sort(key=lambda x: x.priority, reverse=True)

[0168] return [task.description for task in self.tasks]

[0169] The priority decision unit is responsible for dynamically adjusting task priorities according to the urgency, dependencies and real-time feedback information of tasks during the execution of multiple tasks, and performing intelligent mediation when task conflicts occur to ensure stable and efficient operation of the system.

[0170] 1. Conflict Detection Module

[0171] Monitor the task dependencies and resource usage in the polishing task queue in real time to detect whether there are task conflicts, such as two tasks requiring the same polishing head at the same time.

[0172] 2. Real-time mediation module

[0173] When task conflicts are detected, the system automatically adjusts based on task priority and real-time work status, such as postponing low-priority tasks and requesting user confirmation when necessary.

[0174] formula:

[0175]

[0176] Indicates the adjusted task The start time, Indicates the original task start time, To postpone the time, the postponement time is dynamically calculated based on the task priority and conflict degree.

[0177] 3. Decision-making feedback mechanism

[0178] The system provides real-time feedback of conflict detection results and mediation decisions through the user interface, and users can choose to continue, cancel, or modify the current task settings.

[0179] formula:

[0180]

[0181] Indicates the user's task The final decision, Indicates the system’s feedback information on the task. Indicates the user's priority judgment.

[0182] Sample code:

[0183] def detect_conflicts(self):

[0184] # Conflict detection logic

[0185] for i in range(len(self.tasks)):

[0186] for j in range(i + 1, len(self.tasks)):

[0187] if self.tasks[i].resource == self.tasks[j].resource:

[0188] if self.tasks[i].start_time < self.tasks[j].end_time:

[0189] self.conflicts.append((self.tasks[i], self.tasks[j]))

[0190] print(f"Task conflict detected: {self.tasks[i].description} and {self.tasks[j].description}")

[0191] def resolve_conflicts(self):

[0192] # Conflict mediation logic

[0193] for task_i, task_j in self.conflicts:

[0194] if task_i.priority > task_j.priority:

[0195] task_j.start_time += 1 # Delay low priority tasks

[0196] print(f"Postpone task: {task_j.description} start time to {task_j.start_time}")

[0197] else:

[0198] task_i.start_time += 1

[0199] print(f"Postpone task: {task_i.description} start time to {task_i.start_time}")

[0200] def feedback_decision(self):

[0201] # Decision feedback logic

[0202] for task_i, task_j in self.conflicts:

[0203] print(f"User decision: Tasks {task_i.description} and {task_j.description} conflict. Please choose to continue, cancel, or modify task settings.")

[0204] As a further solution of the present invention, the priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task. The priority calculation formula is:

[0205]

[0206] in, and is the weight coefficient, which is dynamically adjusted according to task requirements.

[0207] As a further solution of the present invention, the task scheduling module sorts the task queues according to priority, and the task scheduling formula is:

[0208]

[0209] T represents the task queue, and Sort represents sorting by priority.

[0210] As a further solution of the present invention, the conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module handles task conflicts based on a priority adjustment algorithm and postpones the execution time of the low-priority task by tlowpriority. The calculation formula is:

[0211]

[0212] Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

[0213] As a further solution of the present invention, the task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback. The control strategy formula is:

[0214]

[0215] Among them F 打磨 is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

[0216] As a further solution of the present invention, the multimodal input device includes a microphone with a noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

[0217] As a further solution of the present invention, the multimodal interaction algorithm module supports user-defined voice and gesture instructions to enhance the flexibility and adaptability of task allocation and priority management.

[0218] As a further solution of the present invention, the following steps are included:

[0219] Step 1, multimodal input reception: receiving a user's voice commands and gestures through a multimodal input device, wherein the multimodal input device includes a microphone and a camera for receiving voice and capturing gestures, respectively;

[0220] Step 2, multimodal input parsing: Use the multimodal interaction algorithm module to parse the voice and gesture input, including:

[0221] Use natural language processing technology to parse voice commands and identify semantic information;

[0222] Use deep learning models to identify gesture image features and convert the parsed voice and gesture inputs into robotic arm operation commands;

[0223] Step 3, Task Management: Create, adjust, and manage polishing tasks in the task management unit according to the parsed user instructions, including the following steps:

[0224] Assign an initial priority to each task based on its urgency, importance, and dependencies;

[0225] During task execution, task priorities are dynamically adjusted based on task conditions and real-time feedback;

[0226] Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks;

[0227] Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource usage in the task queue are monitored in real time, including the following steps:

[0228] Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task will be executed first;

[0229] Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts;

[0230] Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts;

[0231] Step 5, Task execution and feedback adjustment: Control the robotic arm to perform the polishing task based on the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on the feedback to ensure polishing accuracy and quality.

[0232] As a further solution of the present invention, the following steps are further included:

[0233] User-defined command input: Users set custom voice and gesture commands through multimodal input devices. The custom commands are used to adjust task priorities, create or cancel tasks;

[0234] Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands;

[0235] Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. At the same time, mediate or postpone the execution of low-priority tasks to enhance the flexibility and adaptability of the system.

[0236] In the polishing of automobile shells, different surface areas (such as doors, roofs, contour edges, etc.) have significantly different requirements for polishing accuracy and force, and the polishing path and angle need to be flexibly adjusted. These complex polishing tasks often involve multiple tasks being carried out simultaneously, requiring the robotic arm to be able to intelligently allocate tasks, adjust priorities in real time, resolve task conflicts, and dynamically optimize the polishing process based on feedback information. However, due to the limitations of fixed programs and manual control, traditional systems are difficult to achieve ideal results in high-demand multi-tasking environments. Therefore, how to achieve intelligent multi-task scheduling, efficiently resolve resource conflicts, and ensure polishing accuracy and efficiency has become an urgent need in the industry. The present invention provides a multimodal interactive robotic arm intelligent task allocation and priority management system, which achieves high efficiency and high precision of polishing operations in the above-mentioned scenarios through the following specific steps and technical solutions.

[0237] First, the system uses multimodal input devices to capture user voice commands and gesture input. In practice, operators can directly specify polishing tasks for different areas through voice, such as "polishing the exterior door surfaces" or "prioritizing the roof polishing task." They can also use gestures to fine-tune the robotic arm's polishing angle, ensuring the polishing direction and posture meet specific surface requirements. This multimodal interaction eliminates the need for complex programming, enabling operators to create and adjust tasks more intuitively and efficiently during operation, flexibly adapting to changing polishing scenarios.

[0238] In a multi-tasking environment, the system's task management unit automatically assigns an initial priority based on the task type and urgency entered by the operator. For example, door polishing may be given a higher priority due to its surface characteristics and importance, and the system automatically sorts it at the front of the task queue to ensure timely execution. Furthermore, when the operator issues new instructions or task requirements change, the priority management module can readjust priorities during task execution based on real-time conditions to adapt to the new task requirements. Unlike existing technologies, this system dynamically manages task priorities through intelligent algorithms, avoiding task delays and frequent manual intervention caused by rigid task priorities, making the system adaptive.

[0239] In addition, in actual polishing scenarios, it is common for multiple tasks to compete for the same polishing head, resulting in task conflicts and affecting overall efficiency. This system monitors task dependencies and resource allocation in real time through the conflict detection module of the priority decision unit. When it detects that two tasks are competing for the same resource, the system will automatically trigger the real-time mediation mechanism to appropriately postpone or suspend the low-priority task. For example, when the polishing tasks of the car door and the roof need to occupy the same polishing head at the same time, the system will automatically give priority to the door polishing task and postpone the roof task to ensure that the high-priority task can be polished unimpeded. This conflict detection and mediation mechanism effectively avoids task delays or task interruptions caused by resource competition in traditional systems, realizes efficient resource utilization and smooth task execution, and ensures the continuity of operations in a multi-tasking environment.

[0240] During the polishing process, this system continuously optimizes the polishing force and path based on real-time feedback to ensure the polishing quality. For example, the complex curves and contours of the roof and edges require fine processing under specific polishing forces. The force control sensor detects the feedback data of the polishing surface in real time, such as surface finish, torque changes, etc. When it is detected that the polishing finish of the door surface is lower than expected, the system will automatically adjust the polishing force and path, and optimize the polishing effect by increasing or decreasing the polishing force. Unlike existing systems, this system uses real-time feedback to adaptively adjust the polishing parameters, and can dynamically adapt to the precision requirements of different materials and complex surfaces, thereby ensuring consistent surface quality across the entire polishing area. This feedback-based polishing optimization not only improves the polishing accuracy of the robotic arm, but also meets the high standards required for complex surfaces and heterogeneous materials in high-precision industrial scenarios.

[0241] The system also supports operator-defined commands. For example, operators can set commands like "reduce door polishing intensity" or "increase roof priority" through custom gestures or voice commands. These commands are recognized by a multimodal interaction algorithm, allowing the system to flexibly adjust task parameters based on the user's actual needs. This customization not only enhances the system's adaptability, enabling it to maintain efficient operation despite changing production demands, but also avoids the problem of fixed program control being unable to cope with complex on-site requirements.

[0242] In summary, this embodiment demonstrates how the multimodal interactive robotic arm intelligent task allocation and priority management system of the present invention can achieve intelligent, multi-task management in complex polishing scenarios. Through natural voice and gesture interaction, intelligent dynamic adjustment of priorities, real-time conflict detection and mediation, and feedback-driven adaptive optimization of polishing paths and forces, the system significantly improves operational flexibility, resource utilization efficiency, and polishing accuracy, and comprehensively solves the technical problems of insufficient multi-task management, frequent task conflicts, and low feedback adjustment capabilities in the prior art. The application of the present invention not only reduces manual intervention in polishing operations, but also achieves high-quality polishing effects under high-precision and high-efficiency requirements.

[0243] In the present invention, unless otherwise clearly specified or limited, terms such as "installation", "setting", "connection", "fixation", and "screw-on" should be understood in a broad sense. For example, they can be fixedly connected, detachably connected, or integrated; they can be mechanically connected or electrically connected; they can be directly connected or indirectly connected through an intermediate medium; they can be internal communication between two elements or an interactive relationship between two elements. Unless otherwise clearly specified or limited, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to specific circumstances.

[0244] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be embraced therein.

Claims

1. A multimodal interactive robotic arm intelligent task allocation and priority management system, characterized in that: include: The robotic arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure precise grinding on different materials and complex surfaces. A multimodal input device for receiving user input of voice commands and gestures, the multimodal input device comprising: A microphone is used to receive user voice commands to create, schedule, and manage tasks; The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm; a multimodal interaction algorithm module, configured to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm; The task management unit is used to create, adjust, and manage polishing tasks, including task priority setting, conflict detection, and task scheduling. The task management unit includes: The task creation module is used to receive voice or gesture input to generate new polishing tasks; Priority management module, used to dynamically adjust the priority of tasks based on their urgency, importance and dependencies; Task scheduling module, used to dynamically adjust the order of task execution based on priority management results; A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes: Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time; A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur; Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

2. A multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal interaction algorithm module uses natural language processing (NLP) and deep learning technology for speech and gesture recognition, which is achieved through convolutional neural networks (CNN) and long short-term memory (LSTM) networks. The recognition of voice commands is based on the following model formula: in, Represents a given input When probability; is the hidden state obtained by LSTM; is the weight matrix; Noise filtering is achieved through short-time Fourier transform STFT, the formula is: in is the time-frequency diagram, is the input signal, is the window function.

3. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task. The priority calculation formula is: in, and is the weight coefficient, which is dynamically adjusted according to task requirements.

4. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The task scheduling module sorts the task queues according to priority. The task scheduling formula is: T represents the task queue, and Sort represents sorting by priority.

5. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module processes task conflicts based on a priority adjustment algorithm and postpones the execution time of the low-priority task by tlowpriority. The calculation formula is: Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

6. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback. The control strategy formula is: Among them F 打磨 is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

7. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal input device includes a microphone with noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

8. The multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal interaction algorithm module supports user-defined voice and gesture commands to enhance the flexibility and adaptability of task allocation and priority management.

9. The management method of a multimodal interactive robotic arm intelligent task allocation and priority management system according to claim 1, characterized in that: The following steps are involved: Step 1, multimodal input reception: receiving a user's voice commands and gestures through a multimodal input device, wherein the multimodal input device includes a microphone and a camera for receiving voice and capturing gestures, respectively; Step 2, multimodal input parsing: Use the multimodal interaction algorithm module to parse the voice and gesture input, including: Use natural language processing technology to parse voice commands and identify semantic information; Use deep learning models to identify gesture image features and convert the parsed voice and gesture inputs into robotic arm operation commands; Step 3, Task Management: Create, adjust, and manage polishing tasks in the task management unit according to the parsed user instructions, including the following steps: Assign an initial priority to each task based on its urgency, importance, and dependencies; During task execution, task priorities are dynamically adjusted based on task conditions and real-time feedback; Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks; Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource usage in the task queue are monitored in real time, including the following steps: Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task will be executed first; Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts; Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts; Step 5, Task execution and feedback adjustment: Control the robotic arm to perform the polishing task based on the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on the feedback to ensure polishing accuracy and quality.

10. The method according to claim 9, characterized in that Further comprising the steps of: User-defined command input: Users set custom voice and gesture commands through multimodal input devices. The custom commands are used to adjust task priorities, create or cancel tasks; Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands; Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. At the same time, mediate or postpone the execution of low-priority tasks to enhance the flexibility and adaptability of the system.

Images