Intelligent task allocation and priority management system and method for multi-mode interactive mechanical arm

Abstract

The invention discloses an intelligent task allocation and priority management system and method for a multi-modal interaction mechanical arm, firstly, real-time and natural operation of a polishing task of the mechanical arm is realized through deep integrated voice and gesture multi-modal interaction, and compared with a traditional single control mode, the operation complexity is greatly simplified; secondly, the dynamic priority management mechanism flexibly adjusts the task sequence according to the urgency, importance and real-time feedback of the tasks, the limitation of a traditional fixed priority mode is broken through, instant response of the high-priority tasks is ensured, and the adaptability of the system is improved. Meanwhile, the method has an intelligent conflict detection and mediation mechanism, monitors task dependence and resource occupation conditions in real time, automatically adjusts low-priority tasks, avoids delay caused by resource competition, and enables multi-task management to be smoother. And finally, the system dynamically optimizes the polishing path and force on the basis of real-time feedback so as to adapt to precise polishing of complex curved surfaces and heterogeneous materials, and the polishing quality and efficiency are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of robotic arms, and in particular to a multi-modal interactive robotic arm intelligent task allocation and priority management system and method. Background Art

[0002] With the development of industrial automation, robotic arms have been widely used in high-precision processing fields such as automobile manufacturing, aerospace, and electronic products. Especially in the grinding operation of complex surfaces, robotic arms are often used to achieve high-quality surface treatment due to their precise control and flexibility. However, existing robotic arm grinding systems are often difficult to achieve intelligent management in a multi-task environment, resulting in efficiency and accuracy that cannot meet the requirements of complex processing scenarios. Existing technologies usually rely on pre-programming or manual control and cannot dynamically respond to real-time task changes and multi-task coordination requirements, resulting in a series of problems.

[0003] In the current polishing scenario, in order to meet the polishing needs of different surface areas, the industry usually adopts the method of pre-setting a single task, that is, assigning fixed polishing tasks and priorities to the robot arm through programming or manual control. However, this method has many limitations in practical applications: 1. Poor operational flexibility and lack of dynamic scheduling capabilities. In complex industrial scenarios, the polishing requirements of different surface areas vary significantly. For example, different surfaces of automobile shells, engine components or aviation parts often require different polishing forces, angles and speeds. Because traditional control methods cannot respond flexibly, they can usually only be controlled by pre-setting a single task priority. Once the task requirements change or new tasks are added, it is difficult for the system to adjust in time, resulting in complex operations and frequent manual intervention, affecting overall efficiency.

[0004] 2. Lack of intelligence in multi-task management can easily lead to task conflicts and waste of resources. In multi-task grinding scenarios, it is often necessary to perform grinding tasks in multiple different areas at the same time. The existing robotic arm system lacks the ability to intelligently manage task priorities and dependencies. It usually executes tasks in a fixed order and lacks intelligent allocation of resources. Once multiple tasks compete for the same grinding head or resources, the system is prone to task conflicts, resulting in increased waiting time and reduced resource utilization efficiency, affecting the overall grinding quality and operation continuity.

[0005] 3. Insufficient feedback and adjustment capabilities make it difficult to meet high-precision grinding requirements. Traditional systems mostly use fixed paths and forces to complete grinding, and lack dynamic adjustment functions based on surface quality. When grinding complex surfaces or heterogeneous materials, fixed force and path settings can easily lead to uneven surface quality, making it difficult to meet the requirements of precision machining. In scenarios that require higher grinding accuracy, such as aerospace and high-end manufacturing, existing systems are difficult to meet the grinding quality of different materials and shapes through real-time adjustments, and cannot guarantee product consistency.

[0006] To address the above problems, the industry has also tried some improvement methods, such as allocating more resources or manually assisting to resolve task conflicts, but these methods increase operating costs and labor burden, and still fail to fundamentally solve the problem of insufficient intelligent scheduling and feedback control of the system. Therefore, how to achieve multi-task allocation and priority management, dynamic conflict mediation, and adaptive control based on real-time feedback in an intelligent way has become a core problem that the existing technology urgently needs to solve.

[0007] Therefore, how to realize intelligent multi-task allocation and priority management, dynamic conflict mediation, and adaptive control based on real-time feedback has become a technical problem to be solved by the present invention. Summary of the invention

[0008] The technical problem solved by the present invention is to provide a multimodal interactive robotic arm intelligent task allocation and priority management system and method in response to the defects existing in the above-mentioned prior art, so as to solve the problems of insufficient flexibility in multi-task scheduling, frequent task conflicts, and low feedback adjustment capability raised in the above-mentioned background technology.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: A multi-modal interactive robot arm intelligent task allocation and priority management system, comprising: The robot arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure accurate grinding on different materials and complex surfaces. A multimodal input device, used to receive voice commands and gesture actions input by a user, the multimodal input device comprising: A microphone for receiving user voice commands to create, schedule, and manage tasks; The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm; A multimodal interaction algorithm module, used to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm; The task management unit is used to create, adjust and manage polishing tasks, including task priority setting, conflict detection and task scheduling, wherein the task management unit includes: The task creation module is used to receive voice or gesture input to generate new polishing tasks; Priority management module, used to dynamically adjust the priority of tasks according to their urgency, importance and dependencies; Task scheduling module, used to dynamically adjust the task execution order based on priority management results; A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes: Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time; A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur; Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

[0010] As a further solution of the present invention, the multimodal interaction algorithm module uses natural language processing NLP and deep learning technology to perform speech and gesture recognition, which is implemented through convolutional neural network CNN and long short-term memory network LSTM, wherein: The recognition of voice commands is based on the following model formula: in, Represents a given input When probability; is the hidden state obtained by LSTM; is the weight matrix; Noise filtering is achieved through short-time Fourier transform STFT, the formula is: in is the time-frequency diagram, is the input signal, is the window function.

[0011] As a further solution of the present invention, the multimodal input device includes a microphone with a noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

[0012] As a further solution of the present invention, the multimodal interaction algorithm module supports user-defined voice and gesture instructions to enhance the flexibility and adaptability of task allocation and priority management.

[0013] As a further solution of the present invention, the following steps are included: Step 1, multimodal input reception: receiving a user's voice command and gesture action through a multimodal input device, wherein the multimodal input device includes a microphone and a camera, which are used to receive voice and capture gesture actions respectively; Step 2: Multimodal input analysis: Use the multimodal interaction algorithm module to analyze the voice and gesture input, including: Use natural language processing technology to parse voice commands and identify semantic information; Use deep learning models to identify gesture image features and convert parsed voice and gesture inputs into robot arm operation commands; Step 3, task management: Create, adjust and manage polishing tasks according to the parsed user instructions in the task management unit, including the following steps: Assign an initial priority to each task based on its urgency, importance, and dependencies; During task execution, dynamically adjust task priorities based on task conditions and real-time feedback; Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks; Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource occupancy in the task queue are monitored in real time, including the following steps: Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task is executed first; Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts; Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts; Step 5, task execution and feedback adjustment: Control the robotic arm to perform the polishing task according to the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on feedback to ensure polishing accuracy and quality.

[0014] As a further solution of the present invention, the following steps are further included: User-defined command input: Users set customized voice and gesture commands through multimodal input devices. The customized commands are used to adjust the priority of tasks, create or cancel tasks; Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands; Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. Meanwhile, mediate or postpone low-priority tasks to enhance system flexibility and adaptability.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Intelligent dynamic management of task priorities: This system has the ability to dynamically adjust task priorities in real time, and can automatically optimize priority settings based on factors such as the urgency, importance, and dependencies of tasks. Compared with traditional fixed-priority or manual scheduling systems, this system is more adaptive under task changes and multi-tasking conditions, ensuring that high-priority tasks are completed in a timely manner, thereby improving task execution efficiency and system response speed.

[0016] 2. Efficient conflict detection and real-time mediation mechanism: In a multi-task environment, the system monitors task dependencies and resource usage in real time through a priority decision unit. When encountering resource conflicts, it can automatically mediate, giving priority to high-priority tasks and postponing low-priority tasks. This function effectively avoids task delays and resource waste caused by resource contention in traditional systems, making the execution of polishing tasks smoother and more coherent, ensuring polishing quality and overall production efficiency.

[0017] 3. Realize natural and efficient multi-modal human-computer interaction: This system combines multi-modal interactive control of voice and gesture input, allowing users to efficiently manage robotic arm grinding tasks through natural voice commands and intuitive gestures. This natural interaction not only reduces complex programming requirements, but also significantly improves the user's operational convenience and flexibility, which is particularly applicable in industrial automation environments.

[0018] 4. Real-time adaptive grinding path and force control: Based on real-time feedback, the system can dynamically adjust the grinding path and force of the robot arm to meet the grinding requirements of different surface qualities, thereby achieving precise grinding effects in the grinding of complex curved surfaces and heterogeneous materials. This adaptive control capability is not seen in the existing technology, which significantly improves the accuracy and efficiency of the grinding process, making it suitable for high-precision industrial application scenarios, such as aerospace and automotive manufacturing.

[0019] 5. Support user-defined commands and enhance system adaptability: The system supports user-defined voice and gesture commands, and automatically parses these commands in real-time task management, further enhancing the system's adaptability in diverse scenarios. This flexible task allocation and priority management method allows users to flexibly adjust polishing tasks according to actual needs, achieve more personalized and variable operation plans, and expand the system's application scenarios.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 This is a structural diagram of the multi-modal interactive robotic arm intelligent task allocation and priority management system of the present invention. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 In an embodiment of the present invention, a multi-modal interactive robot arm intelligent task allocation and priority management system includes: The robot arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure accurate grinding on different materials and complex surfaces. A multimodal input device, used to receive voice commands and gesture actions input by a user, the multimodal input device comprising: A microphone for receiving user voice commands to create, schedule, and manage tasks; The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm; A multimodal interaction algorithm module, used to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm; The task management unit is used to create, adjust and manage polishing tasks, including task priority setting, conflict detection and task scheduling, wherein the task management unit includes: The task creation module is used to receive voice or gesture input to generate new polishing tasks; Priority management module, used to dynamically adjust the priority of tasks according to their urgency, importance and dependencies; Task scheduling module, used to dynamically adjust the task execution order based on priority management results; A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes: Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time; A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur; Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

[0025] As a further solution of the present invention, the multimodal interaction algorithm module uses natural language processing NLP and deep learning technology to perform speech and gesture recognition, which is implemented through convolutional neural network CNN and long short-term memory network LSTM, wherein: The recognition of voice commands is based on the following model formula: in, Represents a given input When probability; is the hidden state obtained by LSTM; is the weight matrix; The parsing process is as follows: 1. Noise filtering: Noise filtering is achieved through short-time Fourier transform (STFT), and the formula is: in is the time-frequency diagram, is the input signal, is the window function.

[0026] 2. Feature extraction, use Mel frequency cepstral coefficient (MFCC) to extract features.

[0027] 3. Model analysis: Input the features into the speech recognition model for analysis.

[0028] Example code (Python): # Read audio files and extract MFCC features def extract_features(file_path): y,sr = librosa.load(file_path, sr=None) mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13) return np.mean(mfcc.T, axis=0) # Speech Recognition def recognize_command(audio_file, model): features = extract_features(audio_file) features = features.reshape(1, -1)# Adjust the shape predicted_label = model.predict(features) return predicted_label # Load the model model = load_model('speech_recognition_model.h5') command = recognize_command('user_command.wav', model) As a further solution of the present invention, the priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task, and the priority calculation formula is: in, and is the weight coefficient, which is adjusted dynamically according to task requirements.

[0029] As a further solution of the present invention, the task scheduling module sorts the task queues according to the priority, and the task scheduling formula is: T represents the task queue, and Sort represents sorting by priority.

[0030] As a further solution of the present invention, the conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module processes task conflicts based on a priority adjustment algorithm and postpones the execution time of low-priority tasks by t low priority. The calculation formula is: Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

[0031] As a further solution of the present invention, the task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback, and the control strategy formula is: Among them, F 打磨is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

[0032] And, the function of the gesture recognition module is to convert the user's gesture movements into operating instructions for the robotic arm, especially for adjusting task parameters.

[0033] The technology is implemented by using deep learning models (such as ResNet or YOLO) for image processing and gesture recognition. This module uses a convolutional neural network (CNN) to process the captured hand images and recognizes the shape, direction, trajectory and other information of the gesture through a posture estimation algorithm.

[0034] The formula is: in, Represents a given image When the class probability; is the feature output by the CNN model.

[0035] The system predefines a library of common gestures (such as "left", "right", "raise priority", "cancel task", etc.), and users can also expand the gesture library by training new gestures.

[0036] The parsing process is as follows: Image preprocessing: denoising and edge detection are performed on the images captured by the camera.

[0037] Feature extraction: Use CNN to extract features of the hand area.

[0038] def preprocess_image(image): # Preprocess the image image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) image = cv2.GaussianBlur(image, (5, 5), 0) return image Gesture recognition uses a posture estimation algorithm to identify gestures and match them with a gesture library.

[0039] Example code: # Gesture Recognition def recognize_gesture(image, model): processed_image = preprocess_image(image) # Assume the input is a 64x64 grayscale image features = model.predict(processed_image.reshape(1, 64, 64, 1)) return np.argmax(features) # Load the gesture recognition model gesture_model = load_model('gesture_recognition_model.h5') frame = cv2.imread('hand_gesture.jpg') gesture = recognize_gesture(frame, gesture_model) The task management unit is responsible for managing and scheduling all polishing tasks, including task creation, priority adjustment, conflict detection and scheduling. Its functional module design ensures that the system runs efficiently in a multi-tasking environment.

[0040] 1. Task creation module.

[0041] Responsible for receiving user voice or gesture input, creating new polishing tasks and assigning initial priorities.

[0042] formula: in, Representation Task The priority of Indicates task requirements, Indicates the task type.

[0043] 2. Priority management module.

[0044] Dynamically adjust task priorities based on factors such as the importance of the polishing area, polishing time, and task dependencies.

[0045] formula: in, Indicates the adjusted priority. is the degree of task dependency, is the urgency of the task, and α, β, and γ are weight coefficients.

[0046] 3. Task Scheduling Module Based on the priority management results and real-time polishing status, the execution order of polishing tasks is dynamically scheduled.

[0047] formula: Among them, T is the task queue, and Sort means sorting by priority.

[0048] Example code: def create_task(self, description, initial_priority): new_task = Task(description, initial_priority) self.tasks.append(new_task) print(f"Create task: {description}, priority: {initial_priority}") def adjust_priority(self, task, dependency_degree, urgency_level): # Dynamically adjust task priority task.priority = (0.5 * task.priority) + (0.3 * dependency_degree) + (0.2 * urgency_level) print(f"Adjust task: {task.description}'s priority is: {task.priority}") def schedule_tasks(self): # Sort tasks by priority self.tasks.sort(key=lambda x: x.priority, reverse=True) return [task.description for task in self.tasks] The priority decision unit is responsible for dynamically adjusting task priorities according to the urgency, dependencies and real-time feedback information of tasks during the execution of multiple tasks, and performing intelligent mediation when task conflicts occur to ensure stable and efficient operation of the system.

[0049] 1. Conflict Detection Module Monitor the task dependencies and resource usage in the polishing task queue in real time to detect whether there is a task conflict, such as two tasks needing to use the same polishing head at the same time.

[0050] 2. Real-time mediation module When task conflicts are detected, the system automatically mediates based on task priority and real-time work status, such as postponing low-priority tasks and requesting user confirmation when necessary.

[0051] formula: Indicates the adjusted task The start time of Indicates the original task start time, To postpone the time, the postponement time is dynamically calculated based on the task priority and conflict degree.

[0052] 3. Decision-making feedback mechanism The system provides real-time feedback of conflict detection results and mediation decisions through the user interface, and users can choose to continue, cancel or modify the current task settings.

[0053] formula: Indicates the user's task The final decision Indicates the system's feedback information on the task. Indicates the user's priority judgment.

[0054] Sample code: def detect_conflicts(self): # Conflict detection logic for i in range(len(self.tasks)): for j in range(i + 1, len(self.tasks)): if self.tasks[i].resource == self.tasks[j].resource: if self.tasks[i].start_time < self.tasks[j].end_time: self.conflicts.append((self.tasks[i],self.tasks[j])) print(f"Task conflict detected: {self.tasks[i].description} and {self.tasks[j].description}") def resolve_conflicts(self): # Conflict mediation logic for task_i, task_j in self.conflicts: if task_i.priority > task_j.priority: task_j.start_time += 1 # postpone low priority tasks print(f"Postpone task: {task_j.description} start time to {task_j.start_time}") else: task_i.start_time += 1 print(f"Postpone task: {task_i.description} start time to {task_i.start_time}") def feedback_decision(self): # Decision feedback logic for task_i, task_j in self.conflicts: print(f"User decision: Tasks {task_i.description} and {task_j.description} conflict. Please choose to continue, cancel or modify task settings.") As a further solution of the present invention, the priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task, and the priority calculation formula is: in, and is the weight coefficient, which is adjusted dynamically according to task requirements.

[0055] As a further solution of the present invention, the task scheduling module sorts the task queues according to the priority, and the task scheduling formula is: T represents the task queue, and Sort represents sorting by priority.

[0056] As a further solution of the present invention, the conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module processes task conflicts based on a priority adjustment algorithm and postpones the execution time of low-priority tasks by t low priority. The calculation formula is: Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

[0057] As a further solution of the present invention, the task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback, and the control strategy formula is: Among them, F 打磨 is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

[0058] As a further solution of the present invention, the multimodal input device includes a microphone with a noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

[0059] As a further solution of the present invention, the multimodal interaction algorithm module supports user-defined voice and gesture instructions to enhance the flexibility and adaptability of task allocation and priority management.

[0060] As a further solution of the present invention, the following steps are included: Step 1, multimodal input reception: receiving a user's voice command and gesture action through a multimodal input device, wherein the multimodal input device includes a microphone and a camera, which are used to receive voice and capture gesture actions respectively; Step 2: Multimodal input analysis: Use the multimodal interaction algorithm module to analyze the voice and gesture input, including: Use natural language processing technology to parse voice commands and identify semantic information; Use deep learning models to identify gesture image features and convert parsed voice and gesture inputs into robot arm operation commands; Step 3, task management: Create, adjust and manage polishing tasks according to the parsed user instructions in the task management unit, including the following steps: Assign an initial priority to each task based on its urgency, importance, and dependencies; During task execution, dynamically adjust task priorities based on task conditions and real-time feedback; Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks; Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource occupancy in the task queue are monitored in real time, including the following steps: Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task is executed first; Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts; Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts; Step 5, task execution and feedback adjustment: Control the robotic arm to perform the polishing task according to the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on feedback to ensure polishing accuracy and quality.

[0061] As a further solution of the present invention, the following steps are further included: User-defined command input: Users set customized voice and gesture commands through multimodal input devices. The customized commands are used to adjust the priority of tasks, create or cancel tasks; Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands; Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. Meanwhile, mediate or postpone low-priority tasks to enhance system flexibility and adaptability.

[0062] In the polishing of automobile shells, different surface areas (such as doors, roofs, contour edges, etc.) have significantly different requirements for polishing accuracy and strength, and the polishing path and angle need to be flexibly adjusted. These complex polishing tasks often involve multiple tasks being performed simultaneously, requiring the robotic arm to be able to intelligently allocate tasks, adjust priorities in real time, resolve task conflicts, and dynamically optimize the polishing process based on feedback information. However, due to the limitations of fixed programs and manual control, traditional systems are difficult to achieve ideal results in high-demand multi-task environments. Therefore, how to achieve intelligent multi-task scheduling, efficiently resolve resource conflicts, and ensure polishing accuracy and efficiency has become an urgent need in the industry. The present invention provides a multimodal interactive robotic arm intelligent task allocation and priority management system, which achieves high efficiency and high precision of polishing operations in the above scenarios through the following specific steps and technical solutions.

[0063] First, the system collects user voice commands and gesture inputs through multimodal input devices. In actual operation, the operator can directly specify the polishing tasks of different areas through voice, such as "polishing the outer surface of the car door" or "prioritizing the roof polishing task", and use gestures to fine-tune the polishing angle of the robot arm to ensure that the polishing direction and posture meet the specific surface requirements. This multimodal interaction method eliminates the need for complex programming, allowing operators to create and adjust tasks more intuitively and efficiently during operation, and flexibly adapt to changes in the needs of the polishing scene.

[0064] In a multi-tasking environment, the system's task management unit automatically assigns an initial priority based on the task type and urgency input by the operator. For example, door polishing may be given a higher priority due to its surface characteristics and importance, and the system automatically sorts it at the front of the task queue to ensure that the task is executed in a timely manner. At the same time, when the operator issues a new instruction or the task requirements change, the priority management module can readjust the priority according to the real-time situation during the task execution process to adapt to the new task requirements. Unlike the existing technology, this system dynamically manages task priorities through intelligent algorithms, avoiding task delays and frequent manual intervention caused by the solidification of task priorities, making the system adaptive.

[0065] In addition, in actual polishing scenarios, multiple tasks often compete for the same polishing head, resulting in task conflicts and affecting overall efficiency. This system monitors task dependencies and resource allocation in real time through the conflict detection module of the priority decision unit. When it detects that two tasks are competing for the same resource, the system will automatically trigger the real-time mediation mechanism to appropriately postpone or suspend the low-priority task. For example, when the polishing tasks of the door and the roof need to occupy the same polishing head at the same time, the system will automatically give priority to the door polishing task and postpone the roof task to ensure that the high-priority task is polished unimpeded. This conflict detection and mediation mechanism effectively avoids task delays or task interruptions caused by resource competition in traditional systems, realizes efficient resource utilization and smooth task execution, and ensures the continuity of operations in a multi-task environment.

[0066] During the polishing process, the system continuously optimizes the polishing force and path based on real-time feedback to ensure the polishing quality. For example, the complex surfaces and contours of the roof and edges need to be finely processed under a specific polishing force, and the force control sensor detects the feedback data of the polishing surface in real time, such as surface finish, torque change, etc. When it is detected that the polishing finish of the door surface is lower than expected, the system will automatically adjust the polishing force and path to optimize the polishing effect by increasing or decreasing the polishing force. Unlike existing systems, this system uses real-time feedback to adaptively adjust the polishing parameters, and can dynamically adapt to the precision requirements of different materials and complex surfaces, thereby ensuring that the entire polishing area achieves consistent surface quality. This feedback-based polishing optimization not only improves the polishing accuracy of the robot arm, but also meets the high standards for complex surfaces and heterogeneous materials in high-precision industrial scenarios.

[0067] In addition, the system supports operator-defined command functions. For example, the operator can set "reduce door polishing intensity" or "increase roof priority" through customized gestures or voice, and recognize these customized commands through multimodal interaction algorithms, so that the system can flexibly adjust task parameters according to the actual needs of users. This customization function not only enhances the adaptability of the system, enabling it to maintain efficient operation under changing production needs, but also avoids the problem that fixed program control is difficult to cope with complex on-site needs.

[0068] In summary, this embodiment demonstrates how the multimodal interactive robotic arm intelligent task allocation and priority management system of the present invention can achieve intelligent, multi-task management in complex polishing scenarios. Through natural voice and gesture interaction, intelligent dynamic priority adjustment, real-time conflict detection and mediation, and feedback-driven adaptive optimization of polishing paths and forces, the system significantly improves operational flexibility, resource utilization efficiency, and polishing accuracy, and comprehensively solves the technical problems of insufficient multi-task management, frequent task conflicts, and low feedback adjustment capabilities in the prior art. The application of the present invention not only reduces manual intervention in polishing operations, but also achieves high-quality polishing effects under high-precision and high-efficiency requirements.

[0069] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "screw-on" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0070] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A multi-modal interactive robot arm intelligent task allocation and priority management system, characterized in that: include: The robot arm is used to perform grinding tasks. It has multi-degree-of-freedom motion capabilities and is equipped with a grinding head and force sensor to ensure accurate grinding on different materials and complex surfaces. A multimodal input device, used to receive voice commands and gesture actions input by a user, the multimodal input device comprising: A microphone for receiving user voice commands to create, schedule, and manage tasks; The camera is used to capture the user's gestures to adjust the grinding posture of the robotic arm; A multimodal interaction algorithm module, used to parse the voice and gesture inputs received by the multimodal input device and convert them into operation commands for the robotic arm; The task management unit is used to create, adjust and manage polishing tasks, including task priority setting, conflict detection and task scheduling, wherein the task management unit includes: The task creation module is used to receive voice or gesture input to generate new polishing tasks; Priority management module, used to dynamically adjust the priority of tasks according to their urgency, importance and dependencies; Task scheduling module, used to dynamically adjust the task execution order based on priority management results; A priority decision unit is used to make real-time decisions when task priorities change or task conflicts occur. The priority decision unit includes: Conflict detection module, used to monitor task dependencies and resource usage in the task queue in real time; A real-time mediation module is used to automatically adjust the order of tasks according to their priorities when task conflicts occur; Decision feedback mechanism, used to provide users with feedback on conflict detection results and mediation status.

2. A multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal interaction algorithm module uses natural language processing (NLP) and deep learning technology to perform speech and gesture recognition, which is achieved through convolutional neural network (CNN) and long short-term memory network (LSTM), where: The recognition of voice commands is based on the following model formula: in, Represents a given input When probability; is the hidden state obtained by LSTM; is the weight matrix; Noise filtering is achieved through short-time Fourier transform STFT, the formula is: in is the time-frequency diagram, is the input signal, is the window function.

3. The multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The priority management module dynamically adjusts the priority based on the urgency, importance and dependency of the task. The priority calculation formula is: in, and is the weight coefficient, which is adjusted dynamically according to task requirements.

4. The multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The task scheduling module sorts the task queues according to the priority, and the task scheduling formula is: T represents the task queue, and Sort represents sorting by priority.

5. The multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The conflict detection module automatically triggers the real-time mediation module when it detects that multiple tasks need to occupy the same resources. The real-time mediation module processes task conflicts based on the priority adjustment algorithm and postpones the execution time of the low-priority task by t low priority. The calculation formula is: Among them, Δt is the delay time, which is determined by the priority difference and the coefficient set by the task dependency.

6. A multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The task scheduling module dynamically adjusts the grinding path and force control strategy based on real-time feedback. The control strategy formula is: where F 打磨 is the current grinding intensity, F 初始 is the initial grinding force, k is the control gain, feedback quality is the real-time feedback surface quality, and target quality is the preset surface quality standard.

7. The multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal input device includes a microphone with noise filtering function and a camera supporting high resolution and high frame rate to adapt to complex industrial environments.

8. The multi-modal interactive robot arm intelligent task allocation and priority management system according to claim 1, characterized in that: The multimodal interaction algorithm module supports user-defined voice and gesture commands to enhance the flexibility and adaptability of task allocation and priority management.

9. A multi-modal interactive robot arm intelligent task allocation and priority management method according to claim 1, characterized in that: The following steps are involved: Step 1, multimodal input reception: receiving a user's voice command and gesture action through a multimodal input device, wherein the multimodal input device includes a microphone and a camera, which are used to receive voice and capture gesture actions respectively; Step 2: Multimodal input analysis: Use the multimodal interaction algorithm module to analyze the voice and gesture input, including: Use natural language processing technology to parse voice commands and identify semantic information; Use deep learning models to identify gesture image features and convert parsed voice and gesture inputs into robot arm operation commands; Step 3, task management: Create, adjust and manage polishing tasks according to the parsed user instructions in the task management unit, including the following steps: Assign an initial priority to each task based on its urgency, importance, and dependencies; During task execution, dynamically adjust task priorities based on task conditions and real-time feedback; Sort the task queue based on the result of priority adjustment and dynamically schedule the execution order of tasks; Step 4, conflict detection and mediation: In the priority decision unit, the task dependencies and resource occupancy in the task queue are monitored in real time, including the following steps: Detect whether there is a task resource conflict. When it is detected that multiple tasks need to share the same resources, the high-priority task is executed first; Postpone or adjust the execution time of low-priority tasks to avoid resource conflicts; Update the priorities and order of tasks in the task queue after detecting and reconciling conflicts; Step 5, task execution and feedback adjustment: Control the robotic arm to perform the polishing task according to the scheduling results, monitor the polishing progress in real time, and adjust the polishing path and force based on feedback to ensure polishing accuracy and quality.

10. The method for intelligent task allocation and priority management of a multimodal interactive robot arm according to claim 1, characterized in that: Further comprising the steps of: User-defined command input: Users set customized voice and gesture commands through multimodal input devices. The customized commands are used to adjust the priority of tasks, create or cancel tasks; Custom instruction parsing and execution: The system receives and parses custom instructions and generates corresponding task operation commands; Task priority and conflict mediation: Update task priorities based on custom instructions, and prioritize high-priority tasks when there are resource conflicts. Meanwhile, mediate or postpone low-priority tasks to enhance system flexibility and adaptability.

Images