Desktop-level multi-task mechanical arm control method and related device

Abstract

The invention relates to the technical field of data processing, and discloses a desktop-level multi-task mechanical arm control method and related device.The desktop-level multi-task mechanical arm control method comprises the steps that target three-dimensional coordinate information of a target object is detected and positioned from a captured two-dimensional image through a visual recognition module; voice information corresponding to the target task is transcribed into target text information through a voice recognition sub-module of the natural language processing module, a user intention is extracted from the target text information through an intention recognition sub-module of the natural language processing module, and a corresponding target task instruction is generated; mechanical arm path planning is conducted on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through the control center module, and a target control command is generated; and the mechanical arm module is adopted, according to the target control command, each joint is controlled to execute the mechanical arm operation corresponding to the target control command, and the accuracy of the desktop-level robot in the aspects of complex task processing and man-machine interaction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a desktop-level multi-task robotic arm control method and related devices. Background Art

[0002] With the rapid development of artificial intelligence and robotics, desktop robots are widely used in education, scientific research, and home furnishing due to their small size, flexibility, and quick response. However, existing desktop robots are limited in computing power and perception capabilities when handling complex tasks and dynamic interactions, making it difficult to achieve efficient multi-tasking operations. Traditional desktop robots rely on simple sensors and preset instructions, lack advanced visual perception and natural language processing capabilities, and their application in a variety of tasks is limited. Therefore, how to improve the accuracy of desktop robots in complex task processing and human-computer interaction in order to achieve more effective and accurate control of desktop multi-tasking robotic arms has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a desktop-level multi-task robotic arm control method and related devices to solve the technical problem of improving the accuracy of desktop-level robots in complex task processing and human-computer interaction.

[0004] In a first aspect, a desktop-level multi-task robotic arm control method is provided, comprising:

[0005] Detecting and locating target three-dimensional coordinate information of a target object from a captured two-dimensional image through a visual recognition module;

[0006] Through the speech recognition submodule of the natural language processing module, the speech information corresponding to the target task is transcribed into target text information.

[0007] Through the intention recognition submodule of the natural language processing module, the user intention is extracted from the target text information and the corresponding target task instructions are generated;

[0008] The control center module performs robot arm path planning based on the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module to generate target control commands;

[0009] Through the robotic arm module, according to the target control command, each joint is controlled to perform the robotic arm operation corresponding to the target control command.

[0010] In a second aspect, a desktop-level multi-task robotic arm control device is provided, comprising:

[0011] A first processing module, used for detecting and locating target three-dimensional coordinate information of a target object from a captured two-dimensional image;

[0012] The second processing module is used to transcribe the speech information corresponding to the target task into target text information.

[0013] The third processing module is used to extract the user intention from the target text information and generate corresponding target task instructions;

[0014] The fourth processing module is used to perform robot arm path planning based on the three-dimensional coordinate information and the target task instructions to generate target control commands;

[0015] The fifth processing module is used to control each joint to perform the robot arm operation corresponding to the target control command according to the target control command.

[0016] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned desktop-level multi-tasking robotic arm control method when executing the computer program.

[0017] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned desktop-level multi-tasking robotic arm control method are implemented.

[0018] In the scheme implemented by the above-mentioned desktop-level multi-task robotic arm control method and related devices, the target three-dimensional coordinate information of the target object is detected and located from the captured two-dimensional image through the visual recognition module, and the voice information corresponding to the target task is transcribed into target text information through the voice recognition submodule of the natural language processing module, so that the user intention is extracted from the target text information through the intention recognition submodule of the natural language processing module, and the corresponding target task instructions are generated, so that the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module can be used for robotic arm path planning through the control center module to generate target control commands, and then the robotic arm module can be used to control each joint to perform the robotic arm operation corresponding to the target control command according to the target control command, which can improve the accuracy of the desktop robot in complex task processing and human-computer interaction, so as to achieve the purpose of more effective and accurate control of the desktop-level multi-task robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1It is a schematic diagram of the structure of a desktop multi-task robotic arm control system in one embodiment of the present invention;

[0021] Figure 2 is a flow chart of a desktop-level multi-task robotic arm control method according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a process of pruning a network in one embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a process for performing intent recognition in one embodiment of the present invention;

[0024] Figure 5 It is a structural schematic diagram of a hardware module of a desktop-level multi-task robotic arm control system in one embodiment of the present invention;

[0025] Figure 6 This is a test schematic diagram of a desktop-level multi-task robotic arm control system performing a light-off operation in one embodiment of the present invention;

[0026] Figure 7 is a structural schematic diagram of a desktop-level multi-task robotic arm control device in one embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Deep learning technologies developed in recent years, especially YOLO (You Only Look Once) for fast and accurate object detection, and pre-trained language models such as Bidirectional Encoder Representations Transformers (BERT) for context-aware natural language processing, provide new methods to enhance the perception and interaction capabilities of robots. However, the above technologies usually require high computing resources. For resource-constrained embedded systems, how to achieve efficient integration and real-time processing of these technologies is a major challenge.

[0031] In view of the limitations of existing desktop robots in complex task processing and human-computer interaction, the present invention proposes a multi-task desktop robotic arm control system based on visual recognition and natural language processing. The system integrates YOLOv7 and BERT models, and can efficiently perform complex multi-task operations on embedded platforms. The computing performance can be optimized through pruning and quantization techniques, and the control center can achieve accurate scheduling and real-time response to multiple tasks through kinematic calculations and state machine management. The present invention can provide strong support for improving the intelligence and multi-functionality of desktop robots, and has broad application prospects.

[0032] See also Figure 1 As shown, Figure 1 A schematic diagram of the structure of a desktop-level multi-task robotic arm control system provided by an embodiment of the present invention. Exemplarily, taking the scenario of controlling a desktop-level multi-task robotic arm as an example, the desktop-level multi-task robotic arm control method provided by the present application can be applied to a desktop-level multi-task robotic arm control system, wherein the desktop-level multi-task robotic arm control system includes a visual recognition module, a natural language processing module, a control center module, and a robotic arm module. The target three-dimensional coordinate information of the target object is detected and located from the captured two-dimensional image through the visual recognition module, and the voice information corresponding to the target task is transcribed into target text information through the voice recognition submodule of the natural language processing module, so that the user intention is extracted from the target text information through the intention recognition submodule of the natural language processing module, and the corresponding target task instruction is generated, so that the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module can be used for robot arm path planning through the control center module to generate a target control command, and then the robot arm module can control each joint to perform the robot arm operation corresponding to the target control command according to the target control command, which can improve the accuracy of the desktop robot in complex task processing and human-computer interaction, so as to achieve the purpose of more efficient and accurate control of the desktop multi-task robot.

[0033] See also Figure 2 As shown, Figure 2 A flowchart of a desktop multi-task robotic arm control method provided by an embodiment of the present invention includes the following steps:

[0034] S10: Detecting and locating target three-dimensional coordinate information of a target object from the captured two-dimensional image through the visual recognition module.

[0035] The visual recognition module can be responsible for detecting and locating the two-dimensional coordinate information of the target object from the captured two-dimensional image through the model, so as to obtain the three-dimensional coordinate information of the target object based on the two-dimensional coordinate information. This module can use advanced computer vision technology, such as a target detection algorithm based on deep learning, to accurately identify and determine the position of the target object in space, so as to provide the position information of the target object for subsequent robot arm operations, and this application does not limit this.

[0036] It should be understood that detecting and locating the target three-dimensional coordinate information of the target object from the captured two-dimensional image by the visual recognition module refers to the process of obtaining the target three-dimensional coordinate information of the target object. Wherein, in step S10, that is, detecting and locating the target three-dimensional coordinate information of the target object from the captured two-dimensional image by the visual recognition module, includes the following steps:

[0037] S11: preprocessing the captured two-dimensional image by the visual recognition module to obtain a preprocessed image;

[0038] S12: performing convolution processing on the pre-processed image through the visual recognition module to obtain a convolution feature image;

[0039] S13: performing maximum pooling processing on the convolution feature image through the visual recognition module to obtain a reduced feature image;

[0040] S14: performing multi-scale feature fusion processing on the reduced feature image through the visual recognition module to obtain a multi-scale feature image;

[0041] S15: performing frame recognition processing on the multi-scale feature image by the visual recognition module to obtain a frame feature image;

[0042] S16: Determine the two-dimensional coordinate information of the target object according to the coordinate information of the frame feature image by the visual recognition module;

[0043] S17: fusing the two-dimensional coordinate information with the depth camera through the visual recognition module to obtain reference three-dimensional coordinate information under the depth camera coordinates;

[0044] S18: The visual recognition module multiplies the reference three-dimensional coordinate information by the rotation matrix of the camera to obtain the three-dimensional coordinate information of the target object.

[0045] The visual recognition module can input the captured image (such as captured by a camera) into the input network of the model for preprocessing, and scale the image up or down to a fixed size (such as 640*640), and further normalize the pixel values ​​of the image to accelerate model training and reasoning.

[0046] The visual recognition module can input the processed image into the backbone network of the model, and can first pass through a convolution layer and a pooling layer to extract features and reduce the image resolution, and then perform multiple convolutions to obtain image features. Optionally, the visual recognition module can perform a pooling operation (such as a maximum pooling operation) after several layers of convolution layers to reduce the feature map by the maximum value of several local feature areas.

[0047] The Backbone network further groups the obtained feature maps into different groups, such as using different convolution kernels for convolution operations to obtain different types of features, and convolving and fusing the features of different paths to obtain feature maps of different scales, and finally obtain feature maps of different scales.

[0048] The visual recognition module can input the feature map into the neck layer for feature fusion, which can be done in two ways: top-down and bottom-up. The top-down method can enlarge the small-scale feature map for feature fusion, and the bottom-up method can reduce the large-scale feature map for feature fusion. It can be understood that by fusing feature maps of different scales, the expression ability of multi-scale features can be enhanced to improve the detection ability of network targets;

[0049] The visual recognition module can convert the input image into a feature map and input it into the last part of the entire network, the detection head. This part can regard each pixel position in the feature map as a grid, and can make a one-to-one correspondence in each grid according to the ratio of the feature map to the image to complete the image recognition task.

[0050] Optionally, the visual recognition module can detect the target for each grid, and finally fuse the predicted adjacent grids to obtain the classification of the detected target. The model can be trained to obtain the object category. The confidence score can be calculated to determine the probability of the grid containing the object and the border of the object.

[0051] Optionally, in order to further obtain the position information of the target object, the visual recognition module can obtain the coordinates of the upper left corner of the frame, such as (x, y), and the width and height of the frame, such as (w, h), so as to obtain the center position of the target object according to the coordinates of the upper left corner and the width and height information, such as (x, y). c,Y c ) indicates that the specific formula can be as follows:

[0052]

[0053] Among them, X c Indicates the horizontal coordinate of the center position of the target object; Y c Represents the ordinate of the center position of the target object; x represents the abscissa of the upper left corner of the border; w represents the width of the border; y represents the ordinate of the upper left corner of the border; h represents the height of the border.

[0054] Optionally, in order to obtain the three-dimensional coordinate information of the target object, the visual recognition module can use a depth camera to obtain a depth map and fuse it with red, green, and blue depth (RGBD) image information, such as multiplying the camera intrinsic parameter matrix K by the position in the pixel coordinate system to obtain the three-dimensional coordinates of the target object in the camera coordinate system, such as expressed as (X, Y, Z).

[0055] Furthermore, in order to obtain the real position of the object in the real coordinate system, the visual recognition module can multiply the three-dimensional coordinates of the target object in the camera coordinates with the rotation matrix of the camera to obtain the target position of the target object in the real world, that is, the above-mentioned target three-dimensional coordinate information.

[0056] It should be noted that the method of obtaining the three-dimensional position information of an object by combining a depth camera places relatively stringent requirements on the computing power of the edge device. In order to improve performance on embedded devices, this application can use pruning and quantization methods to reduce computing and storage overhead to meet the requirements of efficient operation on embedded devices.

[0057] Pruning technology can reduce the computational complexity of the model by removing redundant neural network connections. Pruning can reduce the model size and computational requirements by removing unnecessary parts of the neural network, such as performing forward propagation on the training set and analyzing the activation distribution of each neuron to determine its impact on recognition and remove neurons that have little impact on model performance.

[0058] Optionally, the importance of features can be evaluated by permutation importance evaluation, which permutes the feature vectors of the input convolutional layer, shuffles the order of the feature vectors, keeps other feature vectors unchanged, and performs multiple training to determine the impact of the channel on network recognition. The specific formula can be as follows:

[0059]

[0060] Among them, i jcan represent the importance of feature j, depending on the extent to which the model performance decreases in multiple experiments after permutation; s can represent the score of the model without permuting the feature; K can represent the total number of training times; k can represent the index of the number of training times; s k,j is the model performance score in the kth training after permuting feature j. It can be understood that according to the change in the score, the network can retain channels with higher importance and remove parts with lower importance, so as to obtain a more compact and efficient recognition model.

[0061] Quantization technology can further reduce computing resource usage by converting floating-point calculations to fixed-point calculations. Quantization can use fewer bits to represent weights and activation values. On the premise of ensuring model recognition accuracy, the floating-point convolution layer weights and activation values ​​are converted to fixed-point representations, such as converting 32-bit floating-point numbers FP32 to 8-bit integers INT8, thereby reducing the storage space of model parameters. Therefore, quantization of model weights and activation values ​​can effectively reduce the size of the model.

[0062] Since the activation value will change with the change of weight, a small part of the original data set is used as calibration data to quantize the weight, and the data set is input into the floating-point weight network to record the size range of the activation value transformation recognized by the model. Optionally, if the floating-point range is between [a, b], this range can be mapped to the integer range of [-128, 127]. The specific formula can be as follows:

[0063]

[0064] Among them, a can represent the minimum value obtained from the calibration data set; b can represent the maximum value obtained from the calibration data set; round can represent the rounding operation of floating-point data to finally obtain a quantized value of the INT8 type for acceleration in the network.

[0065] For example, Figure 3 As shown, Figure 3 The schematic diagram of pruning the network is shown as an example. In the convolutional layer on the left, there can be several channels (such as C i1 , C i2 , C i3 , C i4 , C i5 , C i6 ), each channel data can capture the characteristics of different images through the mapping of the convolution layer, and then calculate the importance of the channel data to measure the impact on the subsequent output layer, such as the channel with less impact on the network (C i2 , C i4 ) is cropped, thereby retaining the more important channels (Ci1 , C i3 , C i5 , C i6 ). By analyzing the weights of the convolutional layers using feature importance to assess the contribution of neurons to model accuracy and feature extraction capabilities, identifying and removing weights that have little impact on the model can further reduce model size and computational requirements.

[0066] S20: The speech information corresponding to the target task is transcribed into target text information through the speech recognition submodule of the natural language processing module.

[0067] The speech recognition submodule of the natural language processing module can be used to transcribe the speech information corresponding to the target task into target text information. This process may involve speech signal processing and speech recognition algorithms to convert human speech into text form that can be processed by computers.

[0068] It should be understood that transcribing the speech information corresponding to the target task into target text information through the speech recognition submodule of the natural language processing module refers to the process of obtaining the target text information. Among them, step S20, that is, transcribing the speech information corresponding to the target task into target text information through the speech recognition submodule of the natural language processing module, includes the following steps:

[0069] S21: Segmenting the speech information corresponding to the target task through the speech recognition submodule of the natural language processing module to obtain a set of short-term time frames;

[0070] S22: performing short-time Fourier transform on each short-term time frame in the short-term time frame set through the speech recognition submodule of the natural language processing module to obtain a short-term time frame spectrum set;

[0071] S23: using a speech recognition submodule of a natural language processing module, converting each end-period time frame spectrum in the short-period time frame spectrum set into a frequency domain in a logarithmic scale through a Mel filter to obtain a Mel spectrum;

[0072] S24: encoding the Mel spectrum through a self-attention mechanism through a speech recognition submodule of a natural language processing module to obtain an audio feature vector;

[0073] S25: The audio feature vector is decoded by a decoder through the speech recognition submodule of the natural language processing module to obtain target text information.

[0074] The speech recognition submodule of the natural language processing module can extract the speech signal into time frames of short time periods by segmenting the speech signal, and further perform short-time Fourier transform on each time frame to convert it into frequency domain expression to obtain the frequency spectrum of each frame.

[0075] The speech recognition submodule may convert the obtained spectrum through a Mel filter composed of triangular filters to convert the linear frequency domain into a logarithmic frequency domain to obtain a Mel spectrum (Mel spectrum). The horizontal axis of the Mel spectrum may represent a time frame, and the vertical axis may represent the energy distribution of the Mel frequency channel.

[0076] The speech recognition submodule can normalize the Mel spectrum to reduce the amplitude difference, and further encode the Mel spectrum, such as through a self-attention mechanism, so that the encoder can identify important features in the audio and encode them into context-related representations, that is, obtain the feature vector of the above audio.

[0077] The self-attention mechanism can be used to process sequence data and capture the dependencies between different positions in the sequence. Three different vectors are generated through linear transformation: query, key, and value. The similarity between each query vector and the key vector is calculated based on the three vectors, and the weight of the value vector is assigned according to the similarity. The introduction of adaptive coefficients enables the model to dynamically adjust the intensity and range of attention, so as to flexibly allocate attention weights in different contexts and input situations, and introduce sparsity to reduce computational and memory overhead. The specific formula can be as follows:

[0078]

[0079] Among them, Attention score (Q, K, V) can be an attention score, which is used to indicate the similarity between each query vector and the key vector; softmax can be a normalization operation to convert the dot product result into a probability distribution, that is, the normalized attention weights indicate which keys each query should focus on; QK T It can represent the similarity matrix of each input with other inputs, which can be the dot product operation of query vector and key vector to calculate the similarity between query and each key; k It can be a scaling factor to prevent the dot product value from being too large (especially in high-dimensional space), causing the softmax output to be too sharp, making the gradient unstable; It can be a nonlinear activation function; M can be a sparse matrix to limit each element to focus on only a small number of elements near it; V can represent a value vector, and by multiplying the attention weight with the value vector V to generate a weighted output, important information in the entire sequence can be combined; σ can be an activation function, usually a sigmoid function, which is used to map the result to between 0 and 1 to achieve nonlinear transformation; W α can be a learnable weight matrix; X can be the feature vector of the input sequence; b α Can be a bias term.

[0080] Finally, the speech recognition submodule can perform decoding, such as decoding the encoded feature representation into text through a decoder, thereby sequentially generating a text representation of the entire sentence, and then obtaining the above-mentioned target text information.

[0081] S30: extracting the user intention from the target text information through the intention recognition submodule of the natural language processing module, and generating corresponding target task instructions.

[0082] The intention recognition submodule of the natural language processing module can analyze the semantics and context of the text through natural language understanding technology to understand the user's needs and intentions, that is, extract the user's intention from the target text information, and then convert it into specific task instructions that the robotic arm can execute, that is, generate corresponding target task instructions.

[0083] Optionally, for text intent recognition tasks, the BERT model can be pre-trained, such as performing unsupervised learning on large-scale unlabeled text to learn rich language features. Figure 4 As shown, Figure 4 The flowchart of intent recognition is shown as an example. In order to adapt to the specific tasks involved in this application, the pre-trained BERT model can be fine-tuned, such as adding a "CLS" tag to the beginning of the input sequence, creating an extended vector, and then inputting it into the BERT model. Among them, the tag can be used for sentence classification tasks, and fine-tuned through word embedding to adapt the model to specific task data.

[0084] like Figure 4As shown, the input word embedding (W1, W2, W3, W4) can be fine-tuned to adapt the model to specific task data. The output yellow mark can represent the final representation of each input word vector by the BERT model. From the output of the BERT model, key features are extracted, and then these features are input into a linear classifier (LinerClassifier) ​​for classification tasks. The linear classifier is trained from scratch to map the output of BERT to specific classification labels. After the text string is input into the model, the user's intent is identified.

[0085] Optionally, BERT can be used to extract key features and input them into a linear classifier for classification tasks. For example, a speech signal can be input into a trained network to parse the input natural language text into two parts, for example, Intent: the overall intent expressed by the text; Slots: the specific information involved in the intent. After obtaining the Intent and Slots, they can be further mapped to specific computer program operations, such as converting the Intent string into the corresponding function name through a string matching algorithm, and using the string information in the Slots as the parameter of the function. In this way, the text input can be converted into a function that can be directly called by the program, and then the corresponding task can be performed in the subsequent steps.

[0086] S40: Performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through the control center module to generate a target control command.

[0087] The control center module can perform robot arm path planning based on the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module, so as to comprehensively consider the position of the target object and the user's task requirements, thereby planning the robot arm's motion path and generating target control commands, thereby ensuring that the robot arm can perform tasks accurately.

[0088] That is, the control center module can be responsible for receiving and processing input signals from the visual recognition module and the natural language processing module, and generating control instructions for the robot arm. The control center module may include but is not limited to a kinematics calculation unit and a state machine management module, wherein the kinematics calculation unit calculates the end effector path of the robot arm based on the parameter model and generates angle parameters for controlling the servo motor; the state machine management module is used for task scheduling and state management.

[0089] The desktop-level multi-task robot arm control system can decompose the operation of the robot arm into multiple states according to different task requirements, and perform state transitions according to sensor feedback. For example, when performing the task of "picking up a cup", the desktop-level multi-task robot arm control system can sequentially pass through the states of "searching for target", "moving to target", "grasping target", etc., and this application does not limit this.

[0090] It should be understood that the process of generating a target control command by performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through the control center module refers to the process of generating a target control command. Among them, step S40, that is, performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through the control center module, and generating a target control command, includes the following steps:

[0091] S41: Obtain the current state of the robot arm through the control center module;

[0092] S42: determining a set of transition conditions for the robotic arm according to the current state of the robotic arm, the three-dimensional coordinate information from the visual recognition module, and the target task instruction from the natural language processing module through the control center module;

[0093] S43: determining a set of robot arm extension states according to the current state of the robot arm and the set of robot arm transition conditions through a control center module;

[0094] S44: performing robot arm path planning according to the robot arm extension state set and the target task instruction through the control center module to obtain a robot arm path set;

[0095] S45: performing path simulation on each robot path in the robot path set through a control center module to obtain a target robot path;

[0096] S46: Generate a target control command according to the target robot arm path through the control center module.

[0097] This application can use an embedded artificial intelligence computing platform (such as Jetson) to implement the corresponding steps of the control center to coordinate and control the entire system. Specifically, it can accept the object information recognized by the visual module and save the object position and type in the system respectively; after the natural speech processing module processes the speech and extracts the user's intention, it calls the system by calling the function to determine the next operation of the system.

[0098] Among them, for the motion control of the robot arm, an action state machine can be used to describe the behavior of the robot system, so that the robot can complete complex actions. The action state machine is a formalized model used to describe the behavior of the robot system. The action state machine defines and explains the basic state and special state in the system, and describes the transition relationship between states and the actions performed. The basic state can represent the behavior of the robot under normal operation, while the special state can describe the behavior of the robot under abnormal or special circumstances.

[0099] In the robot's action state machine, the basic state consists of basic actions and durations, which may include but are not limited to: "standby", "search", and "grasp". Each basic state is associated with a specific action performed by the robot arm. For example, in the "search" state, the robot arm will perform the task of searching for the target, and in the "standby" state, the robot arm will wait for the control host to issue an order.

[0100] Special states are states that the robot is in under abnormal or special circumstances. Actions in special states can include but are not limited to "random", "fault", "stuck" or "collision". The definition and handling of special states are related to the requirements of the task and the specific design and application of the robot arm.

[0101] When the current state is in any state, the state of the target to which each "any" state transitions will be judged at the end of each action and when entering the state transition to determine the next state; when the robot arm detects a fault, it will enter the "fault" state and send an alarm to notify the control host.

[0102] An extended state can be a state network composed of multiple basic states and transition conditions. This extended state can implement complex commands, such as "follow the target to grasp". Extended states can implement complex behaviors by defining a series of basic states and state transitions. This combination can give the robot system more advanced functions and flexibility.

[0103] State transitions can describe the transition relationship between different states of the robot. When the robot receives a task from the "Standby" state, it will transition to the "Grasp" state. State transitions are usually triggered by specific conditions, often by sensors detecting specific events or external commands triggering state transitions. Each state transition is accompanied by an action to be performed. When the robot arm detects that it is currently stuck, the condition for transitioning to the "Stuck" state is met, then it enters the "Stuck" state and notifies the control host for processing.

[0104] The movement of the robotic arm depends on the coordinate variable information of a target, which is stored in global variables. Optionally, communication can be established with Jetson through the serial port. Bidirectional communication between Jetson and STM32 allows specific tasks to be performed. After receiving the node coordinate command from Jetson, STM32 can calculate the angle of each relevant node servo through inverse kinematics and return this information to Jetson. When Jetson gives a command for the angle of each node, STM32 can use forward kinematics to determine the coordinates of each node and feed this information back to Jetson. The commands issued by Jetson can selectively instruct STM32 to perform actual movement or just query kinematic information.

[0105] S50: Through the robotic arm module, according to the target control command, control each joint to perform the robotic arm operation corresponding to the target control command.

[0106] The robot module can control each joint to perform the robot operation corresponding to the target control command according to the target control command. It can be understood that the robot can be composed of multiple joints, and each joint can be driven to move by receiving commands from the control center to achieve specific operation tasks.

[0107] It should be understood that, by means of the robotic arm module, according to the target control command, controlling each joint to perform the robotic arm operation corresponding to the target control command refers to the process of controlling each joint to perform the robotic arm operation corresponding to the target control command. Wherein, in step S50, that is, by means of the robotic arm module, according to the target control command, controlling each joint to perform the robotic arm operation corresponding to the target control command includes the following steps:

[0108] S51: Acquire the position information of each joint of the robotic arm through the robotic arm module to obtain a set of robotic arm joint coordinate information;

[0109] S52: determining the angle change relationship of each joint of the robotic arm by the robotic arm module according to the robotic arm joint coordinate information set and the target control command, and obtaining an angle change relationship set by using an inverse kinematics algorithm;

[0110] S53: establishing a robotic arm conversion matrix according to each angle change relationship in the angle change relationship set through the robotic arm module;

[0111] S54: Through the robotic arm module, according to the robotic arm conversion matrix, each joint is controlled to perform the robotic arm operation corresponding to the target control command.

[0112] The robotic arm in the robotic arm module can have six degrees of freedom, and the movement of each joint is controlled by multiple servo motors. The base of the robotic arm can be fixed on the desktop, and can flexibly operate various target objects, such as cups, switches, door handles, etc. The end of the robotic arm is equipped with a gripper, which can perform operations such as grabbing and carrying through the instructions of the lower control center. Optionally, this application can use TM32F07VET6 to complete the kinematic modeling of the six-degree-of-freedom robotic arm. Among them, the robotic arm can be equipped with multiple servo motors to execute STM32 control commands.

[0113] It is understandable that in order to achieve quantitative control of the robot arm, the robot arm needs to be modeled. Optionally, the world coordinate system can be defined as the world coordinate system of the robot arm, and each joint defines a different coordinate system. When the robot arm moves, it is necessary to determine the angle change relationship of each joint. The present application can use the Denavit-Hartenberg (DH) parameters to describe the geometric relationship between adjacent coordinate systems, and then determine the changes between the six-degree-of-freedom robot arm joints. Optionally, the parameters can be as follows:

[0114] α i :z i-1 Axis around x i The rotation angle of the axis;

[0115] c i :z i-1 Axis around z i The rotation angle along the x axis i Axis projection;

[0116] d i :x i-1 Axis to x i The distance along the z axis i-1 Projection of axes;

[0117] θ i : The rotation angle of joint i around z i-1 axis.

[0118] Further, for each joint, a transformation matrix can be determined according to the DH parameters. Optionally, the robot module can use the following formula to implement the step of establishing a robot transformation matrix according to each angle change relationship in the angle change relationship set through the robot module:

[0119]

[0120] Among them, A i Represents the transformation matrix for joint i in the robot transformation matrix; i represents the index of the joint; θ i represents joint i around z i-1The rotation angle of the axis; c i Indicates z i-1 Axis around z i The rotation angle of the axis, that is, along the x i Axis projection; α i Indicates z i-1 Axis around x i The rotation angle of the axis; d i Represents x i-1 Axis to x i The distance along the z axis, i.e. i-1 Axis projection.

[0121] When performing a specific task, the robotic arm module needs to solve the position and posture of the end effector. By multiplying the transformation matrices of all joints, the transformation matrix of the end effector relative to the base coordinate system can be obtained, from which the position and posture information of the end effector can be extracted.

[0122] The present invention proposes a desktop-level multi-task robotic arm control method and related device, which can be applied to a desktop-level multi-task robotic arm control system that can efficiently perform multi-task operations in complex and dynamic environments. Figure 5 , Figure 5 The schematic diagram of the structure of the hardware module of the desktop multi-task robot control system is shown as an example. Figure 5 As shown, a desktop-level multi-task robotic arm control system can be composed of Jetson Nano (i.e., the above-mentioned embedded artificial intelligence computing platform Jetson), STM32, a depth camera and a robotic arm. The visual recognition module may include a depth camera to capture images in the environment and obtain three-dimensional depth information of objects.

[0123] It is understandable that the system integrates advanced deep learning models such as YOLOv7 and BERT to achieve high-precision target detection and natural language processing; through pruning and quantization technology optimization, the system can run efficiently on the embedded platform to ensure real-time and low power consumption; in addition, the design of the control center ensures the flexibility and stability of the system in multi-task execution, adapts to complex environmental changes and task requirements, and significantly improves the intelligence level and operational capabilities of desktop robots in practical applications.

[0124] For example, Figure 6 As shown, Figure 6 The following is a schematic diagram showing a test of a desktop multi-task robotic arm control system performing a light-off operation. Figure 6As shown, the robotic arm is placed on the desktop, with the base of the robotic arm as the center point, about 20 cm away from the wall where the switch is located; the visual recognition module identifies objects according to the surrounding environment; the target user (such as R&D personnel or management personnel) issues instructions, such as "please turn off the lights", "I want to turn off the lights", "turn off the lights", "please turn on the lights for me", etc.; the natural language processing module can recognize language processing and extract operation intentions; the control center processes the information of the visual recognition module and the natural language processing module, and generates instructions to control the robotic arm; the robotic arm performs corresponding actions according to the instructions, such as pressing the switch, and when the robotic arm returns to its original position, it is determined that the robotic arm has completed the "turn off the lights" operation, and this application does not impose any restrictions on this.

[0125] It can be seen that in the above scheme, the target three-dimensional coordinate information of the target object is detected and located from the captured two-dimensional image through the visual recognition module, and the voice information corresponding to the target task is transcribed into target text information through the voice recognition submodule of the natural language processing module, so as to extract the user intention from the target text information through the intention recognition submodule of the natural language processing module, and generate corresponding target task instructions, so that the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module can be used by the control center module to perform robot arm path planning and generate target control commands, and then the robot arm module can control each joint to perform the robot arm operation corresponding to the target control command according to the target control command, which can improve the accuracy of the desktop robot in complex task processing and human-computer interaction, so as to achieve the purpose of more efficient and accurate control of the desktop multi-task robot.

[0126] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0127] In one embodiment, a desktop-level multi-task robotic arm control device is provided, and the desktop-level multi-task robotic arm control device corresponds one-to-one to the desktop-level multi-task robotic arm control method in the above embodiment. Figure 7 As shown, the desktop multi-task robot arm control device includes an acquisition module 101, a determination module 102 and a processing module 103. The detailed description of each functional module is as follows:

[0128] The first processing module 101 is used to detect and locate the target three-dimensional coordinate information of the target object from the captured two-dimensional image;

[0129] The second processing module 102 is used to transcribe the speech information corresponding to the target task into target text information.

[0130] The third processing module 103 is used to extract the user intention from the target text information and generate a corresponding target task instruction;

[0131] The fourth processing module 104 is used to perform robot arm path planning on the three-dimensional coordinate information and the target task instruction to generate a target control command;

[0132] The fifth processing module 105 is used to control each joint to perform the robot arm operation corresponding to the target control command according to the target control command.

[0133] In one embodiment, the first processing module 101 is used to detect and locate the target three-dimensional coordinate information of the target object from the captured two-dimensional image, specifically for:

[0134] Preprocessing the captured two-dimensional image to obtain a preprocessed image;

[0135] The pre-processed image is subjected to convolution processing to obtain a convolution feature image;

[0136] Performing maximum pooling processing on the convolution feature image to obtain a reduced feature image;

[0137] Performing multi-scale feature fusion processing on the reduced feature image to obtain a multi-scale feature image;

[0138] Performing frame recognition processing on the multi-scale feature image to obtain a frame feature image;

[0139] Determining two-dimensional coordinate information of the target object according to the coordinate information of the frame feature image;

[0140] Fusing the two-dimensional coordinate information with the depth camera information to obtain reference three-dimensional coordinate information under the depth camera coordinates;

[0141] The reference three-dimensional coordinate information is multiplied by the rotation matrix of the camera to obtain the three-dimensional coordinate information of the target object.

[0142] In one embodiment, the second processing module 102 is used to transcribe the speech information corresponding to the target task into target text information, specifically for:

[0143] Segment the speech information corresponding to the target task to obtain a set of short-term time frames;

[0144] Performing a short-time Fourier transform on each short-time frame in the short-time frame set to obtain a short-time frame spectrum set;

[0145] Convert each end-period time frame spectrum in the short-period time frame spectrum set into a frequency domain in a logarithmic scale through a Mel filter to obtain a Mel spectrum;

[0146] Encoding the Mel spectrum through a self-attention mechanism to obtain an audio feature vector;

[0147] The audio feature vector is decoded by a decoder to obtain target text information.

[0148] In one embodiment, the fourth processing module 104 is used to perform robot arm path planning on the three-dimensional coordinate information and the target task instruction to generate a target control command, specifically for:

[0149] Get the current state of the robot arm;

[0150] Determine a set of transition conditions for the robotic arm according to the current state of the robotic arm, the three-dimensional coordinate information from the visual recognition module, and the target task instruction from the natural language processing module;

[0151] Determine a set of robot arm extension states according to the current state of the robot arm and the set of robot arm transition conditions;

[0152] Performing robot arm path planning according to the robot arm extension state set and the target task instruction to obtain a robot arm path set;

[0153] Performing path simulation on each robot path in the robot path set to obtain a target robot path;

[0154] A target control command is generated according to the target robot arm path.

[0155] In one embodiment, the fifth processing module 105 is used to control each joint to perform the robot arm operation corresponding to the target control command according to the target control command, specifically to:

[0156] Obtain the position information of each joint of the robotic arm and obtain the set of joint coordinate information of the robotic arm;

[0157] According to the mechanical arm joint coordinate information set and the target control command, an inverse kinematics algorithm is used to determine the angle change relationship of each joint of the mechanical arm to obtain an angle change relationship set;

[0158] Establishing a robot arm transformation matrix according to each angle change relationship in the angle change relationship set;

[0159] According to the robot arm conversion matrix, each joint is controlled to perform the robot arm operation corresponding to the target control command;

[0160] The following formula is used to implement the step of establishing a robot arm transformation matrix according to each angle change relationship in the angle change relationship set:

[0161]

[0162] Among them, A i Represents the transformation matrix for joint i in the robot transformation matrix; i represents the index of the joint; θ i represents joint i around z i-1 The rotation angle of the axis; c i Indicates z i-1 Axis around z i The rotation angle of the axis, that is, along the x i Axis projection; α i Indicates z i-1 Axis around x i The rotation angle of the axis; d i Represents x i-1 Axis to x i The distance along the z axis, i.e. i-1 Axis projection.

[0163] The present invention provides a desktop-level multi-task robotic arm control device, which detects and locates the target three-dimensional coordinate information of the target object from a captured two-dimensional image, and transcribes the voice information corresponding to the target task into target text information, so as to further extract the user intention from the target text information and generate corresponding target task instructions, so as to perform robotic arm path planning on the three-dimensional coordinate information and the target task instructions, generate target control commands, and then control each joint to perform the robotic arm operation corresponding to the target control command according to the target control command, so as to improve the accuracy of the desktop-level robot in complex task processing and human-computer interaction, so as to achieve the purpose of controlling the desktop-level multi-task robotic arm more effectively and accurately.

[0164] For the specific definition of the desktop-level multi-tasking robotic arm control device, please refer to the definition of the desktop-level multi-tasking robotic arm control method above, which will not be repeated here. Each module in the above-mentioned desktop-level multi-tasking robotic arm control device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0165] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a desktop-level multi-task robotic arm control method server side.

[0166] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a desktop-level multi-task robotic arm control method.

[0167] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0168] Detecting and locating target three-dimensional coordinate information of a target object from the captured two-dimensional image by the visual recognition module;

[0169] Through the speech recognition submodule of the natural language processing module, the speech information corresponding to the target task is transcribed into target text information.

[0170] Extracting the user's intention from the target text information through the intention recognition submodule of the natural language processing module, and generating corresponding target task instructions;

[0171] Performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through a control center module to generate a target control command;

[0172] Through the robotic arm module, each joint is controlled to perform the robotic arm operation corresponding to the target control command according to the target control command.

[0173] The present invention provides a computer device, which detects and locates the target three-dimensional coordinate information of the target object from the captured two-dimensional image through the visual recognition module, and transcribes the voice information corresponding to the target task into target text information through the voice recognition submodule of the natural language processing module, so as to extract the user intention from the target text information through the intention recognition submodule of the natural language processing module, and generate corresponding target task instructions, so that the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module can be used for robot arm path planning through the control center module to generate target control commands, and then the robot arm module can control each joint to perform the robot arm operation corresponding to the target control command according to the target control command, which can improve the accuracy of the desktop robot in complex task processing and human-computer interaction, so as to achieve the purpose of more efficient and accurate control of the desktop multi-task robot.

[0174] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0175] Detecting and locating target three-dimensional coordinate information of a target object from the captured two-dimensional image by the visual recognition module;

[0176] Through the speech recognition submodule of the natural language processing module, the speech information corresponding to the target task is transcribed into target text information.

[0177] Extracting the user's intention from the target text information through the intention recognition submodule of the natural language processing module, and generating corresponding target task instructions;

[0178] Performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through a control center module to generate a target control command;

[0179] Through the robotic arm module, each joint is controlled to perform the robotic arm operation corresponding to the target control command according to the target control command.

[0180] The present invention provides a computer-readable storage medium, which detects and locates the target three-dimensional coordinate information of the target object from the captured two-dimensional image through the visual recognition module, and transcribes the voice information corresponding to the target task into target text information through the voice recognition submodule of the natural language processing module, so as to extract the user intention from the target text information through the intention recognition submodule of the natural language processing module, and generate corresponding target task instructions, so that the three-dimensional coordinate information from the visual recognition module and the target task instructions from the natural language processing module can be used for robot arm path planning through the control center module to generate target control commands, and then the robot arm module can control each joint to perform the robot arm operation corresponding to the target control command according to the target control command, which can improve the accuracy of the desktop robot in complex task processing and human-computer interaction, so as to achieve the purpose of more efficient and accurate control of the desktop multi-task robot.

[0181] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0183] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0184] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A desktop-level multi-task robotic arm control method, applied to a desktop-level multi-task robotic arm control system, wherein the desktop-level multi-task robotic arm control system comprises a visual recognition module, a natural language processing module, a control center module and a robotic arm module, characterized in that: The desktop-level multi-task robotic arm control method comprises: Detecting and locating target three-dimensional coordinate information of a target object from the captured two-dimensional image by the visual recognition module; Through the speech recognition submodule of the natural language processing module, the speech information corresponding to the target task is transcribed into target text information. Extracting the user's intention from the target text information through the intention recognition submodule of the natural language processing module, and generating corresponding target task instructions; Performing robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module through a control center module to generate a target control command; Through the robotic arm module, each joint is controlled to perform the robotic arm operation corresponding to the target control command according to the target control command.

2. The desktop multi-task robot arm control method according to claim 1, characterized in that: The detecting and locating the target three-dimensional coordinate information of the target object from the captured two-dimensional image by the visual recognition module includes: Preprocessing the captured two-dimensional image by the visual recognition module to obtain a preprocessed image; The pre-processed image is subjected to convolution processing by the visual recognition module to obtain a convolution feature image; Performing maximum pooling processing on the convolution feature image through the visual recognition module to obtain a reduced feature image; Performing multi-scale feature fusion processing on the reduced feature image through the visual recognition module to obtain a multi-scale feature image; Performing frame recognition processing on the multi-scale feature image by the visual recognition module to obtain a frame feature image; Determining the two-dimensional coordinate information of the target object according to the coordinate information of the frame feature image by the visual recognition module; The visual recognition module fuses the two-dimensional coordinate information with the depth camera information to obtain reference three-dimensional coordinate information under the depth camera coordinates; The visual recognition module multiplies the reference three-dimensional coordinate information with the rotation matrix of the camera to obtain the three-dimensional coordinate information of the target object.

3. The desktop multi-task robot arm control method according to claim 2, characterized in that: The speech recognition submodule of the natural language processing module transcribes the speech information corresponding to the target task into target text information, including: The speech information corresponding to the target task is segmented and processed through the speech recognition submodule of the natural language processing module to obtain a set of short-term time frames; Performing a short-time Fourier transform on each short-time frame in the short-time frame set by a speech recognition submodule of a natural language processing module to obtain a short-time frame spectrum set; By using the speech recognition submodule of the natural language processing module, each end-period time frame spectrum in the short-period time frame spectrum set is converted into a frequency domain in a logarithmic scale through a Mel filter to obtain a Mel spectrum; The Mel spectrum is encoded by a self-attention mechanism through a speech recognition submodule of a natural language processing module to obtain an audio feature vector; The audio feature vector is decoded by a decoder through the speech recognition submodule of the natural language processing module to obtain target text information.

4. The desktop multi-task robot arm control method according to claim 3, characterized in that: The control center module performs robot arm path planning on the three-dimensional coordinate information from the visual recognition module and the target task instruction from the natural language processing module to generate a target control command, including: Obtain the current status of the robot arm through the control center module; Determine, by a control center module, a set of transition conditions for the robotic arm according to the current state of the robotic arm, the three-dimensional coordinate information from the visual recognition module, and the target task instruction from the natural language processing module; Determine, by a control center module, a set of robot arm extension states according to the current state of the robot arm and the set of robot arm transition conditions; Through the control center module, the robot arm path planning is performed according to the robot arm extension state set and the target task instruction to obtain the robot arm path set; Through the control center module, each robot path in the robot path set is simulated to obtain a target robot path; A target control command is generated according to the target robot arm path through the control center module.

5. The desktop-level multi-task robotic arm control method according to any one of claims 1 to 4, characterized in that: The method of controlling each joint to perform a robotic arm operation corresponding to the target control command through the robotic arm module according to the target control command includes: Through the robotic arm module, the position information of each joint of the robotic arm is obtained to obtain a set of robotic arm joint coordinate information; By means of the robotic arm module, according to the robotic arm joint coordinate information set and the target control command, using an inverse kinematics algorithm, the angle change relationship of each joint of the robotic arm is determined to obtain an angle change relationship set; By means of the robot module, a robot transformation matrix is ​​established according to each angle change relationship in the angle change relationship set; Through the robotic arm module, according to the robotic arm conversion matrix, each joint is controlled to perform the robotic arm operation corresponding to the target control command; The following formula is used to implement the step of establishing a robotic arm transformation matrix according to each angle change relationship in the angle change relationship set through the robotic arm module: Among them, A i represents the transformation matrix for joint i in the robot transformation matrix; i represents the index of the joint; θ i represents joint i around z i-1 The rotation angle of the axis; c i Indicates z i-1 Axis around z i The rotation angle of the axis, that is, along the x i Axis projection; α i Indicates z i-1 Axis around x i The rotation angle of the axis; d i Represents x i-1 Axis to x i The distance along the z axis, i.e. i-1 Axis projection.

6. A desktop multi-task robot arm control device, characterized in that: The desktop-level multi-task robotic arm control device comprises: A first processing module, used for detecting and locating target three-dimensional coordinate information of a target object from a captured two-dimensional image; The second processing module is used to transcribe the speech information corresponding to the target task into target text information. A third processing module is used to extract the user intention from the target text information and generate a corresponding target task instruction; A fourth processing module, used for performing robot arm path planning on the three-dimensional coordinate information and the target task instruction, and generating a target control command; The fifth processing module is used to control each joint to perform the robot arm operation corresponding to the target control command according to the target control command.

7. The desktop multi-task robot arm control device according to claim 6, characterized in that: The first processing module is used to detect and locate the target three-dimensional coordinate information of the target object from the captured two-dimensional image, specifically for: Preprocessing the captured two-dimensional image to obtain a preprocessed image; The pre-processed image is subjected to convolution processing to obtain a convolution feature image; Performing maximum pooling processing on the convolution feature image to obtain a reduced feature image; Performing multi-scale feature fusion processing on the reduced feature image to obtain a multi-scale feature image; Performing frame recognition processing on the multi-scale feature image to obtain a frame feature image; Determining two-dimensional coordinate information of the target object according to the coordinate information of the frame feature image; Fusing the two-dimensional coordinate information with the depth camera information to obtain reference three-dimensional coordinate information under the depth camera coordinates; The reference three-dimensional coordinate information is multiplied by the rotation matrix of the camera to obtain the three-dimensional coordinate information of the target object.

8. The desktop-level multi-task robotic arm control method according to claim 7, characterized in that: The second processing module is used to transcribe the speech information corresponding to the target task into target text information, specifically for: Segment the speech information corresponding to the target task to obtain a set of short-term time frames; Performing a short-time Fourier transform on each short-time frame in the short-time frame set to obtain a short-time frame spectrum set; Convert each end-period time frame spectrum in the short-period time frame spectrum set into a frequency domain in a logarithmic scale through a Mel filter to obtain a Mel spectrum; Encoding the Mel spectrum through a self-attention mechanism to obtain an audio feature vector; The audio feature vector is decoded by a decoder to obtain target text information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the desktop-level multi-tasking robotic arm control method as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the desktop-level multi-tasking robotic arm control method as described in any one of claims 1 to 5 is implemented.

Images