A multi-modal AI fusion-based mechanical arm autonomous interaction control method and system

Abstract

The application discloses a kind of mechanical arm autonomous interaction control method and system based on multi-modal AI fusion.The method comprises: synchronously collecting audio stream, video frame and joint pose data;User operation intention and environmental constraint are analyzed using a multi-modal fusion model, target joint motion parameters are generated;If the first parameter based on intention conflicts with the second parameter based on constraint, fusion correction is carried out according to dynamic weight strategy;Servo motor is driven according to final parameter, and torque sensor data is read in real time, and if torque change rate exceeds threshold, the instruction is interrupted and obstacle avoidance fallback program is triggered.The application supports three kinds of interaction modes of voice, gesture and vision, can autonomously perceive user posture, line of sight every 3 seconds and automatically fine tune posture.The application significantly improves the intelligence, safety and naturalness of human-computer interaction through multi-modal fusion and conflict resolution mechanism.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a multi-degree-of-freedom autonomous interactive control method based on multimodal AI fusion, a robotic arm support for implementing the method, and a computer-readable storage medium. Background Technology

[0002] Existing robotic arm support products mostly focus on mechanical structure adjustment, relying primarily on manual knobs, simple remote control, or preset fixed programs. This presents several key pain points: They cannot proactively sense user status or scenario needs, requiring repeated manual adjustments and failing to adapt to dynamic changes in height, posture, and usage scenarios; they lack two-way communication with users, resulting in a rigid user experience; interactive functions are missing, only mechanical adjustments are implemented, such as the display not proactively adapting to the user's viewing angle or adjusting its posture based on user movements; and extended functions are disconnected, lacking integration of power, data, network, audio, and video functions, requiring users to adapt them themselves, significantly reducing compatibility and convenience. Addressing these common industry problems, this invention breaks through the limitations of traditional manual mechanical adjustment of robotic arms, using AI as the core driver to create a multi-degree-of-freedom robotic arm support with autonomous sensing, intelligent interaction, and autonomous adjustment capabilities. Summary of the Invention

[0003] This invention provides a multi-degree-of-freedom autonomous interactive control method based on multimodal AI fusion, applied to a multi-degree-of-freedom robotic arm. The robotic arm includes multiple joints driven by servo motors, and joint encoders and torque sensors disposed at each joint. The method includes the following steps: Multimodal data synchronous acquisition: Acquire audio data streams, visual image data streams, and current pose data of robotic arm joints in the environment; Intent and Constraint Parsing: Feature extraction is performed on the audio stream and video frames using a deep neural network model to generate first motion parameters based on user voice / gestures; simultaneously, the human spinal curvature angle θ is calculated based on the video frames to generate second motion parameters representing environmental / human constraints. Conflict resolution and correction based on posture constraints: When the first motion parameter conflicts with the second motion parameter, a dynamic weighting coefficient α(θ) is calculated based on the spinal curvature angle θ, and the first motion parameter is nonlinearly attenuated to generate the final motion parameter; where α(θ) is a monotonically decreasing nonlinear function with respect to θ. Safety servo drive: Drive the servo motor according to the corrected final motion parameters and monitor the torque sensor data in real time. If the torque change rate exceeds the threshold, obstacle avoidance backtracking will be triggered.

[0004] Furthermore, the conflict resolution and correction based on attitude constraints specifically includes: When the user's voice command is detected to be an intention to raise the robotic arm, but the calculated spinal curvature angle θ is greater than the preset comfort threshold θ threshold At that time, the target height H of the lifting joint target Perform nonlinear attenuation correction. The corrected formula is: , in, To parse the initial height adjustment amount based on the voice command, satisfy: That is, when θ changes from θ threshold Increase to the limiting angle θ max hour, The value smoothly decays from 1.0 to below 0.1. A fine-tuning increment of 0.1 is retained to prevent the robotic arm from completely freezing, providing compliant physical feedback.

[0005] Preferably, the The deep neural network model is implemented using a sigmoid function or a piecewise linear decay function. The deep neural network model is a control network fine-tuned based on a visual-language pre-trained model (VLP) with a Transformer architecture, or a temporal convolutional network (TCN) containing convolutional layers and long short-term memory layers.

[0006] Preferably, the multimodal data synchronization acquisition step further includes: establishing a hardware-triggered timestamp alignment mechanism, using the motion state changes detected by the joint encoder as a benchmark, synchronously triggering the visual camera to capture images, and controlling the timestamp difference between the audio stream and the image stream within ±50ms.

[0007] Furthermore, the obstacle avoidance and backoff procedure in the closed-loop servo drive and safety monitoring steps specifically includes: when the torque change rate ΔT of the i-th joint is detected... i / Δt exceeds the threshold T threshold When the time comes, immediately cut off the enable signal of the joint servo motor, and based on the current position of the adjacent joint, plan a joint space trajectory that allows the end of the robotic arm to smoothly retreat in the reverse direction of the intrusion, and control the robotic arm to retreat a preset distance ΔS along the trajectory.

[0008] This invention also provides a multi-degree-of-freedom autonomous interactive robotic arm for implementing the above-mentioned control method, comprising: a frame; a robotic arm body mounted on the frame, the robotic arm body being a multi-degree-of-freedom serial structure, comprising, in sequence, a base rotation joint, a lifting joint, a telescopic joint, and an end effector pitch joint; a drive module comprising four servo motors for driving the four joints respectively; a sensing module comprising joint encoders and torque sensors disposed on each joint; a perception module comprising a microphone array and a vision camera fixed on the housing of the end effector pitch joint, wherein the main beam direction of the microphone array is parallel to the optical axis direction of the vision camera; a controller disposed on the frame and electrically connected to the drive module, the sensing module, and the perception module, the controller integrating an AI processing chip; and an end effector mounted on the output end of the end effector pitch joint, having a clamp for holding a display terminal. The controller is configured to: execute the multimodal data synchronization acquisition step, receiving data from the sensing module and the perception module; run a multimodal fusion model in the AI ​​processing chip, executing the fusion control parameter generation step and the conflict resolution and parameter correction step, and output the final joint motion parameters; generate a PWM drive signal based on the final joint motion parameters and send it to the servo motor; monitor the data of the torque sensor in real time, and execute the obstacle avoidance and backtracking procedure when conditions are met.

[0009] Specifically, the lifting joint can adopt an electric push rod structure, the telescopic joint adopts a synchronous belt drive structure, and the swing angle range of the end-effector pitch joint is -90° to +90°; the microphone array of the sensing module and the vision camera are integrated into a single module, which is mounted on the housing of the end-effector pitch joint through a pitch adjustment mechanism, so that the main beam direction of the microphone array and the optical axis direction of the vision camera maintain a fixed angle mechanically, which is less than 5°.

[0010] As a further optimization, the end effector also integrates a wireless charging coil; the controller further includes a power management submodule, which is configured to monitor the output power P of the wireless charging coil. out When P out When the power exceeds 80% of its rated power, the computing power allocation of the AI ​​processing chip is dynamically adjusted. Specifically, the operating frequency of the processor core used for running non-real-time visual rendering tasks is reduced, while the operating frequency and priority of the processor core used for real-time servo control and torque monitoring tasks remain unchanged. Furthermore, the power management submodule is configured to prioritize allocating computing resources to the closed-loop control algorithm used to stabilize the attitude of the end effector when a load is detected through the wireless charging coil.

[0011] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method as described in any of the preceding claims.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. Through the synchronous acquisition of multimodal data (audio, video, joint pose) and a deep multimodal fusion model, it is possible to comprehensively and accurately understand the user's operating intentions and the constraints of the surrounding environment, and achieve natural and intelligent human-computer interaction.

[0013] 2. Introduce conflict resolution and dynamic weight adjustment mechanisms. When user intent conflicts with environmental safety (such as uncomfortable posture or proximity to obstacles), the strategy is dynamically adjusted according to user identity, security level, etc., to achieve the best balance between intent and safety, and avoid rigid execution or rejection of instructions.

[0014] 3. Employing a safety monitoring and obstacle avoidance retreat procedure based on torque change rate, it can quickly respond before physical contact or collision occurs, cut off power, and plan a safe retreat path, greatly improving the safety of human-machine collaboration.

[0015] 4. The robotic arm hardware arranges the microphone array and vision camera coaxially / parallel to ensure spatial consistency of perception data; the power management submodule dynamically allocates computing power when wireless charging is under high load to ensure the stability of real-time control tasks.

[0016] 5. With AI as the core, it realizes a closed loop of "perception-decision-execution-feedback-autonomous adaptation", supports three interaction methods: voice, gesture and vision, and can actively sense the user's status (sitting posture, line of sight, distance) every 3 seconds and automatically fine-tune the posture to always maintain the best use position of the monitor, truly achieving manual and fully intelligent autonomous interaction. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a perspective view of the mechanical structure of Embodiment 1 of the present invention; Figure 3 This is a perspective view of the mechanical structure of Embodiment 2 of the present invention; Figure 4 This is a control flowchart of the present invention; Figure 5 This is a schematic diagram of the conflict resolution algorithm of the present invention.

[0018] In the diagram: 1. Base rotation joint; 2. Lifting joint; 3. Telescopic joint; 4. End effector pitch joint; 5. Pitch adjustment mechanism; 6. End effector. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1: Control Method like Figures 1 to 5 As shown, this embodiment provides a multi-degree-of-freedom autonomous interactive control method based on multimodal AI fusion, applied to a four-degree-of-freedom robotic arm support. The base rotation joint 1, lifting joint 2, telescopic joint 3, and end effector pitch joint 4 of the robotic arm are all driven by servo motors, and each joint is equipped with a joint encoder and a torque sensor.

[0021] Step S1: Synchronous acquisition of multimodal data The microphone array at the end joint acquires ambient audio streams at a sampling rate of 16kHz, while the vision camera captures video frames at 30fps. The system establishes a hardware triggering mechanism: using the moment when the joint encoder detects the motor crossing zero point as the reference clock, the camera is synchronously triggered to capture a frame image, and the same timestamp group is added to the frame image and the audio data packets within 100ms before and after it, ensuring that the difference in all data timestamps is controlled within ±50ms.

[0022] Step S2: Generation of fusion control parameters The synchronized audio stream, video frames, and current joint pose data are input into a pre-trained deep neural network model. Specifically, the deep neural network model is a cross-modal alignment network based on the Transformer architecture. This network first extracts the speech semantic feature vector V through an audio encoder. audio Human pose feature vector V is extracted through a visual encoder. poseThe model then uses cross-attention to calculate the coupling degree between intention and constraint. When the coupling degree between the intention command (e.g., 'raise') and the constraint state (e.g., 'bend over') exceeds a preset threshold, the conflict resolution module is activated, and the corrected motion parameters are output. The model recognizes voice commands from audio (e.g., "adjust the monitor to a comfortable position") and identifies user gestures, facial orientation, key points of the human skeleton, and spinal curvature angle θ from video frames. Based on this information, the model parses the operation intention, such as "raise the monitor to eye level." Simultaneously, the model calculates environmental constraints based on depth information in the video frames and the current robotic arm pose, such as an obstacle 20cm in front of the end effector or a spinal curvature angle θ of 25 degrees (the preset comfort threshold is 20 degrees). The model initially generates the first motion parameter (target height increase + 15cm, medium speed) and the second motion parameter (target height increase + 3cm, slow speed).

[0023] The "deep multimodal fusion model" described in this invention is not limited to a single network structure. Instead, it can be implemented using various specific neural network architectures depending on the real-time requirements, computing resources, and interaction complexity of the actual application scenario. Two preferred embodiment architectures are listed below: Example 1: Static / Quasi-Static Scene Processing Model Based on Transformer Architecture This embodiment is applicable to interactive scenarios where the user's position is relatively fixed and the focus is on semantic understanding (such as adjusting office posture). The multimodal fusion model is obtained by fine-tuning a vision-language pre-trained model (VLP) based on the Transformer architecture.

[0024] Specifically, the model includes a visual encoder (such as Vision Transformer, ViT) and a text / audio encoder (such as BERT or Wav2Vec).

[0025] Input processing: The visual encoder receives single-frame or multi-frame stitched human pose images, segments them into image patches and embeds them linearly; the audio encoder receives the spectral features of speech commands.

[0026] Feature fusion: By introducing a cross-attention mechanism, audio features are used as queries, and visual features are used as keys and values ​​to calculate a cross-modal correlation matrix. This enables the model to accurately locate the specific body parts (such as the neck or waist) corresponding to voice commands (such as "raise a little higher") in images.

[0027] Advantages: This architecture leverages the global receptive field of the Transformer, enabling it to understand the spatial relationship between complex semantic instructions and human posture very well, making it suitable for handling high-precision static position control.

[0028] Example 2: Dynamic Temporal Scene Processing Model Based on CNN-LSTM This embodiment is applicable to interactive scenarios where the user is in motion and the focus is on the continuity of actions and temporal prediction (such as object delivery scenarios that follow the user's movement). The multimodal fusion model adopts a hybrid architecture of convolutional neural network and long short-term memory network (CNN-LSTM).

[0029] Specifically, the model consists of a spatial feature extraction module and a temporal decision module connected in series: Spatial feature extraction: A convolutional neural network (such as ResNet-18 or MobileNet) is used as the backbone network to extract features from each frame of the video stream and output a feature vector containing the coordinates of human key points and environmental depth information.

[0030] Timing decision: Using consecutive time steps The feature vectors are input into a Long Short-Term Memory (LSTM) network. By utilizing the gating mechanism (input gate, forget gate, output gate) within the LSTM, the model can memorize the user's historical movement trajectory and filter out instantaneous background noise interference (such as passersby).

[0031] Smooth output: The hidden layer state output of LSTM is regressed through a fully connected layer to directly predict the joint velocity or acceleration of the robotic arm in the next moment.

[0032] Advantages: By introducing a temporal dimension, this architecture effectively solves the "jitter" problem of the robotic arm during dynamic following, ensuring the smoothness and safety of the movement.

[0033] Example 3: Model Selection Strategy In a real system, the main control chip can dynamically load the above models according to the current task mode. For example, when the user is detected to be in "voice command mode", the Transformer model of Example 1 is activated to obtain high-precision positioning; when the user is detected to be in "gesture guidance mode" or "automatic follow mode", the CNN-LSTM model of Example 2 is switched to obtain low-latency dynamic response.

[0034] The deep neural network model used in this embodiment is a modified ViLT (Vision-and-Language Transformer) network based on the Transformer architecture. The audio signal is first converted into an ASR (Automatic Sequence of Text), and then used as the text input modality of the ViLT model.

[0035] 1. Network parameter configuration: Visual encoder: It adopts a lightweight ViT-Base architecture with an image patch size of 16x16, a hidden layer dimension of 768, 12 attention heads, and 12 layers.

[0036] Text encoder: Based on BERT-Base fine-tuning, with a maximum sequence length of 64.

[0037] Fusion layer: Introduces a cross-modal attention module, which contains 3 Transformer blocks to calculate the coupling degree between the speech intent vector and the visual pose vector.

[0038] Output layer: A fully connected layer (FC Layer) is used to map to the 4D joint space (rotation, lift, telescoping, pitch).

[0039] 2. Training dataset construction: To train the above model, this invention constructs a dedicated "Human-Computer Interaction Pose-Command Dataset (HRI-Pose-Command Dataset)".

[0040] Data scale: Includes 50 subjects of different body types (covering different heights and weights), and a total of 12,000 sets of synchronous multimodal samples were collected.

[0041] Data composition: Each sample group contains: (1) MFCC features of voice commands (such as “raise a little”, “too low”); (2) RGB video frames at 30fps and corresponding 2D skeletal key points extracted by OpenPose; (3) labeled data, including user intent (raise / lower / keep) and human spinal curvature angle θ (calibrated by Vicon motion capture system, range 0°-60°).

[0042] Labeling rule: When θ < 20 ∘ When the label is "Comfort State - Execute Commands"; when 20 ∘ ≤θ<40 ∘ When θ ≥ 40, the label is "Warning State - Decay Execution"; ∘ At that time, the label is "dangerous state - rejection / fine-tuning".

[0043] 3. Training process: The model was trained end-to-end using the mean squared error loss function (MSE Loss). After sufficient training, the model's intent recognition accuracy on the test set stabilized at over 96%.

[0044] Step S3: Conflict resolution and parameter adjustment The system determines that there is a conflict between the first and second motion parameters. Based on the current user's identity (elderly) and security level (high), a weighting coefficient α = 0.6 is set. Furthermore, since the environmental constraints include a human spinal curvature angle θ = 28°, a nonlinear attenuation correction formula is used: , ,set up +50mm, Using the Sigmoid function, the result is calculated as follows: (28°)≈0.3, so the final height increase is only 15mm. At the same time, if the user's voice command "raise 20cm" conflicts with the sitting posture of bending over, the system will automatically reduce the command amplitude and provide a voice prompt "You are being bent over, and we have adjusted you to a safe height".

[0045] Step S4: Closed-loop servo drive and safety monitoring The controller generates a PWM waveform to drive the servo motor based on the corrected final joint motion parameters. During motion, it reads data from the torque sensors of each joint at a frequency of 1kHz. When the rate of change of torque ΔT at the telescopic joint is detected... i / Δt exceeds T threshold When the threshold of 500 N·m / s is reached, the joint enable signal is immediately cut off. At the same time, a joint space trajectory of 100mm reverse retraction is planned based on the current position of the adjacent joints to control the smooth retraction of the robotic arm.

[0046] Example 2: Robotic Arm Structure and Interaction Process like Figure 2 and Figure 3 As shown, this embodiment provides a multi-degree-of-freedom autonomous interactive robotic arm support for implementing the above control method, specifically used to support a display and integrate wireless charging functionality for mobile phones.

[0047] Robotic arm structure: aluminum alloy frame; multi-degree-of-freedom serial body: base rotary joint (0-340°), electric push rod lifting joint (300mm stroke), synchronous belt telescopic joint (400mm stroke), end effector pitch joint (-90° to +90°). The drive module includes four servo motors and drivers. The sensing module includes an absolute encoder and a strain gauge torque sensor. The perception module is an integrated module, mounted on the end effector pitch joint housing via a pitch adjustment mechanism 5, containing a four-microphone linear array (main beam forward) and an RGB-D camera (optical axis forward), both mounted parallel to each other with a fixed angle of less than 3°. The end effector 6 is a gripper with rubber pads for holding a tablet / monitor, and integrates a wireless charging coil. The controller is built into the frame and contains a multi-core AI processing chip.

[0048] The controller is configured to execute the following autonomous interaction process: Perception layer: The AI ​​autonomous perception module collects voice signals, gesture images, and visual scene data (human eyes / face / sitting posture) in real time, and converts them into structured data in a unified manner through a multimodal data fusion algorithm.

[0049] Decision-making layer: Based on fused data and a preset "optimal user usage model" (e.g., the center of the display is aligned with the user's eyes, and the line-of-sight deviation is ≤5°), the AI ​​decision-making unit autonomously generates adjustment commands. Every 3 seconds is a perception cycle, continuously updating the user's status. If a change in posture (bending over / sitting upright / leaning back), a shift in gaze, or a change in distance is detected, the robotic arm's posture is autonomously fine-tuned. For example: when the user is detected bending over, the height of joint 2 is autonomously lowered by 10-15cm, and the pitch angle of joint 4 is fine-tuned by +15°; when the user is detected standing up, the height of joint 2 is automatically raised by 15-20cm, and the pitch angle of joint 4 is adjusted by -10°.

[0050] Voice control: Users activate the AI ​​voice module using a wake word (such as "Xiao Zhi Assistant"), and the AI ​​voice responds, "I'm here, how can I help you?" The AI ​​voice recognition algorithm is based on the Transformer deep learning model and supports fuzzy commands (such as "Adjust the monitor to a position that is comfortable for me to look at"). After parsing, it executes the corresponding joint movement and provides proactive voice feedback after the movement is completed.

[0051] Gesture Control: A gesture-sensing camera captures user gesture images, and an AI gesture recognition algorithm (based on CNN+YOLO) extracts features, supporting over 50 preset gestures with a recognition response time of ≤0.2s. Gesture-Joint Mapping Rules: Sliding the palm up and down controls the lifting joint (the sliding amplitude is proportional to the lifting distance), waving left and right controls the rotation joint, clenching and releasing the fist controls the extension joint, and flipping the thumb up and down controls the pitch joint. Dynamic gestures, such as drawing "↑", quickly raise the hand, and drawing "←", rotate the hand to the left. AI automatically adjusts the movement speed based on hand distance and gesture amplitude, and monitors obstacles in real time to avoid risks.

[0052] Expansion Interface Integration: Includes DC power input, USB-A, USB-C, RJ45 Ethernet port, HDMI, DP, 3.5mm audio jack, and wireless charging module. The power management submodule monitors the wireless charging output power; when the power exceeds 80% of the rated power, it reduces the frequency of the non-real-time visual rendering core to ensure the priority of the real-time servo control and torque monitoring cores remains unchanged. When a load is detected, it prioritizes allocating computing resources to the closed-loop control algorithm for stabilizing the end effector's attitude.

[0053] Through the aforementioned closed-loop control of "perception-decision-execution-feedback-re-adaptation", this robotic arm support achieves fully intelligent, manual-free, safe and natural autonomous interaction.

[0054] Example 3: Storage Medium This embodiment provides a computer-readable storage medium, such as a non-volatile memory (FLASH, EEPROM), a magnetic disk, or an optical disk. A computer program is stored thereon, which, when loaded and executed by a processor (such as a controller for a robotic arm or a host computer), can implement all or part of the steps in the control method described in Embodiment 1 or Embodiment 2.

[0055] The above description is merely a preferred embodiment of the present invention and does not limit the scope of the patent. Any equivalent structural or procedural modifications made based on the content of this specification, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this invention.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-degree-of-freedom autonomous interactive control method based on multimodal AI fusion, characterized in that, The method includes the following steps: Multimodal data synchronous acquisition: Acquire audio data streams, visual image data streams, and current pose data of robotic arm joints in the environment; Intent and Constraint Parsing: Feature extraction is performed on the audio stream and video frames using a deep neural network model to generate first motion parameters based on user voice / gestures; simultaneously, the human spinal curvature angle θ is calculated based on the video frames to generate second motion parameters representing environmental / human constraints. Conflict resolution and correction based on posture constraints: When the first motion parameter conflicts with the second motion parameter, a dynamic weighting coefficient α(θ) is calculated based on the spinal curvature angle θ, and the first motion parameter is nonlinearly attenuated to generate the final motion parameter; where α(θ) is a monotonically decreasing nonlinear function with respect to θ. Safety servo drive: Drive the servo motor according to the corrected final motion parameters and monitor the torque sensor data in real time. If the torque change rate exceeds the threshold, obstacle avoidance backtracking will be triggered.

2. The control method according to claim 1, characterized in that, The conflict resolution and correction based on attitude constraints specifically includes: When the user's voice command is detected to be an intention to raise the robotic arm, but the calculated spinal curvature angle θ is greater than the preset comfort threshold θ threshold At that time, the target height H of the lifting joint target Perform nonlinear attenuation correction. The corrected formula is: ， in, To parse the initial height adjustment amount based on the voice command, satisfy: That is, when θ changes from θ threshold Increase to the limiting angle θ max hour, The value smoothly decreases from 1.0 to below 0.

1.

3. The control method according to claim 2, characterized in that, The The deep neural network model is implemented using a sigmoid function or a piecewise linear decay function. The deep neural network model is a control network fine-tuned based on a visual-language pre-trained model (VLP) with a Transformer architecture, or a temporal convolutional network (TCN) containing convolutional layers and long short-term memory layers.

4. The control method according to claim 1, characterized in that, The multimodal data synchronization acquisition step further includes: establishing a hardware-triggered timestamp alignment mechanism, using the motion state changes detected by the joint encoder as a benchmark, synchronously triggering the visual camera to capture images, and controlling the timestamp difference between the audio stream and the image stream within ±50ms.

5. The control method according to claim 4, characterized in that, The preset time tolerance range is ±50ms.

6. The control method according to claim 1, characterized in that, The obstacle avoidance and backoff procedure in the servo drive and safety monitoring steps specifically includes: when the torque change rate ΔT of the i-th joint is detected... i / Δt exceeds the threshold T threshold When the time comes, immediately cut off the enable signal of the joint servo motor, and based on the current position of the adjacent joint, plan a joint space trajectory that allows the end of the robotic arm to smoothly retreat in the reverse direction of the intrusion, and control the robotic arm to retreat a preset distance ΔS along the trajectory.

7. A multi-degree-of-freedom autonomous interactive robotic arm, characterized in that, include: frame; The robotic arm body is mounted on the frame. The robotic arm body is a multi-degree-of-freedom serial structure, which includes, in sequence, a base rotation joint, a lifting joint, a telescopic joint, and an end-effector pitch joint. The drive module includes four servo motors that drive the four joints respectively; The sensing module includes joint encoders and torque sensors installed at each joint; The sensing module includes a microphone array and a vision camera fixed to the housing of the end-effector pitch joint, wherein the main beam direction of the microphone array is parallel to the optical axis direction of the vision camera; The controller is mounted on the rack and is electrically connected to the drive module, the sensing module and the perception module. The controller integrates an AI processing chip. An end effector, installed at the output end of the end pitch joint, is provided with a clamp for holding the display terminal; The controller is configured as follows: Perform the multimodal data synchronous acquisition step to receive data from the sensing module and the perception module; The multimodal fusion model is run in the AI ​​processing chip, and the fusion control parameter generation step and conflict resolution and parameter correction step are executed to output the final joint motion parameters. A PWM drive signal is generated based on the final joint motion parameters and sent to the servo motor; The data from the torque sensor is monitored in real time, and the obstacle avoidance and retraction procedure is executed when the conditions are met.

8. The multi-degree-of-freedom autonomous interactive robotic arm according to claim 7, characterized in that, The lifting joint adopts an electric push rod structure, the telescopic joint adopts a synchronous belt drive structure, and the swing angle range of the end pitch joint is -90° to +90°. The microphone array of the sensing module is integrated with the visual camera into a single module. This module is mounted on the housing of the end-effector pitch joint via a pitch adjustment mechanism, so that the main beam direction of the microphone array and the optical axis direction of the visual camera maintain a fixed angle mechanically, which is less than 5°.

9. The multi-degree-of-freedom autonomous interactive robotic arm according to claim 7, characterized in that, The end effector also integrates a wireless charging coil; The controller further includes a power management submodule, which is configured to: Monitor the output power P of the wireless charging coil out ; When P out When the power exceeds 80% of its rated power, the computing power allocation of the AI ​​processing chip is dynamically adjusted. Specifically, the operating frequency of the processor core used for running non-real-time visual rendering tasks is reduced, while the operating frequency and priority of the processor core used for real-time servo control and torque monitoring tasks remain unchanged.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the control method as described in any one of claims 1-6.

Images