Intelligent cockpit mechanical arm system

Abstract

This invention relates to the field of automotive electronics and intelligent control technology, and discloses an intelligent cockpit robotic arm system, including: a foldable robotic arm body installed in the cockpit, an end-effector sensing and interaction module, a driver state monitoring subsystem, and a central control unit. The central control unit controls the robotic arm's retraction or operating mode based on the driver's comprehensive fatigue index. In operating mode, visual gaze information is used as a gating signal to activate EEG analysis, and the operational intent is confirmed by combining visual prior probability with EEG signals. The robotic arm's impedance parameters are adjusted in real time based on the driver's fatigue state. This invention solves the problem of unstable single-modal control through Bayesian fusion of visual and EEG signals; by utilizing variable impedance control and momentum observer collision detection, it achieves active safety protection based on the driver's physiological state, providing a seamless and highly safe intelligent assistance experience within the limited cockpit space.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics and intelligent control technology, specifically to an intelligent cockpit robotic arm system. Background Technology

[0002] As car cabins become increasingly intelligent, drivers are increasingly needing to perform non-driving tasks while driving, such as operating central control devices or retrieving items. These actions can distract drivers and increase driving safety risks. To assist drivers in these tasks, the industry has begun to explore the introduction of robotic arm technology into the in-vehicle environment, using automation to reduce driver workload. Current technologies include robotic arm solutions that utilize visual positioning or EEG signal control. For example, binocular cameras capture environmental images to calculate the coordinates of target objects, or EEG acquisition devices control the movement of the robotic arm, aiming to achieve assisted grasping in specific scenarios.

[0003] However, existing robotic arm technology has significant limitations when applied to the confined and complex environment of a car cabin. Firstly, there is the issue of space adaptability. Traditional robotic arm designs primarily focus on operating range and load capacity, resulting in a large size that is difficult to store effectively even when not in use. This often occupies valuable storage or passenger space within the cabin, failing to balance the conflict between deployment and concealed storage.

[0004] Furthermore, there is a serious disconnect between the control logic of the existing system and the driver's status monitoring function. Although current vehicles are generally equipped with fatigue monitoring systems, they can usually only issue alarms and cannot be linked with the actuators of the on-board equipment; while the existing robotic arm control system often operates independently and cannot sense the driver's current physiological state (such as fatigue level), which means that the robotic arm cannot adjust its movement speed or contact stiffness according to the driver's reaction ability when performing actions, posing a physical safety hazard in the human-machine co-driving environment.

[0005] Finally, in terms of interaction methods, existing technologies still primarily rely on explicit voice commands or touch operations to trigger robotic arm movements. When a driver is under high stress or fatigue, being distracted to issue voice commands or search for touch buttons is inherently unsafe. Although brain-controlled technologies exist, relying solely on EEG signals often results in poor stability, low recognition rates, and a lack of effective integration with other perceptual modalities (such as gaze). This makes it difficult for the system to accurately understand the driver's intentions without explicit commands, leading to a poor user experience in practical applications and failing to meet the demands of intelligent cockpits for proactive, seamless services. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent cockpit robotic arm system that solves the problems of low accuracy in intention recognition, insufficient physical safety, and difficulty in adapting to changes in the driver's physiological state during human-computer interaction.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an intelligent cockpit robotic arm system, comprising a foldable robotic arm body installed in a vehicle cockpit, with a drive module at its joints; a perception and interaction module located at the end of the robotic arm, the module including a visual acquisition unit for collecting environmental data and a display and interaction unit; and a driver state monitoring subsystem, including an EEG acquisition device located in the steering wheel area and an infrared camera located in front of the cockpit. The system also includes a central control unit electrically connected to the foldable robotic arm body, the perception and interaction module, and the driver state monitoring subsystem.

[0008] The central control unit operates an active control logic based on the fusion of physiological signals and environmental perception: First, it calculates a comprehensive fatigue index based on data from the driver's state monitoring subsystem, and determines whether the foldable robotic arm should enter working mode or remain in a retracted standby state based on this index. Upon entering working mode, the system does not immediately respond to EEG signals, but instead uses gaze information acquired by the visual acquisition unit as a gating signal, activating the EEG signal analysis function only when the gaze information meets preset conditions. Subsequently, the system combines visual prior probabilities with EEG signal classification results to confirm the operational intent and controls the foldable robotic arm to perform the corresponding action. This hierarchical control architecture ensures that the system only initiates intent recognition when the driver has clear visual attention, effectively filtering out invalid EEG fluctuations.

[0009] In a preferred embodiment, regarding the calculation of the comprehensive fatigue index, the central control unit receives facial image data captured by an infrared camera and calculates the PERCLOS value and blink frequency per unit time. The system constructs a normalized model and performs a linear weighted calculation on the PERCLOS value and blink frequency to generate the comprehensive fatigue index. When this index exceeds a preset trigger threshold, or when the system receives an active interaction command, it determines that the system has entered the working mode. This achieves a seamless switch between passive wake-up based on the driver's actual physiological load and active service.

[0010] For the generation of visual gating signals, the central control unit uses data from the visual acquisition unit to construct a 3D environment map of the cockpit and generate bounding boxes for each object. By calculating the driver's gaze direction vector in real time, the system calculates the spatial intersection of the gaze direction vector and the bounding box, and also calculates the continuous dwell time of the gaze within any object's bounding box. Only when this continuous dwell time exceeds a preset dwell time threshold does the system generate a gating activation signal to allow EEG signals to enter the control loop, thereby using a time integration mechanism to eliminate false triggers caused by gaze saccades.

[0011] In the intent confirmation stage, the central control unit employs a Bayesian fusion strategy. Upon receiving a gating activation signal, it extracts the frequency domain features of the EEG signals acquired by the EEG acquisition device and outputs the EEG intent probability using a Long Short-Term Memory (LSTM) network classification model. Simultaneously, it identifies the functional attributes of the object that triggered the gating activation signal and generates a visual prior probability based on these attributes. The posterior intent probability is calculated by fusing the EEG intent probability and the visual prior probability using a Bayesian formula. If this probability exceeds a preset confirmation threshold, the target object and corresponding action are locked. This method utilizes visual scene information to constrain the uncertainty of EEG classification, improving the robustness of intent recognition.

[0012] To enhance the physical safety of human-machine co-driving, the central control unit also implements variable impedance control. The system establishes a mapping relationship between physiological state and impedance parameters, and adjusts the Cartesian stiffness matrix and damping matrix of the folding robotic arm's end effector in real time based on the aforementioned calculated comprehensive fatigue index. Specifically, as the comprehensive fatigue index increases, the stiffness matrix parameter value is reduced to decrease contact force, and the damping matrix parameter value is increased to suppress motion overshoot, resulting in a more compliant and safer movement characteristic of the robotic arm when the driver is fatigued.

[0013] In terms of motion planning and obstacle avoidance, the central control unit constructs an obstacle avoidance environment model based on an artificial potential field algorithm. The bounding boxes of objects identified by the perception and interaction module are used as obstacle sources to establish a repulsive potential field, while the target position is set as an attractive source to establish a gravitational potential field. The two are superimposed to form the total potential field. By calculating the negative gradient of the total potential field, the collision-free motion velocity vector of the folding robotic arm is planned. This velocity vector is used as the desired motion input, combined with the adjusted stiffness and damping matrices, and the driving torque commands for each joint are generated through an impedance control law, achieving compliant obstacle avoidance.

[0014] To ensure smooth motion, the central control unit uses a minimum jerk criterion to interpolate the motion path points when controlling the movement of the folding robotic arm, generating a position trajectory based on a fifth-order polynomial. This position trajectory has zero velocity and acceleration at the start and end of the motion, and the jerk remains continuous in the time domain, eliminating mechanical shocks during the start-up, stopping, and operation of the robotic arm.

[0015] In addition, the system also features tactile feedback. When a task is detected as completed or a confirmation command is received, the central control unit generates a high-frequency, low-amplitude tactile feedback force signal and adds it to the drive torque, causing the robotic arm end to vibrate as a notification. This signal uses a decaying sine wave model to provide the driver with intuitive, non-visual confirmation of the status.

[0016] In terms of collision safety protection, the central control unit operates collision detection logic based on a generalized momentum observer. A momentum observation residual vector characterizing the external torque is constructed based on the dynamic model of the folding robotic arm, and its modulus length is monitored in real time. When the modulus length exceeds a preset safety threshold, a collision is determined, and the system immediately controls the drive module to switch to zero-gravity mode, outputting only gravity compensation torque, placing the robotic arm in a passive compliant state to minimize collision damage.

[0017] In terms of specific hardware configuration, the vision acquisition unit uses a 3D-ToF camera to acquire depth information; the display interaction unit uses a flexible OLED display to adapt to curved structures; the EEG acquisition device includes a non-invasive dry electrode array integrated into a specific grip area of ​​the steering wheel to achieve concealed signal acquisition; the central control unit reads real-time driving status data of the vehicle through the vehicle CAN bus interface, and uses this data to perform time synchronization and alignment of the acquired physiological signals to ensure the spatiotemporal consistency of multimodal data.

[0018] This invention provides an intelligent cockpit robotic arm system. It has the following beneficial effects:

[0019] 1. This invention uses visual gaze information as the gating signal for the brain-computer interface and leverages Bayesian formulas to deeply fuse EEG intention probabilities with visual prior probabilities. This effectively solves the technical challenge of being easily interfered with and falsely triggered when relying solely on EEG signals for control. This mechanism ensures that the system only locks onto the target when the driver's gaze is fixed for an extended period and the EEG characteristics are clear. This improves the accuracy and anti-interference capability of intention recognition in complex cockpit environments without the need for physical contact or voice commands.

[0020] 2. This invention establishes a dynamic mapping relationship between the driver's comprehensive fatigue index and the impedance parameters of the robotic arm's end effector, realizing variable stiffness control based on physiological state. When the system detects that the driver is in a state of high fatigue, it can automatically reduce the Cartesian stiffness of the robotic arm and increase the damping, making it exhibit compliant characteristics with low force. This active adjustment strategy can not only prevent hard collision injuries caused by the driver's slow reaction, but also suppress oscillations during movement by increasing damping, thereby improving physical safety in the human-machine co-driving environment.

[0021] 3. This invention combines fifth-order polynomial trajectory planning based on the minimum jerk criterion with sensorless collision detection logic based on a generalized momentum observer, optimizing the motion quality of the robotic arm within the confined cabin space. Continuous jerk planning eliminates mechanical shocks and vibrations during the start and stop of the robotic arm, while the momentum observer allows the system to detect accidental contact in real time without the addition of external torque sensors and immediately switch to a zero-gravity passive compliance state, effectively protecting the safety of occupants and equipment. Attached Figure Description

[0022] Figure 1 This is a system architecture diagram of an intelligent cockpit robotic arm system according to an embodiment of the present invention;

[0023] Figure 2 This is a flowchart illustrating an intelligent cockpit robotic arm method according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the confusion matrix generated during the intent recognition test in the comparative experiment;

[0025] Figure 4 This is a schematic diagram comparing the end-point normalized displacement trajectory of the embodiments of the present invention with that of the prior art in motion planning;

[0026] Figure 5 This is a schematic diagram comparing the jerk curves in motion planning of the embodiments of the present invention and the prior art.

[0027] Among them, 100 is the folding robotic arm body; 200 is the perception and interaction module; 201 is the vision acquisition unit; 202 is the display and interaction unit; 300 is the central control unit; 400 is the driver status monitoring subsystem; 401 is the EEG acquisition device; and 402 is the infrared camera. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] See attached document Figure 1 The present invention provides an intelligent cockpit robotic arm system, which includes a foldable robotic arm body 100, a perception and interaction module 200, a central control unit 300, and a driver status monitoring subsystem 400.

[0030] The foldable robotic arm body 100 is installed inside the vehicle cabin, serving as the physical actuator of the system. It employs a multi-level nested telescopic structure, enabling it to switch between an extended working state and a retracted standby state through mechanical retraction and joint folding. Drive modules are located at the joints of the foldable robotic arm body 100 to drive the arm to perform multi-degree-of-freedom movements within the cabin space. The perception and interaction module 200 is integrated at the end of the foldable robotic arm body 100, comprising a vision acquisition unit 201 and a display and interaction unit 202. The vision acquisition unit 201 uses a 3D-ToF camera to acquire environmental depth image data and object point cloud data within the cabin. The display and interaction unit 202 uses a flexible OLED display screen to provide feedback on system status information. The driver status monitoring subsystem 400 includes an EEG acquisition device 401 located in the steering wheel area and an infrared camera 402 located in front of the driver's cabin. The EEG acquisition device 401 uses non-invasive dry electrodes to acquire the driver's brain signals and transmit them to the central control unit 300. The infrared camera 402 is used to acquire real-time facial feature data of the driver. The central control unit 300 is communicatively connected to the folding robotic arm body 100, the perception and interaction module 200, and the driver status monitoring subsystem 400, and reads the real-time driving status data of the vehicle through the vehicle CAN bus interface.

[0031] See attached document Figure 2 This invention provides a method for controlling a robotic arm in an intelligent cockpit, comprising the following steps:

[0032] S10, the driver status monitoring subsystem 400 continuously collects driver physiological status data, the infrared camera 402 acquires facial images to calculate PERCLOS value and blink frequency, and the EEG acquisition device 401 is in standby or low-power listening state; at the same time, the central control unit 300 monitors vehicle driving status data through the vehicle CAN bus.

[0033] S20, the central control unit 300 calculates the comprehensive fatigue index based on the PERCLOS value and blink frequency, and compares the fatigue index with a preset threshold; if the fatigue index is lower than the trigger threshold and there is no external interaction command, the foldable robotic arm body 100 remains in a storage standby state; if the fatigue index exceeds the trigger threshold or an active interaction request is received, the system enters the working mode.

[0034] S30, in working mode, executes EEG signal gating logic based on visual gaze; the visual acquisition unit 201 tracks the driver's gaze vector and calculates its spatial intersection with objects in the cockpit. When the gaze vector falls into a preset region of interest and the dwell time meets the preset conditions, the EEG signal analysis function is activated, and the operation intention is confirmed by combining the visual prior probability and the EEG classification result.

[0035] S40, the central control unit 300 plans the obstacle avoidance motion path of the folding robotic arm body 100 using an artificial potential field algorithm based on the confirmed operation intention and the three-dimensional environment map constructed by the vision acquisition unit 201; at the same time, based on the fatigue index calculated in step S20, the stiffness parameters and damping parameters of the end of the folding robotic arm body 100 are dynamically calculated using a variable impedance control model.

[0036] S50, the central control unit 300 controls the folding robotic arm body 100 to perform the task of grasping or delivering items according to the planned path and the calculated impedance parameters, and controls the display interaction unit 202 to display the preset feedback screen; during the execution, if a vehicle emergency braking signal is detected, the movement range of the folding robotic arm body 100 is immediately adjusted.

[0037] See attached document Figure 2 In step S10 of the intelligent cockpit robotic arm control method, the system performs synchronous acquisition of multi-source physiological and vehicle state data. This step specifically includes the following sub-steps:

[0038] An infrared camera 402 located in front of the cockpit collects facial image data of the driver. The infrared camera 402 is equipped with an infrared illumination unit, enabling it to adapt to changes in in-vehicle lighting conditions and continuously capture image sequences containing the driver's eye area at a frame rate of 30fps to 60fps. The central control unit 300 receives these image sequences and applies a facial landmark detection algorithm to extract the coordinates of the upper and lower edges of the eyelids. The system calculates the eye opening degree based on the Euclidean distance between the upper and lower edge coordinates, and combines the opening degree data from consecutive frames to establish a time-series data of the eye opening and closing state. For the specific algorithm implementation of facial landmark detection, those skilled in the art can use existing cascaded regression trees or deep convolutional neural network models, which are well-known technologies in the field and will not be elaborated upon here.

[0039] An EEG acquisition device 401, located in the steering wheel area of ​​a vehicle, collects the driver's EEG signals. The EEG acquisition device 401 includes a non-invasive dry electrode array integrated into the three o'clock and nine o'clock grip areas of the steering wheel. This placement corresponds to the contact area of ​​the driver's hands in a standard driving posture, ensuring stable signal acquisition. When the driver's skin contacts the steering wheel grip area, the non-invasive dry electrodes acquire microvolt-level EEG potential signals through capacitive coupling or direct contact. The EEG acquisition device 401 integrates an analog front-end circuit that amplifies, converts, and bandpass filters the raw analog signal. The bandpass filter cutoff frequency range is set from 0.5Hz to 50Hz to effectively filter out power frequency interference and high-frequency electromyographic noise, outputting a digitized raw EEG signal sequence. .

[0040] The central control unit 300 reads vehicle driving status data in real time via the vehicle's CAN bus interface. This driving status data includes the vehicle's longitudinal speed. Longitudinal acceleration lateral acceleration and the vibration frequency of the vehicle's suspension system The above parameters reflect the current physical environment excitation. The central control unit 300 uses the above vehicle driving status data as system environment parameters for the dynamic adjustment of the robotic arm and the triggering judgment of safety protection logic in subsequent steps.

[0041] The central control unit 300 performs time synchronization and alignment on the aforementioned collected facial image data, electroencephalogram (EEG) signal sequences, and vehicle driving status data. The system establishes a unified time reference. Considering the frequency of visual data acquisition Sampling frequency of EEG data There are differences and usually The central control unit 300 uses a linear interpolation algorithm, taking the acquisition timestamp of the visual data as the alignment reference, to map multi-source data of different frequencies onto the same time axis, forming a synchronization state vector. .

[0042] ;

[0043] in, Indicates time Facial feature data, Indicates time EEG signal data, , , Representing time respectively The corresponding vehicle speed, longitudinal acceleration, and lateral acceleration components. Through the above steps, the system has completed the digital reconstruction of the driver's physiological state and the vehicle's physical environment, providing an input benchmark for subsequent fatigue assessment and active intervention.

[0044] See attached document Figure 2 In step S20 of the intelligent cockpit robotic arm control method, the system calculates a comprehensive fatigue index based on the processed physiological data and determines the system's operating mode based on this index. This step specifically includes the following sub-steps:

[0045] The central control unit 300 calculates the PERCLOS value per unit time based on the time-series data of eye opening and closing obtained in step S10. The PERCLOS value, as a key indicator characterizing the delayed eyelid closure caused by driver drowsiness, is calculated based on a preset time reference. In this embodiment, the system sets a sliding time window. The sliding time window The value range is set to 60 to 180 seconds. Within this window period, the system counts the total number of image frames where the eyelid closure exceeds 80%. PERCLOS value Defined as the number of frames that satisfy the above closing condition relative to the sliding time window. The proportion of the total number of frames.

[0046] The central control unit 300 synchronously calculates the driver's blink frequency. The system detects the open-close-open sequence of eye opening and closing state transitions in the eye opening and closing data, and statistically analyzes the sliding time window. The total number of blinks within a given timeframe is counted, and the blink frequency is calculated accordingly. This indicator is used to reflect the driver's current level of visual attention and neural excitability.

[0047] The central control unit 300 constructs a normalized comprehensive fatigue index model. To map physiological characteristics from different physical dimensions to a unified evaluation system for comprehensive assessment, the system employs a linear weighted normalization method, combined with PERCLOS values. With blinking frequency Generate comprehensive fatigue index The calculation model for this index is as follows:

[0048] ;

[0049] in, The normalized comprehensive fatigue index has a range of values. and Let be the weight coefficient, and satisfy... In this embodiment, considering the significance of the PERCLOS value in fatigue characterization, Set to 0.6, Set to 0.4; and These are the preset lower and upper limits for PERCLOS value normalization. In this embodiment, The value is set to 0.02, representing the baseline for a conscious state. The value is set to 0.4, representing the baseline for severe fatigue. and These are the preset lower and upper limits for normalized blink frequency, respectively. In this embodiment... Set to 5 times / minute. The setting is 30 times per minute. The above upper and lower limits are empirical constants built into the system. In specific application scenarios, personalized benchmark values ​​for specific drivers can also be obtained through preset calibration programs.

[0050] The central control unit 300 will calculate the comprehensive fatigue index. The system's operating mode is determined by comparing the calculated threshold with a preset threshold and combining this with external interactive commands. Specifically, when the calculated threshold is reached... When the threshold is less than the preset first-level trigger threshold (set to 0.3 in this embodiment), and the system does not detect any active interaction commands such as voice or touch, the system is determined to be in a storage standby state. In this state, the foldable robotic arm body 100 maintains an S-shaped folding configuration, all levels of the arm tubes are fully retracted and nested along the axial direction, and the worm gear self-locking mechanism at the joint is in the locked position to maintain posture stability and ensure that the overall storage volume does not exceed 5L. When When the threshold value is greater than or equal to the preset first-level trigger threshold, or when an active interaction command is received, the system is determined to enter the working mode. At this time, the central control unit 300 releases the joint self-locking and sends an enable signal to the joint drive module of the folding robotic arm body 100, preparing to execute subsequent gate control detection and motion planning.

[0051] In step S30 of the intelligent cockpit robotic arm control method, the system executes EEG signal gating logic based on visual gaze, and solves the instability problem of single EEG control through spatiotemporal synchronization and probability fusion of multimodal data. This step specifically includes the following sub-steps:

[0052] The system constructs a 3D semantic map of the cockpit environment and calculates the driver's gaze vector in real time. Specifically, the central control unit 300 uses the 3D-ToF camera in the perception and interaction module 200 to acquire depth point cloud data within the cockpit and segments independent object point cloud clusters using an Euclidean clustering algorithm. In this embodiment, the system sets the clustering tolerance threshold to 2cm to 5cm and generates a corresponding 3D bounding box for each interactive object (such as a water cup, tissue box, or mobile phone), forming a set of regions of interest. ,in This represents the total number of objects identified. Simultaneously, based on acquired facial feature points and combined with 3D-ToF depth information, the system uses the Perspective-n-Point (PnP) algorithm to calculate the rotation matrix and translation vector of the driver's head relative to the camera coordinate system, constructing a 6-DOF model of head pose. Building upon this, the system utilizes the vector relationship between the aperture center and the corneal reflection point, combined with a pre-calibrated gaze mapping matrix, to calculate the gaze direction vector. Subsequently, the system will assign the line-of-sight vector... The coordinates are transformed to the vehicle world coordinate system using a coordinate transformation matrix, and a ray casting algorithm is executed to calculate the set of line-of-sight rays and regions of interest. Spatial intersection of the bounding boxes of each object If an intersection exists, then it is determined that the current line of sight is on the corresponding object. superior.

[0053] The central control unit 300 performs time-integral-based visual gating decisions to filter out unintentional saccades. Due to the physiological characteristics of the human eye, unconscious saccades are typically very short-lived, while conscious gazes are sustained over time. Therefore, the system defines a gating activation function. The integral is used to determine the time it takes for the line of sight to dwell on a specific object. Only when the line of sight is on a certain object... When the continuous dwell time on the brain exceeds a preset threshold, the gating is activated, allowing EEG signals to access the control circuit. (Gating activation function) The mathematical expression for is defined as follows:

[0054] ;

[0055] in, For a moment The gating state, Indicates activation. Indicates that the window is closed; In this embodiment, the duration of the sliding observation window is... Set to 1.5 seconds, slightly longer than the trigger threshold, to ensure sufficient historical states are stored; As an indicator function, when the lines of sight intersect... Located in the object The value is 1 when the value is within the bounding box of the function, and 0 otherwise. In this embodiment, the preset dwell time threshold is used. The time limit is set to 0.8 to 1.2 seconds, which can effectively distinguish between unconscious squinting and conscious staring.

[0056] In gated state Upon triggering, the system activates the EEG signal analysis algorithm and calculates the visual prior probability. The central control unit 300 extracts a pre-defined sequence of EEG signals from before the current moment (e.g., 2 seconds). The system first... Frequency domain feature extraction is performed, focusing on extracting the power spectral density (PSD) or differential entropy (DE) features of the motion sensory rhythm (Mu rhythm, 8-12Hz) and Beta rhythm (13-30Hz) to construct a feature vector sequence. Subsequently, the feature vector sequence is input into a pre-trained LSTM (Long Short-Term Memory) classification model, which outputs a preliminary classification probability distribution of EEG intent. ,in Representing the Such operation intent (e.g., grab, push away), This represents the total number of intent categories supported by the system. Simultaneously, the system considers the visually recognized gazed object. The attributes are used to generate a visual prior probability distribution. The prior probability is generated based on a preset object function availability association table. For example, in this embodiment, when the object being gazed at is a water cup, the prior probability corresponding to the grasping intention is set to a high value (e.g., 0.8), while the probability corresponding to the pushing intention is set to a low value.

[0057] The system performs Bayesian intent fusion to calculate the final probability of the intended action. To improve the robustness of intent recognition, the system uses visual prior probabilities to perform posterior correction on the EEG classification results, and then calculates the final fused intent probability. The Bayesian fusion formula is as follows:

[0058] ;

[0059] in, For the first The final posterior probability of the intention; The EEG classification probability output by the LSTM model; This represents the prior probability of an object based on visual gaze. This represents the total number of intent categories supported by the system.

[0060] The central control unit 300 will calculate the maximum posterior probability. The system compares the result with a preset confirmation threshold (set to 0.85 in this embodiment). If the maximum posterior probability exceeds this threshold, the system confirms the intent as a genuine driver command and locks onto the target object. and corresponding actions The system generates corresponding robotic arm motion planning tasks. If the threshold is not exceeded, the system keeps the gating open, updates the sliding window data in the next time step, and repeats the above probability calculation until the threshold condition is met or the gaze moves out of the region of interest. Through the above visual gaze gating and Bayesian fusion mechanism, the system can use visual context to constrain the EEG decoding space and effectively eliminate random artifacts in pure EEG control.

[0061] In step S40 of the intelligent cockpit robotic arm control method, the system performs environmental modeling and physiological state-based variable impedance control. This step, by constructing a virtual force field and dynamically adjusting the controller gain, enables the robotic arm to possess compliant characteristics at the physical interaction level that match the driver's current physiological state. This step specifically includes the following sub-steps:

[0062] The central control unit 300 constructs a dynamic obstacle avoidance environment model based on an Artificial Potential Field (APF). Specifically, the system utilizes a generated set of regions of interest. The bounding box of objects in the system is used as an obstacle source, and the target position of the robotic arm's end effector (such as the operator's hand position) is set as the gravitational source. In Cartesian space, the system defines the total potential field function. The function is determined by the gravitational potential field pointing towards the target. and the repulsive potential field away from the obstacle It is formed by superposition, that is, it satisfies

[0063] ;

[0064] To achieve global convergence towards the target, the gravitational potential field in this embodiment... Using the definition of a quadratic function:

[0065] ;

[0066] in, The preset gravity gain coefficient is set to 10 to 20 in this embodiment to establish the dominant trend of the robotic arm moving toward the target; These are the coordinates of the target point.

[0067] Repulsive potential field The calculation formula is as follows:

[0068] ;

[0069] in, This is a preset repulsive force gain coefficient used to adjust the intensity of the obstacle avoidance response; The coordinates of the current position of the robotic arm's end effector; Distance The point on the nearest obstacle (i.e., the surface of the object's bounding box), which is determined by calculating the intersection of the line connecting the current position and the geometric center of the bounding box with the surface of the bounding box; This represents the Euclidean distance between the current position and the nearest obstacle. The threshold for the influence radius of the obstacle is set to 0.15 meters to 0.2 meters in this embodiment. When the robotic arm enters this distance range, the repulsive field takes effect, generating a virtual force away from the obstacle. The system adjusts the total potential field function... By finding the negative gradient, we can obtain the planned collision-free motion velocity vector.

[0070] The central control unit 300 establishes a nonlinear mapping relationship between physiological state and impedance parameters. To adapt to different physiological states of the driver, the system calculates the comprehensive fatigue index based on step S20. Adjusting the Cartesian stiffness matrix at the end of the robotic arm in real time With damping matrix In this embodiment, and All are 6×6 diagonal matrices, corresponding to three translational degrees of freedom and three rotational degrees of freedom in Cartesian space, respectively. The system employs a high-fatigue, high-damping, and low-stiffness adjustment strategy. Specifically, when the driver's fatigue index is high, the system reduces the stiffness of the robotic arm to decrease the contact force, while simultaneously increasing damping to suppress motion overshoot and oscillation. The dynamic adjustment model for the impedance parameters is as follows:

[0071] ;

[0072] ;

[0073] in, The preset reference stiffness matrix represents the standard stiffness of the robotic arm when the driver is awake. The preset reference damping matrix; This is the stiffness attenuation coefficient, with a value range of [0.3, 0.5]. The damping enhancement factor has a value range of [0.2, 0.4]. The normalized comprehensive fatigue index.

[0074] The system executes position-based Cartesian space variable impedance control. The central control unit 300 combines the planned reference trajectory with the calculated dynamic impedance parameters to generate joint driving torque. This control law aims to simulate the dynamic characteristics of the robotic arm's end effector as a second-order mass, spring, and damping system. Specifically, the system calculates the desired force of the end effector in Cartesian space. This force is used to eliminate the deviation between the actual position and the desired position. The control law expression is as follows:

[0075] ;

[0076] in, , , These are the desired position, velocity, and acceleration planned using the artificial potential field method, respectively. , , These are the actual position, velocity, and acceleration fed back by the encoder and inertial measurement unit, respectively. This is a preset virtual mass matrix used to set the inertial characteristics of the robotic arm during the interaction process. It is usually set as a constant diagonal matrix.

[0077] The central control unit 300 maps the desired force in Cartesian space to the driving torque of each joint and superimposes dynamic compensation terms. The system utilizes the Jacobian matrix of the robotic arm. transpose Virtual forces in Cartesian space This is converted into torque in joint space. The final total torque command is output to the motor driver. The calculation is as follows:

[0078] ;

[0079] in, The joint angle vector; This is the gravity compensation torque term, used to counteract the effects of the robot arm's own gravity. This is a compensation term for Coriolis force and centrifugal force, used to eliminate the nonlinear coupling effects during high-speed movement of the robotic arm. For the gravity compensation term... With Coriolis force Those skilled in the art can perform real-time calculations using Lagrange dynamics modeling or the Newton-Euler iterative method, combined with the mass, center of mass position, and moment of inertia parameters of each link of the robotic arm. Through this control loop, the robotic arm can exhibit variable compliance based on the operator's fatigue state during movement, improving the safety of human-machine interaction while ensuring operational accuracy.

[0080] In step S50 of the intelligent cockpit robotic arm control method, the system executes a human-like motion strategy, multimodal interactive feedback, and emergency safety logic. This step ensures the safety and comfort of human-machine physical interaction by planning a high-order smooth trajectory, providing interactive confirmation through tactile channels, and monitoring disturbances based on a dynamic model. This step specifically includes the following sub-steps:

[0081] The central control unit 300 generates a human-like motion trajectory based on the Minimum Jerk criterion. To avoid sudden acceleration changes in the robotic arm at the start and end of its motion, this embodiment uses the minimum jerk criterion to interpolate path points. This criterion effectively suppresses residual vibrations by minimizing the integral of the jerk over the entire motion process. The system uses a fifth-order polynomial interpolation algorithm to plan the trajectory in Cartesian space. The total motion time of the robotic arm is denoted as... The normalized time variable is denoted as ,in For the current time, Location trajectory Calculate using the following formula:

[0082] ;

[0083] in, This is the starting position of the movement; The target position; polynomial coefficients 10, -15,6 are based on boundary conditions (i.e., the initial time). and the end time The fixed constants obtained by solving the equation (where velocity and acceleration are both zero) ensure the continuity of position, velocity, and acceleration in the time domain, resulting in a smooth bell-shaped velocity curve for the robotic arm, consistent with the biological characteristics of natural human upper limb movements.

[0084] When the robotic arm interacts with the driver or an object, the central control unit 300 executes vibration-tactile interactive feedback logic. In this embodiment, when the system detects task completion (such as grasp confirmation) or receives a specific EEG confirmation command, the impedance control force... On top of this, a high-frequency, low-amplitude tactile feedback force is superimposed. This feedback force does not drive the robotic arm to produce macroscopic displacement; instead, it transmits high-frequency vibration signals through the end effector, informing the operator of the operational status via a tactile channel. Tactile feedback force Defined using a decaying sine wave model:

[0085] ;

[0086] in, The relative time calculated from the self-feedback trigger moment, i.e. The vibration amplitude is set to 2N to 5N, which is within the comfortable range perceived by the human body. The attenuation coefficient is used to control the duration of the feedback signal. In this embodiment, it is set to 5.0, so that the vibration signal attenuates to imperceptible levels within about 0.6 seconds. The vibration frequency is set to 20Hz to 40Hz, which corresponds to the sensitive response frequency of tactile corpora in human skin. The system will... Superimposed on the expected force in Cartesian space, the driving motor generates torque fluctuations, enabling information transmission through non-visual channels.

[0087] The system operates sensorless collision detection and emergency safety logic based on a Generalized Momentum Observer. This logic utilizes the robotic arm's own analytical dynamics model to decouple external contact torque from complex nonlinear dynamic terms, thereby achieving sensitive collision detection without the need for external torque sensors. The central control unit 300 defines the generalized momentum. And construct the momentum observation residual vector Momentum observation residual vector The iterative calculation formula is as follows:

[0088] ;

[0089] in, The gain matrix is ​​the observer, which is usually a positive definite diagonal matrix. The values ​​of its diagonal elements determine the dynamic response speed and filtering characteristics of the observer. Here is the inertia matrix of the robotic arm; This represents the actual output torque of the motor. The effects of the Coriolis force and centrifugal force terms on momentum; This is the gravity compensation torque term; Let be the initial momentum; For integration variables. In this model, the residuals... Physically converges to the external torque. The first-order low-pass filter value.

[0090] The system monitors the residual vector in real time. The modulus length. When Exceeding the preset security threshold (In this embodiment, the torque is set to 10Nm to 15Nm) when an unexpected collision is determined to have occurred. At this time, the emergency safety logic takes precedence over impedance control, and the system immediately executes the following safety strategy: First, the motor control mode is switched to zero gravity mode, that is, only gravity compensation torque is output. This allows the robotic arm to be in a passive compliant state, enabling external forces to change its posture with minimal resistance. Secondly, if the collision duration exceeds a preset value (e.g., 0.5 seconds), the system performs an emergency stop braking operation, locking the brakes on each joint to prevent secondary damage. For inertial matrix... The identification and calibration of relevant dynamic parameters can be obtained by those skilled in the art using the least squares method or excitation trajectory optimization algorithm, and will not be elaborated here.

[0091] See attached document Figure 3 - Appendix Figure 5 Specific application examples:

[0092] To more clearly illustrate the technical solutions of the embodiments of the present invention, a detailed description is provided below in conjunction with specific application scenarios and control processes.

[0093] Scene setting:

[0094] This example is set in a highway driving scenario at 2:00 PM. The driver (Mr. Zhang) is driving the vehicle in adaptive cruise control mode. A bottle of mineral water is placed in the cup holder on the passenger side of the vehicle's cabin.

[0095] Fatigue monitoring and mode activation:

[0096] During system operation, the driver status monitoring subsystem 400 continuously collects data. The infrared camera 402 captures the driver's facial features at a frame rate of 60fps. Within a time window... Inside, the central control unit 300 calculated the driver's PERCLOS value (percentage of time spent with eyes closed) to be 0.18, while the blinking frequency decreased to 10 times per minute. The central control unit 300 then calculated the overall fatigue index based on a preset normalized model. :

[0097]

[0098] Visual fixation and brainwave gating:

[0099] The driver needs a drink and turns to look at the bottled water in the passenger-side cup holder. The visual acquisition unit 201 (3D-ToF) in the perception and interaction module 200 constructs an environmental map in real time and generates a bounding box of the water bottle. The system calculates the driver's line-of-sight vector in real time and detects when the driver's gaze falls into... The central control unit 300 performs time integration and detects the line of sight within the area. The continuous dwell time within the eye reaches 1.0 second, exceeding the preset threshold (0.8 seconds). The system then generates a high-level gating activation signal, allowing the EEG signal to enter the control loop and masking the EEG data from the previous non-fixation period.

[0100] Intent fusion and confirmation:

[0101] After the gating signal is activated, the system performs intent confirmation:

[0102] Visual Prior: The gaze target is identified as a mineral water bottle; the object attribute database is queried to generate prior probabilities. (Grab | Water Bottle) = 0.85, (Push open | water bottle) = 0.1.

[0103] EEG decoding: The EEG acquisition device 401 extracts Mu rhythm features, which are then classified and output by an LSTM model to determine the probability of right-hand grasping intention. Scraping .

[0104] Bayesian fusion: The above probabilities are fused to calculate the posterior probability, resulting in... The value of 0.92 (grab) exceeds the confirmation threshold of 0.85. Since 0.92 exceeds the threshold, the system identifies the target as a mineral water bottle and determines to perform the grab and delivery action.

[0105] Variable impedance control and path planning:

[0106] Based on the calculated comprehensive fatigue index The central control unit 300 adjusts the impedance parameters of the robotic arm's end effector in real time.

[0107] Stiffness adjustment: Adjusting the Cartesian stiffness matrix parameters The value is reduced to 70% of the baseline, making the robotic arm compliant to accommodate the driver's slow response.

[0108] Damping adjustment: Appropriately increase the damping matrix parameters To suppress motion overshoot, the path planning module uses an artificial potential field method to identify the gear shift lever as an obstacle and establish a repulsive potential field, while setting the water bottle as an attractive source, and plans a collision-free motion velocity vector.

[0109] Execution, Feedback, and Security:

[0110] The robotic arm moves along a minimum acceleration trajectory based on a fifth-order polynomial programming algorithm, smoothly grasping the water bottle and delivering it to the driver. Upon reaching the delivery position, the end effector applies a 30Hz, 3N amplitude, damped sinusoidal tactile feedback force. During delivery, if the driver's arm accidentally touches the robotic arm, collision detection logic based on a generalized momentum observer monitors the residual vector in real time. When the residual modulus exceeds the safety threshold of 10Nm, the drive module immediately switches to zero-gravity mode and outputs a gravity compensation torque, causing the robotic arm to retreat in the direction of the collision force, avoiding physical injury to the driver.

[0111] Experimental verification and effect comparison:

[0112] To verify the technical effect of the present invention, a simulation test platform containing a 6-DOF robotic arm dynamics model was constructed, and the performance differences between the existing technical solution (Solution A) and the present invention solution (Solution B) were compared.

[0113] Option A: Control is based solely on EEG signals, without visual gating, and employs fixed high-rigidity position control and trapezoidal velocity planning.

[0114] Option B: Employs the visual gated Bayesian fusion, variable impedance control, and minimum jerk planning of this invention.

[0115] Intent recognition accuracy was compared in simulated driving tests, which statistically analyzed the system's ability to recognize grasping intents. For example... Figure 3 As shown, Scheme A, lacking visual constraints, is easily affected by electromyographic artifacts generated by the driver's head movements, resulting in a high false trigger rate. Scheme B introduces visual gaze as gating and prior probability, increasing the average recognition accuracy from 72.4% to 94.1%, and reducing the false trigger rate from 8.5 times / hour to 0.8 times / hour.

[0116] The collision safety verification simulated an accidental collision between a robotic arm and a driver during task execution (fatigue index set at 0.57). In Scheme A, due to its fixed stiffness and inability to sense external torque, the peak contact force reached 65N. Scheme B reduced stiffness through variable impedance control and utilized a momentum observer to quickly switch to zero-gravity mode upon collision detection (contact duration shortened to 45ms), controlling the peak contact force below 12N and improving physical safety.

[0117] Comparison of motion trajectory smoothness:

[0118] Compare the end-effector motion characteristics of the two schemes. For example... Figures 4-5 As shown, Scheme A uses trapezoidal velocity planning, which results in a sudden acceleration change at the acceleration-deceleration switching point, causing the jerk to reach theoretical infinity (pulse) and causing the robotic arm to vibrate. Scheme B uses fifth-order polynomial interpolation, which ensures the continuity of the jerk in the time domain, reduces the residual vibration amplitude at the end of the arm by 85%, and ensures the stability of liquid delivery.

[0119] For scheme B (this invention, fifth-order polynomial interpolation): simulation data shows that its acceleration curve presents a continuous, smooth, parabolic shape without any discontinuities or abrupt changes. Specifically, the curve starts at a positive value (38 m / s²). 3 ), then descended smoothly, in It crosses the zero axis; then continues to descend, reaching the midpoint of the motion. Reaching a negative peak (approximately) 30m / s 3 After that, the curve began to rise again. It crosses the zero axis again and eventually rises to a positive value. The corresponding normalized displacement curve exhibits a perfect S-shaped characteristic: the tangent slope is gentle at the beginning and end (both velocity and acceleration are zero), and the transition in the middle is smooth.

[0120] Experimental conclusion: The continuously varying jerk characteristic of Scheme B across the entire time domain effectively avoids the system's flexible impact. Actual measurement data shows that, compared to Scheme A, Scheme B reduces the residual vibration amplitude at the robotic arm's end effector by 85%, thus ensuring extremely high stability when delivering liquid items such as mineral water.

Claims

1. An intelligent cockpit robotic arm system, characterized in that, include: The foldable robotic arm body (100) is installed in the vehicle cabin and has a drive module at the joint. The drive module is used to drive the foldable robotic arm body to perform multi-degree-of-freedom movements and folding / unfolding switching in the cabin space. The perception and interaction module (200) is located at the end of the foldable robotic arm body (100) and includes a vision acquisition unit (201) for collecting environmental data and a display and interaction unit (202). The driver status monitoring subsystem (400) includes an EEG acquisition device (401) located in the steering wheel area and an infrared camera (402) located in front of the cockpit. The central control unit (300), electrically connected to the folding robotic arm body (100), the perception and interaction module (200), and the driver state monitoring subsystem (400), is used to execute the following control logic: The comprehensive fatigue index is calculated based on the data from the driver status monitoring subsystem (400), and the folding robotic arm body (100) is controlled to enter the working mode or remain in the storage standby state accordingly. In the working mode, the gaze information acquired by the visual acquisition unit (201) is used as a gating signal. When the gaze information meets the preset conditions, the EEG signal analysis is activated. The operation intention is confirmed by combining the visual prior probability and the EEG signal classification result, and the foldable robotic arm body (100) is controlled to perform the action. The central control unit (300) is also used to perform variable impedance control, specifically including: establishing a mapping relationship between physiological state and impedance parameters, and adjusting the Cartesian stiffness matrix and damping matrix at the end of the folding robotic arm body (100) in real time according to the calculated comprehensive fatigue index; the adjustment includes: as the comprehensive fatigue index increases, reducing the parameter value of the stiffness matrix to reduce the contact force, and increasing the parameter value of the damping matrix to suppress motion overshoot; When controlling the movement of the foldable robotic arm body (100), the central control unit (300) uses the minimum acceleration criterion to interpolate the movement path points and generate a position trajectory based on a fifth-order polynomial; wherein the velocity and acceleration of the position trajectory are both zero at the start and end of the movement, and the acceleration is continuous in the time domain. 2.The intelligent cabin mechanical arm system according to claim 1, wherein, The central control unit (300) specifically includes the following when calculating the comprehensive fatigue index: Receive facial image data collected by the infrared camera (402), and calculate the PERCLOS value and blink frequency per unit time; A normalized model is constructed to perform linear weighted calculation on the PERCLOS value and the blink frequency to generate the comprehensive fatigue index; When the overall fatigue index exceeds the preset trigger threshold, or when an active interaction command is received, the system is determined to enter the working mode. 3.The intelligent cabin mechanical arm system of claim 1, wherein, The central control unit (300) uses the gaze information acquired by the vision acquisition unit (201) as a gating signal to perform the following steps: The data from the visual acquisition unit (201) is used to construct a three-dimensional environment map of the cockpit and generate bounding boxes of objects; The driver's line of sight direction vector is calculated in real time, and the spatial intersection point of the line of sight direction vector and the bounding box is calculated. The continuous dwell time of the line of sight within the bounding box of any of the objects is calculated, and a gating activation signal is generated only when the continuous dwell time exceeds a preset dwell time threshold, allowing the EEG signal to enter the control loop.

4. The intelligent cabin robotic arm system of claim 3, wherein, The central control unit (300) combines visual prior probability with EEG signal classification results to confirm the operation intention and executes the following steps: After receiving the gating activation signal, the frequency domain features of the EEG signal collected by the EEG acquisition device (401) are extracted, and the probability of EEG intention is output using the long short-term memory network classification model. Simultaneously, the functional attributes of the object that triggers the gating activation signal are identified, and visual prior probabilities are generated based on the functional attributes. The EEG intention probability and the visual prior probability are fused using Bayes' theorem to calculate the posterior intention probability. If the posterior intention probability exceeds a preset confirmation threshold, the target object and corresponding action are locked.

5. The intelligent cockpit robotic arm system according to claim 1, characterized in that, The central control unit (300) is also used to construct an obstacle avoidance environment model based on an artificial potential field algorithm, specifically including: The object bounding box identified by the perception interaction module (200) is used as an obstacle source to establish a repulsive potential field, and the target position is set as a gravitational source to establish a gravitational potential field, and the two fields are superimposed to form a total potential field. Calculate the negative gradient of the total potential field, and plan the collision-free motion velocity vector of the folding robotic arm body (100) accordingly; Using the collision-free motion velocity vector as the desired motion input, and combining it with the adjusted stiffness matrix and damping matrix, the driving torque command for each joint is generated through an impedance control law.

6. The intelligent cockpit robotic arm system according to claim 1, characterized in that, The central control unit (300) is also used to perform haptic interaction feedback, specifically including: When the task is detected to be completed or a confirmation command is received, a high-frequency, low-amplitude tactile feedback force signal is generated. The tactile feedback force signal is superimposed on the driving torque of the foldable robotic arm body (100) to generate vibration at the end; the tactile feedback force signal adopts a decaying sine wave model.

7. The intelligent cockpit robotic arm system according to claim 1, characterized in that, The central control unit (300) is also used to run collision detection logic based on a generalized momentum observer, specifically including: Momentum observation residual vector is constructed based on the dynamic model of the foldable robotic arm body (100), and the residual vector represents the external torque. The magnitude of the residual vector is monitored in real time, and a collision is determined to have occurred when it exceeds a preset safety threshold. Upon determining that a collision has occurred, the drive module is immediately switched to zero-gravity mode, outputting only gravity compensation torque, so that the foldable robotic arm body (100) is in a passive compliant state.

8. The intelligent cockpit robotic arm system according to claim 1, characterized in that, The visual acquisition unit (201) uses a 3D-ToF camera; the display interaction unit (202) uses a flexible OLED display screen; the EEG acquisition device (401) includes a non-invasive dry electrode array integrated into the three o'clock and nine o'clock grip areas of the steering wheel; the central control unit (300) reads the real-time driving status data of the vehicle through the vehicle CAN bus interface, and uses the real-time driving status data of the vehicle to perform time synchronization and alignment of the acquired physiological signals.

Images