A multi-modal fusion charging gun line intelligent assistance control method and system

Abstract

This invention provides a multimodal fusion-based intelligent assist control method and system for charging gun cables, comprising: acquiring multimodal sensor data; performing attitude calculation on the charging gun based on the multimodal sensor data to obtain the attitude data of the charging gun, and calculating the effective operating force of the user on the charging gun to obtain a comprehensive perception state vector of the charging gun; constructing a user intention feature vector based on the comprehensive perception state vector, and using an intention recognition algorithm to calculate the probability of the user performing a corresponding operation state intention, and using a state machine to calculate the state transition probability of each detected user intention; predicting the user's hand position parameters based on the user intention, and actually acquiring the charging gun's hanging ring position parameters, and constructing a tracking error function based on the predicted user hand position parameters and the charging gun's hanging ring position parameters; generating a control force for a corresponding following mechanism based on the tracking error function, and converting the control force into a driving command for the following mechanism.

Description

Technical Field

[0001] This invention relates to the field of charging gun technology, and in particular to a multimodal fusion intelligent assist control method and system for charging gun lines. Background Technology

[0002] Currently, traditional electric vehicle charging station cables are generally heavy, making manual plugging and unplugging difficult for users. Therefore, to assist with charging cable operation, the following existing technologies are mainly used: 1. Gravity balancing scheme: Using a spring or pulley counterweight mechanism to partially offset the vertical gravity of the cable, but it cannot provide active assistance in the horizontal direction, and the user still needs to bear the horizontal dragging force; 2. Electric cable winder scheme: Using an electric cable winder to assist in winding the charging cable, but it can only achieve automatic winding and cannot provide unidirectional assistance during user dragging, and the winding action may conflict with the user's intention. Therefore, the above existing technical solutions have technical problems such as lack of intention recognition capability, insufficient active following capability, and a single direction of assistance. Summary of the Invention

[0003] One objective of this invention is to provide a multimodal fusion intelligent assist control method and system for charging gun cables. This method and system utilizes multimodal sensors, including image detection sensors, IMUs, tension sensors, and angle encoders, for fusion analysis. It employs attitude calculation algorithms and state machines to determine the user's intention in holding the charging gun, accurately distinguishing between different user intentions such as moving, aligning, and plugging / unplugging the charging gun. Based on these different intentions, it executes flexible assist control of the corresponding charging gun cable, enabling precise perception and accurate assistance of user behavior.

[0004] One objective of this invention is to provide a multimodal fusion intelligent assist control method and system for charging gun cables. The method and system utilize a vision sensor to detect visual errors in the charging gun and construct a three-axis force-driven PID control strategy based on these visual errors. It provides corresponding charging gun assist prediction data based on the user's operational intent regarding the charging gun. By constructing the visual error of the charging gun based on the actual detected position and the predicted position, it achieves charging gun following actions based on the user's operational intent, effectively improving the assist effect of the charging gun in time-varying states and making the human-machine operation experience smoother.

[0005] One of the objectives of this invention is to provide a multimodal fusion intelligent assist control method and system for charging gun cables. The method and system provide a visual behavior detection model based on user image information. The visual behavior detection model can effectively identify abnormal user behavior states, and then perform emergency shutdown and early warning for the charging gun based on the abnormal behavior states, thereby effectively improving the safety of the charging gun.

[0006] To achieve at least one of the above-mentioned objectives, the present invention further provides a multi-modal fusion intelligent assist control method for charging gun cables, the method comprising:

[0007] Acquire multimodal sensor data, perform attitude calculation on the charging gun based on the multimodal sensor data to obtain the attitude data of the charging gun, and calculate the effective operating force of the user on the charging gun to obtain the comprehensive perception state vector of the charging gun.

[0008] The user intent feature vector is constructed based on the comprehensive perception state vector, and the probability of the user performing the corresponding operation state intent is calculated using the intent recognition algorithm. The state transition probability of the user intent detected each time is calculated using a state machine.

[0009] Based on the user's intent, predict the user's hand position parameters and obtain the charging gun ring position parameters. Construct a tracking error function based on the predicted user hand position parameters and the charging gun ring position parameters.

[0010] The control force of the corresponding following mechanism is generated based on the tracking error function, and the control force is converted into the driving command of the following mechanism.

[0011] According to a preferred embodiment of the present invention, the attitude calculation method of the charging gun includes: acquiring acceleration and angular velocity data of the charging gun, and calculating the quaternion attitude of the charging gun by fusing the acceleration and angular velocity data using an extended Kalman filter. Where k represents the detection time, the charging gun angular velocity is obtained under the same detection time k. Build the charging gun state Predict the quaternion pose at the current detection time k based on the charging gun state at the previous detection time k-1. .

[0012] According to another preferred embodiment of the present invention, corresponding to quaternion poses The prediction equation is: The observation equation corrected using the direction of gravitational acceleration is: ;in This represents the quaternion predicted for the current detection time after updating the quaternion obtained from the previous detection time k-1. This indicates the angular velocity of the charging gun during the previous detection time. Indicates the time interval between adjacent detection times. Indicates the observed acceleration, This represents the rotation matrix expressed in quaternions, where g represents the acceleration due to gravity. This represents the linear acceleration generated by the movement of the charging gun itself. The accelerometer measurement noise is represented by the extended Kalman filter, and the optimal charging gun attitude quaternion is obtained recursively. And calculate the Euler angles. .

[0013] According to another preferred embodiment of the present invention, the formula for calculating the effective operating force of the user on the charging gun includes: ;in This represents the effective operating force vector applied by the user to the gun wire. A unit vector representing the direction of the cable. Let g represent the mass of the gun wire, and g represent the acceleration due to gravity. This represents the estimated frictional force between the cable and the lifting ring, the ground, etc.; further, the following comprehensive sensing state vector S is obtained: ;in Indicates the user's joint position based on visual recognition. This indicates hand speed based on visual recognition.

[0014] According to another preferred embodiment of the present invention, the formula for constructing the intent feature vector is:

[0015] ;

[0016] in This represents the effective operating force vector applied by the user to the gun wire. Indicates the angular velocity of the charging gun. This indicates hand speed based on visual recognition. Indicates the user's joint position based on visual recognition. This indicates the estimated location of the charging port for the trolley.

[0017] According to another preferred embodiment of the present invention, a Bayesian classification algorithm is used as the intent recognition algorithm to calculate the probability of the user's operation state on the charging gun: the current user's state is defined as... The Bayesian posterior state probability is:

[0018] ;

[0019] in , and These represent the operation states of moving, aligning, and inserting / removing, respectively. Let be the class conditional probability density, expressed as in state . The likelihood of feature f is observed and obtained by pre-modeling it as a Gaussian mixture model. For state The prior probabilities are given, where i and j represent the corresponding operation state indices, and the operation state probabilities of the user are determined by constructing the Bayesian posterior state probability threshold and corresponding conditions.

[0020] According to another preferred embodiment of the present invention, the method for constructing the control force and drive command of the following mechanism includes: estimating the horizontal position of the user's hand according to the following formula:

[0021] ;

[0022] in This indicates the horizontal position of the lifting ring, and L indicates the length of the protruding cable. This indicates the horizontal angle of the cable from the lifting ring to the nozzle. Indicates the angle of elevation or depression of the cable relative to the horizontal plane;

[0023] The formula for calculating the tracking error function is as follows:

[0024] ;

[0025] The control force is calculated based on the correction error using the following formula. :

[0026] ;

[0027] in Represents the proportional gain matrix. Represents the integral gain matrix. Represents the integral term of the error. Represents the feedforward gain matrix. This represents the horizontal speed of the hand. Further calculation yields the following motor speed command:

[0028] ;

[0029] in This is the equivalent damping coefficient.

[0030] To achieve at least one of the above-mentioned objectives, the present invention further provides a multimodal fusion intelligent assist control system for charging cables, wherein the system executes the above-mentioned multimodal fusion intelligent assist control method for charging cables.

[0031] The present invention further provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described multimodal fusion intelligent assist control method for charging gun cables. Attached Figure Description

[0032] Figure 1The diagram shown is a flowchart of a multimodal fusion intelligent assist control method for charging gun cables according to the present invention. Detailed Implementation

[0033] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0034] It is understood that the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0035] Please combine Figure 1 This invention discloses a multimodal fusion intelligent assist control method and system for charging gun cables, wherein the method mainly includes the following steps:

[0036] S01. Acquire multimodal sensor data, perform attitude calculation on the charging gun based on the multimodal sensor data, obtain the attitude data of the charging gun, calculate the effective operating force of the user on the charging gun, and obtain the comprehensive perception state vector of the charging gun.

[0037] S02. Construct a user intent feature vector based on the comprehensive perception state vector, and use an intent recognition algorithm to calculate the probability of the user performing the corresponding operation state intent, and use a state machine to calculate the state transition probability of the user intent detected each time.

[0038] S03. Based on the user's intention, predict the user's hand position parameters and obtain the charging gun ring position parameters. Construct a tracking error function based on the predicted user's hand position parameters and the charging gun ring position parameters.

[0039] S04. Generate the control force of the corresponding following mechanism according to the tracking error function, and convert the control force into the driving command of the following mechanism.

[0040] It is worth mentioning that the system described above in this invention includes the following units: a sensing unit, a decision-making unit, and an execution unit. The sensing unit comprises sensors such as an IMU (Inertial Measurement Unit), a tension sensor, an angle encoder, and a site camera, used to acquire sensing data from the charging gun, cable, lifting ring, and user operations. The execution unit includes an attitude calculation module, a state machine, a visual behavior analysis module, and a safety protection logic module, wherein the state machine is used to determine and predict the user's operational intentions. The execution layer includes a two-dimensional motion platform (X / Y axis) driven by a stepper motor, which drives the charging gun cable via the lifting ring.

[0041] This invention further defines the sensing data of the multimodal sensor as follows: IMU output: triaxial acceleration. Three-axis angular velocity: Tension sensor output: gun wire tension The direction of the cable tension is determined by the geometric constraints of the lifting eye and an angle encoder. The angle encoder measures the horizontal angle of the cable at the lifting eye. and pitch angle Visual sensor data: User joint position The visual sensor data, such as user type, user's posture while holding the charging gun, and hand speed, can be obtained based on existing image recognition models, which will not be elaborated upon in detail in this invention.

[0042] It is worth mentioning that the present invention provides a method for calculating the attitude of a charging gun. The method includes: collecting acceleration and angular velocity data of the charging gun, wherein the acceleration and angular velocity can be obtained from a corresponding visual sensor or inertial measurement unit; and using extended Kalman filtering to fuse the acceleration and angular velocity data to calculate the quaternion attitude of the charging gun. Where k represents the detection time, the charging gun angular velocity is obtained under the same detection time k. Build the charging gun state Predict the quaternion pose at the current detection time k based on the charging gun state at the previous detection time k-1. .

[0043] Furthermore, corresponding to quaternion poses The prediction equation is: The observation equation corrected using the direction of gravitational acceleration is: ;in This represents the quaternion predicted for the current detection time after updating the quaternion obtained from the previous detection time k-1. This indicates the angular velocity of the charging gun during the previous detection time. Indicates the time interval between adjacent detection times. Indicates the observed acceleration, This represents the rotation matrix expressed in quaternions, where g represents the acceleration due to gravity. This represents the linear acceleration generated by the movement of the charging gun itself. The accelerometer measurement noise is represented by the extended Kalman filter, and the optimal charging gun attitude quaternion is obtained recursively. And calculate the Euler angles. .

[0044] It should be noted that the formula for calculating the effective operating force of the user on the charging gun in this invention includes: ;in This represents the effective operating force vector applied by the user to the gun wire. A unit vector representing the direction of the cable. Let g represent the mass of the gun wire, and g represent the acceleration due to gravity. This represents the estimated frictional force between the cable and the lifting ring, the ground, etc.; further, the following comprehensive sensing state vector S is obtained: ;in Indicates the user's joint position based on visual recognition. This indicates hand speed based on visual recognition.

[0045] Furthermore, the present invention constructs an intent feature vector based on the comprehensive perception state vector S, and the formula for constructing the intent feature vector is as follows:

[0046] ;

[0047] in This represents the effective operating force vector applied by the user to the gun wire. Indicates the angular velocity of the charging gun. This indicates hand speed based on visual recognition. Indicates the user's joint position based on visual recognition. This indicates the estimated location of the charging port for the trolley.

[0048] According to another preferred embodiment of the present invention, a Bayesian classification algorithm is used as the intent recognition algorithm to calculate the probability of the user's operation state on the charging gun: the current user's state is defined as... The Bayesian posterior state probability is:

[0049] ;

[0050] in , and These represent the operation states of moving, aligning, and inserting / removing, respectively. Let be the class conditional probability density, expressed as in state . The likelihood of feature f is observed and obtained by pre-modeling it as a Gaussian mixture model. For state The prior probabilities are given, where i and j represent the corresponding operation state indices, and the operation state probabilities of the user are determined by constructing the Bayesian posterior state probability threshold and corresponding conditions.

[0051] In one preferred embodiment of the present invention, the state transition logic of the Bayesian posterior state probability can be referenced as follows:

[0052] 1. If If so, the current state transitions to the insertion (Move) state;

[0053] 2. If If the nozzle is close to the car charging port, the current state will switch to aiming. ;

[0054] 3. If Then the current state transitions to plugging / unplugging. .

[0055] In one preferred embodiment of the present invention, the method for constructing the control force and drive command of the following mechanism includes: estimating the horizontal position of the user's hand according to the following formula:

[0056] ;

[0057] in This indicates the horizontal position of the lifting ring, and L indicates the length of the protruding cable. This indicates the horizontal angle of the cable from the lifting ring to the nozzle. Indicates the angle of elevation or depression of the cable relative to the horizontal plane;

[0058] The formula for calculating the tracking error function is as follows:

[0059] ;

[0060] The control force is calculated based on the correction error using the following formula. :

[0061] ;

[0062] in Represents the proportional gain matrix. Represents the integral gain matrix. Represents the integral term of the error. Represents the feedforward gain matrix. This represents the horizontal speed of the hand. Further calculation yields the following motor speed command:

[0063] ;

[0064] in This is the equivalent damping coefficient.

[0065] The processes described in the flowcharts above, as disclosed in the embodiments of this invention, can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the methods of this application are not limited to the aforementioned functions. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0066] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0067] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.

Claims

1. A multimodal fusion intelligent assist control method for charging gun cables, characterized in that, The method includes: Acquire multimodal sensor data, perform attitude calculation on the charging gun based on the multimodal sensor data to obtain the attitude data of the charging gun, and calculate the effective operating force of the user on the charging gun to obtain the comprehensive perception state vector of the charging gun. The user intent feature vector is constructed based on the comprehensive perception state vector, and the probability of the user performing the corresponding operation state intent is calculated using the intent recognition algorithm. The state transition probability of the user intent detected each time is calculated using a state machine. Based on the user's intent, predict the user's hand position parameters and obtain the charging gun ring position parameters. Construct a tracking error function based on the predicted user hand position parameters and the charging gun ring position parameters. The control force of the corresponding following mechanism is generated based on the tracking error function, and the control force is converted into the driving command of the following mechanism.

2. The multimodal fusion intelligent assist control method for charging gun cables according to claim 1, characterized in that, The attitude calculation method for the charging gun includes: collecting acceleration and angular velocity data of the charging gun, and using extended Kalman filtering to fuse the acceleration and angular velocity data to calculate the quaternion attitude of the charging gun. Where k represents the detection time, the charging gun angular velocity is obtained under the same detection time k. Build the charging gun state Predict the quaternion pose at the current detection time k based on the charging gun state at the previous detection time k-1. .

3. The multimodal fusion intelligent assist control method for charging gun cables according to claim 2, characterized in that, Corresponding quaternion pose The prediction equation is: The observation equation corrected using the direction of gravitational acceleration is: ;in This represents the quaternion predicted for the current detection time after updating the quaternion obtained from the previous detection time k-1. This indicates the angular velocity of the charging gun during the previous detection time. Indicates the time interval between adjacent detection times. Indicates the observed acceleration, This represents the rotation matrix expressed in quaternions, where g represents the acceleration due to gravity. This represents the linear acceleration generated by the movement of the charging gun itself. The accelerometer measurement noise is represented by the extended Kalman filter, and the optimal charging gun attitude quaternion is obtained recursively. And calculate the Euler angles. .

4. The multimodal fusion intelligent assist control method for charging gun cables according to claim 1, characterized in that, The formula for calculating the effective operating force of the user on the charging gun includes: ;in This represents the effective operating force vector applied by the user to the gun wire. A unit vector representing the direction of the cable. Let g represent the mass of the gun wire, and g represent the acceleration due to gravity. This represents the estimated frictional force between the cable and the lifting ring, the ground, etc.; further, the following comprehensive sensing state vector S is obtained: ;in Indicates the user's joint position based on visual recognition. This indicates hand speed based on visual recognition.

5. The multimodal fusion intelligent assist control method for charging gun cables according to claim 1, characterized in that, The formula for constructing the intent feature vector is as follows: ； in This represents the effective operating force vector applied by the user to the gun wire. Indicates the angular velocity of the charging gun. This indicates hand speed based on visual recognition. Indicates the user's joint position based on visual recognition. This indicates the estimated location of the charging port for the trolley.

6. The multimodal fusion intelligent assist control method for charging gun cables according to claim 1, characterized in that, The Bayesian classification algorithm is used as the intent recognition algorithm to calculate the probability of the user's operation state on the charging gun: the current user state is defined as... The Bayesian posterior state probability is: ； in , and These represent the operation states of moving, aligning, and inserting / removing, respectively. Let be the class conditional probability density, expressed as in state . The likelihood of feature f is observed and obtained by pre-modeling it as a Gaussian mixture model. For state The prior probabilities are given, where i and j represent the corresponding operation state indices, and the operation state probabilities of the user are determined by constructing the Bayesian posterior state probability threshold and corresponding conditions.

7. The multimodal fusion intelligent assist control method for charging gun cables according to claim 1, characterized in that, The method for constructing the control force and drive command of the following mechanism includes estimating the horizontal position of the user's hand according to the following formula: ； in This indicates the horizontal position of the lifting ring, and L indicates the length of the protruding cable. This indicates the horizontal angle of the cable from the lifting ring to the nozzle. Indicates the angle of elevation or depression of the cable relative to the horizontal plane; The formula for calculating the tracking error function is as follows: ； The control force is calculated based on the tracking error using the following formula. : ； in Represents the proportional gain matrix. Represents the integral gain matrix. Represents the integral term of the error. Represents the feedforward gain matrix. This represents the horizontal speed of the hand. Further calculation yields the following motor speed command: ； in This is the equivalent damping coefficient.

8. A multimodal fusion intelligent assist control system for charging gun cables, characterized in that, The system executes the multimodal fusion intelligent assist control method for charging gun cables as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement a multimodal fusion intelligent assist control method for charging gun cables as described in any one of claims 1-8.

Images