Intelligent automatic wheelchair driving system and method based on large language model

Through large language model and reinforcement learning algorithm, combined with environmental perception and natural language processing, the problem of insufficient autonomy and interactivity of traditional wheelchairs is solved, intelligent obstacle avoidance and path optimization are achieved, and users' travel safety and convenience are improved.

CN118939228APending Publication Date: 2024-11-12ANHUI UNIV
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202410983991.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional wheelchairs have shortcomings in autonomy, intelligence and user-friendliness. The existing assisted wheelchairs are limited to specific scenarios and functions, and lack sufficient interactivity and control convenience.

Method used

Adopt an intelligent automatic wheelchair driving system based on large language models, collect data through environment perception modules, integrate algorithms to integrate sensor data, combine natural language processing and path planning algorithms to generate driving instructions, and dynamically adjust strategies through reinforcement learning algorithms to provide customized Navigation service.

Benefits of technology

It has realized intelligent obstacle avoidance and path optimization, improved user travel safety and convenience, simplified user control, provided a personalized driving experience, and greatly improved user satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939228A_ABST
    Figure CN118939228A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent automatic wheelchair driving system and method based on a large language model, relates to the technical field of automatic driving, and solves the technical problem that an existing auxiliary driving wheelchair is limited to specific scenes and functions and lacks enough interactivity and control convenience. The method comprises the following steps: acquiring environment basic data, and obtaining environment sensing data through a fusion algorithm; analyzing the user voice instruction through a large language model to obtain a navigation task; dividing the navigation task into a plurality of navigation sub-tasks; calling a path planning API (Application Program Interface), and combining a path planning algorithm to obtain an optimized path of the plurality of navigation subtasks; performing analysis based on the environment sensing data, the voice instruction and the optimized path, and generating a driving instruction according to an analysis result; a voice instruction fed back by a user in real time is analyzed, and a driving instruction and a path planning strategy are dynamically adjusted; the driving instruction and the path planning are dynamically adjusted according to the feedback of the user, and deep personalized driving experience is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic driving, and in particular relates to an intelligent automatic wheelchair driving system and method based on a large language model. Background Art

[0002] Traditional wheelchairs have obvious deficiencies in autonomy, intelligence, and user-friendliness. Although existing assisted-driving wheelchairs have been improved, they are still limited to specific scenarios and functions, lacking sufficient interactivity and ease of operation. With the rapid development of technologies such as artificial intelligence, autonomous driving, big data, and cloud computing, the development of an intelligent wheelchair system with natural language interaction, action intention understanding, autonomous path planning, and driving capabilities is not only technically possible and feasible, but also a key way to improve the quality of life of people with disabilities and the elderly. An advanced intelligent wheelchair system should not only have the basic function of helping users to move autonomously, but also be able to interact with users in natural language, understand users' emotions and needs, and provide emotional comfort and support. Such a system can help users achieve a more efficient, comfortable, and enjoyable lifestyle, thereby greatly improving their quality of life and happiness. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an intelligent automatic wheelchair driving system and method based on a large language model, which is used to solve the technical problems that traditional wheelchairs have obvious deficiencies in autonomy, intelligence and user-friendliness, and that although the existing assisted driving wheelchairs have been improved, they are still limited to specific scenarios and functions and lack sufficient interactivity and ease of operation. The present invention obtains a navigation task by collecting environmental perception data and integrating the environmental perception data using a fusion algorithm and parsing user voice commands through a language model; obtains path planning of multiple navigation subtasks according to the navigation task using API technology and optimizes the path planning to obtain an optimized path; performs analysis based on environmental perception data, language commands and optimized paths, and generates driving commands by calling an API and passes the commands to an execution unit of the wheelchair; the system dynamically adjusts the driving commands and path planning strategies according to real-time feedback during use to provide more customized navigation services to solve the above problems.

[0004] To achieve the above object, a first aspect of the present invention provides an intelligent automatic wheelchair driving method based on a large language model, comprising:

[0005] S1: Collect basic environmental data through the environmental perception module, and perform fusion processing and target recognition on the environmental perception data through the fusion algorithm to obtain environmental perception data; wherein, the environmental perception module is integrated with multiple sensors, including laser radar and monocular camera;

[0006] Fusion algorithm is a technology that processes and combines data from two or more information sources to obtain more accurate, complete or reliable results;

[0007] S2: Analyze user voice commands through a large language model to obtain navigation tasks; use natural language processing technology to divide the navigation task into several navigation subtasks; among which, the navigation subtasks include locating the current position, determining the target position, and planning the path; the large language model includes Chat-GPT4.0, PaLM, Tongyi Qianwen:

[0008] Natural language processing is a branch of artificial intelligence that focuses on studying how to enable computers to understand, parse, generate, and interact with natural languages ​​used by humans;

[0009] S3: calling the path planning API and combining the path planning algorithm to obtain the optimized paths of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm;

[0010] S4: Analyzes environmental perception data, voice commands, and optimized paths, and generates driving commands based on the analysis results; transmits the driving commands to the execution unit of the wheelchair;

[0011] S5: Analyze the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjust the driving instructions and path planning strategies; among them, the reinforcement learning algorithm includes the Q-learning algorithm.

[0012] Preferably, obtaining environmental perception data includes:

[0013] Converting the data collected by the monocular camera into a first BEV feature, and converting the point cloud data obtained by the laser radar into a second BEV feature; wherein the BEV is a bird's-eye view;

[0014] At the BEV feature level, the first BEV feature and the second BEV feature are fused through a fusion algorithm to obtain environmental fusion data; the environmental fusion data is input into a target detection model for target recognition to obtain environmental perception data; wherein, the target detection model is constructed based on a deep learning algorithm.

[0015] Sensor integration plays a key role in the realization of autonomous wheelchairs. By utilizing multiple sensors, such as cameras, radars, and lidars, and using fusion technology to integrate and analyze the data from these sensors, the limitations of each sensor can be compensated, providing a more comprehensive and reliable environmental perception, allowing the wheelchair to obtain rich semantic information about the environment. This semantic information covers important data such as the current location of the wheelchair, the characteristics of the surrounding environment, and possible obstacles. By integrating the data from these sensors, the autonomous wheelchair can achieve efficient environmental perception, thereby effectively avoiding collisions and ensuring that the wheelchair remains stable during driving.

[0016] Preferably, the implementation steps of the fusion algorithm include:

[0017] S11: converting camera data into BEV features and mapping image information to a plane to facilitate cross-sensor data fusion; the BEV is a bird's-eye view;

[0018] S12: The point cloud data acquired by the lidar is also converted into BEV features to provide accurate environmental information in a dense form;

[0019] S13: At the BEV feature level, camera data is fused with laser point cloud data to integrate the information of the two sensors to enhance the comprehensiveness and accuracy of environmental perception;

[0020] S14: The fused BEV features are input into the target detection model, and the deep learning algorithm is used to identify and locate objects in the environment to obtain accurate detection results.

[0021] The object detection model is an existing intelligent model that is widely used in various scenarios.

[0022] Furthermore, the user's voice commands are parsed through a large language model to obtain navigation tasks, including:

[0023] S21: The user performs voice input through the terminal device, and these voice commands are transmitted to the system;

[0024] S22: Inside the system, these voice commands are passed into a pre-prepared large language model for parsing;

[0025] S23: The parsed data is pattern matched using a hidden Markov model, which helps the system understand the user's intention.

[0026] Among them, the Markov chain is a random process, which is characterized by the fact that given the current state, the probability distribution of the future state depends only on the current state and has nothing to do with the historical path of the past state.

[0027] Preferably, the natural language processing technology is used to divide the navigation task into several navigation subtasks, including:

[0028] Large language models are represented as The output of the navigation subtask that has occurred is (r0, r1, … r i-1 ), the current navigation subtask is (s i ), the output of the current navigation subtask is r i , i is the task number of a positive integer;

[0029] By formula Get complex task S;

[0030] The output r of each navigation subtask i As the input of the next navigation subtask, when all subtasks are completed, the model will integrate all intermediate results and historical information and output the final execution result y; where

[0031] Preferably, the optimized paths of several navigation subtasks are obtained by combining the path planning algorithm, including:

[0032] S31: abstract the map into a set of nodes v, where node u represents the currently selected node and node v represents any adjacent node of node u;

[0033] S32: Calculate the current known shortest path length d[v] from the source point to the node v according to the weights ω[u,v] between the nodes through the path planning algorithm;

[0034] S33: In each iteration, the shortest path length d[v] to node v is updated according to the currently known shortest path length d[u] and the weight ω[u,v] from node u to v;

[0035] S34: Use an obstacle avoidance algorithm to perform path optimization to obtain an optimized path.

[0036] Preferably, an obstacle avoidance algorithm is used for path optimization, including:

[0037] By formula U a (q)=a0ρ 2 (x,x g ) Get the attraction U of the target point to the wheelchair a (q); where x g is the coordinate of the target point in the wheelchair coordinate system, x is the current wheelchair position, ρ(x,x g ) is defined as the straight-line distance between the wheelchair and the target point, and a0 is a variable parameter;

[0038] By formula To obtain the virtual gravitational force generated by the virtual force field on the wheelchair, we can use the negative gradient F of the force field. a ;

[0039] By formula Get the repulsive force U generated by the obstacle on the wheelchair obtained by the perception module req (X);

[0040] By formula Get the repulsive force F generated by the virtual repulsive field req (X);

[0041] By formula Get the virtual resultant force U(x) and the final resultant force F(x).

[0042] A path planning algorithm is introduced to ensure that the wheelchair can avoid obstacles efficiently and safely during movement.

[0043] Preferably, analyzing the environmental perception data, the voice command and the optimized path, and generating driving instructions according to the analysis results, includes:

[0044] S41: The optimized path is passed to the instruction generation module through an API call; the instruction generation module is used to parse the optimized path, and the optimized path includes determining the travel path and action strategy that the wheelchair needs to follow;

[0045] S42: According to the parsed optimized path, the instruction generation module calculates and generates driving instructions; wherein the driving instructions include speed control and steering angle parameter calculation; the steering angle calculation is based on the turning point on the path and the current position of the vehicle; the speed control is obtained based on the road conditions and the expected arrival time;

[0046] S43: The generated driving instruction is transmitted to the control unit of the wheelchair via a communication protocol;

[0047] S44: The control unit receives and analyzes the driving instruction, and extracts action information and a target path from the driving instruction;

[0048] S45: According to the action information and target path provided by the driving instruction, the control unit adjusts the execution unit of the wheelchair; wherein the execution unit includes a motor and a steering gear so that it can travel according to the specified route and speed; and the control unit continuously monitors the environmental perception data and dynamically adjusts the execution strategy based on the environmental perception data.

[0049] Users can also connect the wheelchair's Bluetooth module through a smartphone APP to issue movement commands through the phone, which increases the interactivity of the smart wheelchair and the user's controllability.

[0050] Such as obstacle information and road conditions obtained by sensors. Based on this data, the control unit may need to dynamically adjust the execution strategy to deal with emergencies or avoid potential dangers to ensure the safe driving of the wheelchair.

[0051] Preferably, the system uses a reinforcement learning algorithm to dynamically adjust driving instructions and path planning strategies based on real-time feedback from users to provide more customized navigation services, including:

[0052] A reinforcement learning algorithm is used to adjust driving instructions and path planning strategies. The Q-learning algorithm is a reinforcement learning algorithm based on value functions, which is used to learn the action value function Q(s,a) to achieve the optimal path planning strategy.

[0053] A second aspect of the present invention provides an intelligent automatic wheelchair driving system based on a large language model, comprising:

[0054] Environmental perception module: collects basic environmental data; the environmental perception module is integrated with multiple sensors, including lidar and monocular camera;

[0055] Main control module: It uses fusion algorithm to carry out fusion processing and target recognition on environmental perception data to obtain environmental perception data;

[0056] Language processing module: parses user voice commands through a large language model to obtain navigation tasks; uses natural language processing technology to divide navigation tasks into several navigation subtasks; among which, navigation subtasks include locating the current position, determining the target position, and performing path planning;

[0057] Path planning module: calls the path planning API and combines the path planning algorithm to obtain the optimized path of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm;

[0058] Instruction generation module: Analyzes environmental perception data, voice instructions and optimized paths, and generates driving instructions based on the analysis results; transmits the driving instructions to the execution unit of the wheelchair;

[0059] User feedback module: Analyzes the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjusts the driving instructions and path planning strategies; the reinforcement learning algorithm includes the Q-learning algorithm.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows: the present application realizes the depth perception of the surrounding environment and the real-time data collection by integrating data from multiple sensors such as high-precision laser radar and monocular camera. Combined with advanced path planning algorithms, the system effectively realizes intelligent obstacle avoidance and path optimization, ensuring the efficient and safe driving of wheelchairs in complex environments, greatly improving the travel safety and convenience of users; using advanced semantic understanding and task decomposition technology, the present application realizes the accurate recognition and automatic processing of user voice commands, simplifying user control. In addition, the system adopts a reinforcement learning algorithm to dynamically optimize driving instructions and path planning according to user feedback, providing a deeply personalized driving experience, meeting the personalized needs of users, and greatly improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 Schematic diagram of an intelligent automatic wheelchair driving method based on a large language model in an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the working method of the wheelchair main control module in an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of an intelligent automatic wheelchair driving system based on a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] See also Figure 1 The first aspect of the present invention provides an intelligent automatic wheelchair driving method based on a large language model, comprising:

[0067] S1: Collect basic environmental data through the environmental perception module, and perform fusion processing and target recognition on the environmental perception data through the fusion algorithm to obtain environmental perception data; wherein, the environmental perception module is integrated with multiple sensors, including laser radar and monocular camera;

[0068] Among them, the fusion algorithm is a technology that processes and combines data from two or more information sources to obtain more accurate, complete or reliable results; the fusion algorithm includes Kalman filtering, particle filtering, deep learning and other methods.

[0069] Among them, environmental perception data is collected through the environmental perception module, and a fusion algorithm is used to integrate and analyze the environmental perception data to provide more comprehensive and reliable environmental perception data, so that the wheelchair can obtain semantic information of the environment, which includes the current location of the wheelchair, the characteristics of the surrounding environment, and possible obstacle data.

[0070] Sensor integration plays a key role in the realization of autonomous wheelchairs. By utilizing multiple sensors, such as cameras, radars, and lidars, and using fusion technology to integrate and analyze the data from these sensors, the limitations of each sensor can be compensated, providing a more comprehensive and reliable environmental perception, allowing the wheelchair to obtain rich semantic information about the environment. This semantic information covers important data such as the current location of the wheelchair, the characteristics of the surrounding environment, and possible obstacles. By integrating the data from these sensors, the autonomous wheelchair can achieve efficient environmental perception, thereby effectively avoiding collisions and ensuring that the wheelchair remains stable during driving.

[0071] For example, the application of sensors is as follows:

[0072] Camera: High-definition cameras are installed on the front and both sides of the wheelchair to capture visual images and identify visual features such as color, texture, and signs, as well as dynamic objects such as pedestrians and pets.

[0073] Radar: Millimeter-wave radar is installed at the bottom of the wheelchair and uses radio waves to detect the distance, speed and direction of surrounding objects. It is particularly suitable for obstacle detection in adverse weather conditions such as rain and fog.

[0074] LiDAR: The 360-degree rotating LiDAR is located on top of the wheelchair. It emits laser pulses and receives reflected signals to generate a high-precision 3D point cloud map of the surrounding environment, accurately measuring the location and shape of obstacles.

[0075] The implementation steps of the fusion algorithm include:

[0076] S11: converting camera data into BEV features and mapping image information to a plane to facilitate cross-sensor data fusion; the BEV is a bird's-eye view;

[0077] S12: The point cloud data acquired by the lidar is also converted into BEV features to provide accurate environmental information in a dense form;

[0078] S13: At the BEV feature level, camera data is fused with laser point cloud data to integrate the information of the two sensors to enhance the comprehensiveness and accuracy of environmental perception.

[0079] S14: The fused BEV features are input into the target detection model, and the deep learning algorithm is used to identify and locate objects in the environment to obtain accurate detection results.

[0080] Among them, the target detection model is a computer vision technology and an existing technology model.

[0081] For example, the application of the fusion algorithm is as follows:

[0082] First, the data from different sensors are synchronized through timestamps to ensure that all information reflects the environmental status at the same moment.

[0083] Preprocess the raw data of each sensor to extract key features, such as the image edge of the camera, the distance information of the radar, and the point cloud features of the LiDAR.

[0084] Using algorithms such as Kalman filtering, particle filtering or deep learning fusion networks, the characteristic information of various sensors is comprehensively considered to reduce false positives and false negatives and improve perception accuracy. For example, deep neural networks are used to perform semantic segmentation on the fused data to distinguish between ground, obstacles, pedestrians and other categories.

[0085] S2: Analyze user voice commands through a large language model to obtain a navigation task; use natural language processing technology to divide the navigation task into several navigation subtasks; among which, the navigation subtasks include locating the current position, determining the target position, and performing path planning; among which, the large language model includes Chat-GPT4.0, PaLM, and Tongyi Qianwen.

[0086] In this embodiment, a large language model (Chat-GPT4.0) is used to parse the user's voice instructions, understand the user's navigation intention, decompose the complex navigation task into multiple simple subtasks, and call various APIs to realize the automatic processing of the task. The specific process includes:

[0087] The user's voice commands are parsed through a large language model to obtain navigation tasks, including:

[0088] S21: The user performs voice input through the terminal device, and these voice commands are transmitted to the system;

[0089] S22: Inside the system, these voice commands are passed into a pre-prepared large language model for parsing;

[0090] S23: The parsed data is pattern matched using a hidden Markov model, which helps the system understand the user's intention.

[0091] Among them, the large language model parses the user's voice commands. After the user inputs the voice from the terminal, the received voice information is transmitted to the pre-prepared Chat-GPT4.0 interface for recognition and processing. The processed data uses the hidden Markov model for pattern matching, and the Markov chain is used to describe the transition between states and the corresponding relationship between the state and the observed value. The large language model accurately understands the user's navigation intentions by parsing the user's voice commands, such as going to a specific location or getting close to a person.

[0092] Natural language processing technology is used to subdivide the navigation task into multiple navigation subtasks, including:

[0093] Large language models are represented as The output of the navigation subtask that has occurred is (r0, r1, … r i-1 ), the current navigation subtask is (s i ), the output of the current navigation subtask is r i , i is the task number of a positive integer;

[0094] By formula Get complex task S;

[0095] The output r of each navigation subtask i As the input of the next navigation subtask, the corresponding service or function is called through the API to ensure the consistency and automation of the task.

[0096] When all subtasks are completed, the model will integrate all intermediate results and historical information and output the final execution result y, where

[0097] S3: calling the path planning API and combining the path planning algorithm to obtain the optimized paths of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm;

[0098] In this embodiment, a path planning API is called and a path planning algorithm is introduced to ensure that the wheelchair can avoid obstacles efficiently and safely during movement.

[0099] Among them, the optimized paths of several navigation subtasks are obtained by combining the path planning algorithm, including:

[0100] S31: The system abstracts the map into a set of nodes v, where node u represents the currently selected node and node v represents any adjacent node of node u;

[0101] S32: Through the path planning algorithm, the system calculates the current known shortest path length d[v] from the source point to node v according to the weight ω[u, v] between nodes;

[0102] S33: In each iteration, based on the current known shortest path length d[u] and the weight ω[u, v] from node u to v, update the shortest path length d[v] to reach node v;

[0103] S34: The system adopts an obstacle avoidance algorithm to further optimize the path to avoid sharp turns and frequent direction changes.

[0104] For example: Through the formula: the condition d[u] + ω(u, v) < d[v] holds; then d[v] = d[u] + ω(u, v).

[0105] The goal of this task is to determine a safe and effective path according to the task requirements and environmental conditions. The user provides the coordinate or address information of the starting point and the ending point, and the system calculates the optimal path based on this information and generates the corresponding response file.

[0106] Among them, considering the dynamic characteristics of the wheelchair and the user's comfort, optimize the path to avoid sharp turns and frequent direction changes. Use the obstacle avoidance algorithm (artificial potential field method) to optimize the generated path. The attraction of the target point to the wheelchair can be described as:

[0107] U a (q) = a0ρ 2 (x, x g )

[0108] Among them, U a (q) is the attraction of the target point to the wheelchair, xg is the coordinate of the target point in the wheelchair coordinate system, x is the current wheelchair position, ρ(x, xg) is defined as the straight-line distance between the wheelchair and the target point, and the coefficient a0 is a variable parameter. The virtual gravitational force generated by this virtual force field on the wheelchair can be calculated using the negative gradient of the force field as follows:

[0109]

[0110] Similarly, the repulsive force generated by the obstacle obtained by the perception module on the wheelchair can be described by the following formula:

[0111]

[0112] Among them, k is a parameter used to adjust the intensity of the repulsive force generated by the obstacle on the wheelchair

[0113] The repulsive force generated by the virtual repulsive force field can be described as follows:

[0114]

[0115] By adding the vector forces of attraction and repulsion to the wheelchair, the virtual resultant force is:

[0116]

[0117] where F req (X) represents repulsive force, F aff (x) represents the attractive force, U(x) represents the virtual resultant force, represents the negative gradient of the attraction field, represents the gradient of the repulsive force, and F(x) represents the final resultant force

[0118] S4: Analyzes environmental perception data, voice commands, and optimized paths, and generates driving commands based on the analysis results; transmits the driving commands to the execution unit of the wheelchair;

[0119] In this embodiment, based on the analysis of the environment, voice input and path planning results, driving instructions are generated through API calls and passed to the execution unit of the wheelchair.

[0120] S5: Analyze the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjust the driving instructions and path planning strategies; among them, the reinforcement learning algorithm includes the Q-learning algorithm.

[0121] In this embodiment, at each time step, an action is selected based on the current state and the Q value is updated based on the reward signal.

[0122] By the formula Q(S t ,a t )=max(πR t+1 ) to get the state S at time step t t Take action t The expected return Q(S t ,a t ), where R t+1 is the immediate reward obtained at time step t+1; S t is the state at time step t; a t is the action taken at time step t; π is the path planning strategy;

[0123] Q(s,a) represents the expected return of taking action a in state s.

[0124] By the formula π(s) = argmax a Get the path planning strategy π(s) for selecting the path in Q(s,a) states;

[0125] Through the formula Q(s,a) = r + γmax a′Q(s′,a′) is the expected return Q(s,a) of taking action a in state s; where s′ is the next state; s is the current state; and γ is the proportional coefficient.

[0126] See also Figure 2 The embodiment of the invention provides a working method of a wheelchair master control module, the steps comprising:

[0127] S101: program initialization;

[0128] For example: after the system is powered on, the parameters in the non-registers are cleared first; the program is restored to the initialization setting program.

[0129] S102: Waiting for driving instructions;

[0130] S103: Determine whether a driving instruction is received; if yes, proceed to S104; if no, proceed to S102;

[0131] S104: moving along the driving instruction path;

[0132] S105: adjusting the driving direction;

[0133] S106: Determine whether an obstacle is detected; if yes, proceed to S107; if no, proceed to S108;

[0134] S107: bypass obstacles;

[0135] S108: Correct the driving direction to the driving direction before the detour;

[0136] S109: Determine whether the destination has been reached; if yes, end the trip; if no, proceed to S110;

[0137] S110: Determine whether there is a new driving instruction; if yes, proceed to S111; if no, proceed to S106;

[0138] S111: Update the current driving instruction to a new driving instruction, and proceed to S104 after completion.

[0139] In the present embodiment, the main control module is the core module of the entire hardware circuit control, which needs to be networked and communicated with the entire hardware system, and is responsible for the arrangement of the entire wheelchair control process and the control of computing tasks. The STM32F205 chip is selected as the main control module, and the main control module includes three submodules, namely the first submodule, the second submodule and the third submodule. The first submodule is mainly responsible for data communication with hardware devices such as various sensors. The second submodule is responsible for sensing the environment around the wheelchair, facilitating the timely detour and avoidance of obstacles. The second submodule communicates with sensors such as lasers and monocular cameras for distance measurement. The third module is mainly responsible for controlling the movement of the motor. Both wheels on the wheelchair are equipped with independent motors, which are controlled by the motor drive module to realize the straight-line and turning movements of the wheelchair so that it can reach the predetermined destination.

[0140] When the smart wheelchair is powered on from the controller, the main control module will first try to initialize each submodule and configure relevant parameters such as the addresses and related parameters of each module and sensor. If some modules cannot be initialized normally during the initialization process, the location of the problem will be reported by voice. After each module is initialized normally, it will start waiting to receive the command to control the movement of the wheelchair. After receiving the movement command, the main control program will also obtain the current movement information of the wheelchair, including the wheelchair's speed, direction and distance. With this information, the motor drive module will control the wheelchair to move accordingly. During the movement process, sensor modules such as laser radar will continuously identify the environmental information around the wheelchair, and when obstacles are identified, they will adopt appropriate obstacle avoidance and detour strategies to bypass the obstacles and continue to move forward in the previous direction. During the entire movement process, the wheelchair user can control it at any time through the joystick or mobile phone to compensate for the automatic driving algorithm. The movement speed of the wheelchair is recorded in real time by the photoelectric encoder installed on the wheel. By recording the rotation information of the two wheels, the current position of the wheelchair and the entire wheelchair movement record from power-on to the current can also be obtained.

[0141] See also Figure 3 In a first aspect, an embodiment provides an intelligent automatic wheelchair driving system based on a large language model, comprising:

[0142] Environmental perception module: collects basic environmental data; the environmental perception module is integrated with multiple sensors, including lidar and monocular camera;

[0143] Main control module: It uses fusion algorithm to carry out fusion processing and target recognition on environmental perception data to obtain environmental perception data;

[0144] Fusion algorithms include Kalman filtering, particle filtering, deep learning and other methods.

[0145] Fusion algorithm is a technology that processes and combines data from two or more information sources to obtain more accurate, complete or reliable results.

[0146] Language processing module: parse user voice commands through a large language model to obtain navigation tasks; use natural language processing technology to divide navigation tasks into several navigation subtasks; among them, navigation subtasks include locating the current position, determining the target position, and planning the path; large language models include Chat-GPT4.0, PaLM, and Tongyi Qianwen:

[0147] Natural language processing is a branch of artificial intelligence that focuses on studying how to enable computers to understand, parse, generate, and interact with natural languages ​​used by humans;

[0148] Path planning module: calls the path planning API and combines the path planning algorithm to obtain the optimized path of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm;

[0149] Instruction generation module: Analyzes environmental perception data, voice instructions and optimized paths, and generates driving instructions based on the analysis results; transmits the driving instructions to the execution unit of the wheelchair;

[0150] User feedback module: Analyzes the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjusts the driving instructions and path planning strategies; the reinforcement learning algorithm includes the Q-learning algorithm.

[0151] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0152] The working principle of the present invention is as follows: the present invention collects basic environmental data and obtains environmental perception data through a fusion algorithm; integrates the environmental perception data through a fusion algorithm and parses user voice commands through a language model to obtain a navigation task; obtains path planning for multiple navigation subtasks based on the navigation task using API technology and optimizes the path planning to obtain an optimized path; performs analysis based on environmental perception data, language commands and optimized paths, generates driving commands by calling an API and passes the commands to an execution unit of the wheelchair; the system dynamically adjusts driving commands and path planning strategies based on real-time feedback during use to provide more customized navigation services.

[0153] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An intelligent automatic wheelchair driving method based on a large language model, characterized in that: S1: Collect basic environmental data through the environmental perception module, and perform fusion processing and target recognition on the environmental perception data through the fusion algorithm to obtain environmental perception data; wherein, the environmental perception module is integrated with multiple sensors, including laser radar and monocular camera; S2: Analyze the user's voice instructions through a large language model to obtain a navigation task; use natural language processing technology to divide the navigation task into several navigation subtasks; the navigation subtasks include locating the current position, determining the target position, and planning the path; S3: calling the path planning API and combining the path planning algorithm to obtain the optimized paths of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm; S4: Analyzes environmental perception data, voice commands, and optimized paths, and generates driving commands based on the analysis results; transmits the driving commands to the execution unit of the wheelchair; S5: Analyze the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjust the driving instructions and path planning strategies; among them, the reinforcement learning algorithm includes the Q-learning algorithm.

2. The intelligent automatic wheelchair driving method based on a large language model according to claim 1, characterized in that: The obtaining of environmental perception data includes: Converting the data collected by the monocular camera into a first BEV feature, and converting the point cloud data obtained by the laser radar into a second BEV feature; wherein the BEV is a bird's-eye view; At the BEV feature level, the first BEV feature and the second BEV feature are fused through a fusion algorithm to obtain environmental fusion data; the environmental fusion data is input into a target detection model for target recognition to obtain environmental perception data; wherein, the target detection model is constructed based on a deep learning algorithm.

3. The intelligent automatic wheelchair driving method based on a large language model according to claim 1, characterized in that: The natural language processing technology is used to divide the navigation task into several navigation subtasks, including: Large language models are represented as The output of the navigation subtask that has occurred is (r0, r1, … r i-1 ), the current navigation subtask is (s i ), the output of the current navigation subtask is r i , i is the task number of a positive integer; By formula Get complex task S; The output r of each navigation subtask i As the input of the next navigation subtask, when all subtasks are completed, the model will integrate all intermediate results and historical information and output the final execution result y; where 4. The intelligent automatic wheelchair driving method based on a large language model according to claim 1, characterized in that: The combined path planning algorithm obtains optimized paths for several navigation subtasks, including: S31: abstract the map into a set of nodes v, where node u represents the currently selected node and node v represents any adjacent node of node u; S32: Calculate the current known shortest path length d[v] from the source point to the node v according to the weights ω[u,v] between the nodes through the path planning algorithm; S33: In each iteration, the shortest path length d[v] to node v is updated according to the currently known shortest path length d[u] and the weight ω[u,v] from node u to v; S34: Use an obstacle avoidance algorithm to perform path optimization to obtain an optimized path.

5. The intelligent automatic wheelchair driving method based on a large language model according to claim 4, characterized in that: The obstacle avoidance algorithm is used to perform path optimization, including: By formula U a (q)=a0ρ 2 (x,x g ) Get the attraction U of the target point to the wheelchair a (q); where x g is the coordinate of the target point in the wheelchair coordinate system, x is the current wheelchair position, ρ(x,x g ) is defined as the straight-line distance between the wheelchair and the target point, and a0 is a variable parameter; By formula To obtain the virtual gravitational force generated by the virtual force field on the wheelchair, we can use the negative gradient F of the force field. a ; By formula Get the repulsive force U generated by the obstacle on the wheelchair obtained by the perception module req (X); By formula Get the repulsive force F generated by the virtual repulsive field req (X); By formula Get the virtual resultant force U(x) and the final resultant force F(x).

6. The intelligent automatic wheelchair driving method based on a large language model according to claim 1, characterized in that: The analysis is performed based on the environmental perception data, the voice command and the optimized path, and the driving command is generated according to the analysis result, including: S41: The optimized path is passed to the instruction generation module through an API call; the instruction generation module is used to parse the optimized path, and the optimized path includes determining the travel path and action strategy that the wheelchair needs to follow; S42: According to the parsed optimized path, the instruction generation module calculates and generates driving instructions; wherein the driving instructions include speed control and steering angle parameter calculation; the steering angle calculation is based on the turning point on the path and the current position of the vehicle; the speed control is obtained based on the road condition and the expected arrival time; S43: The generated driving instruction is transmitted to the control unit of the wheelchair via a communication protocol; S44: The control unit receives and analyzes the driving instruction, and extracts action information and a target path from the driving instruction; S45: According to the action information and target path provided by the driving instruction, the control unit adjusts the execution unit of the wheelchair; wherein the execution unit includes a motor and a steering gear so that it can travel according to the specified route and speed; and the control unit continuously monitors the environmental perception data and dynamically adjusts the execution strategy based on the environmental perception data.

7. An intelligent automatic wheelchair driving system based on a large language model, characterized in that: Environmental perception module: collects basic environmental data; the environmental perception module is integrated with multiple sensors, including lidar and monocular camera; Main control module: It uses fusion algorithm to carry out fusion processing and target recognition on environmental perception data to obtain environmental perception data; Language processing module: parses user voice commands through a large language model to obtain navigation tasks; uses natural language processing technology to divide navigation tasks into several navigation subtasks; among which, navigation subtasks include locating the current position, determining the target position, and planning the path; Path planning module: calls the path planning API and combines the path planning algorithm to obtain the optimized path of several navigation subtasks; the path planning algorithm includes the Dijkstra algorithm; Instruction generation module: Analyzes environmental perception data, voice instructions and optimized paths, and generates driving instructions based on the analysis results; transmits the driving instructions to the execution unit of the wheelchair; User feedback module: Analyzes the user's real-time voice feedback based on the reinforcement learning algorithm, and dynamically adjusts the driving instructions and path planning strategies; the reinforcement learning algorithm includes the Q-learning algorithm.

Citation Information

Cited By

  • Navigation path planning method, device, equipment, robot, medium and product

    CN119347786A

  • Navigation path planning method, device, equipment, robot, medium and product

    CN119347786B

  • Safe flight agent system for executing complex language instructions in high dynamic environment

    CN119992387A

  • Safe flying agent system for executing complex language instructions in high dynamic environments

    CN119992387B

  • Autonomous mobile platform system based on VLN and multi-mode perception and control method

    CN120871824A