An electric wheelchair assisted driving system and method based on a mobile terminal

Abstract

The present invention discloses an electric wheelchair assisted driving system and method based on a mobile terminal, which relates to the technical field of wheelchairs. The system includes: a mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module, and a communication module that are connected in sequence; the human-computer interaction module is used to acquire driving setting parameters; the navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and generate navigation information through path planning according to the driving setting parameters and the current coordinates; the perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect the real-time road condition images, and determine the new control instruction of the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road condition images, and the current control instruction of the electric wheelchair; the wheelchair controller is communicatively connected to the communication module, and the wheelchair controller is used to control the electric wheelchair to drive according to the control instruction. Based on the above solution, the user experience when using an electric wheelchair is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of wheelchairs, and in particular to an electric wheelchair assisted driving system and method based on a mobile terminal. Background Art

[0002] Electric wheelchairs are commonly used travel tools for special groups such as the elderly, the disabled, and patients who need assistance in moving. However, most electric wheelchairs on the market can only be manually controlled by users to drive. For special groups, traveling often poses a double test of physical strength and mental state. Therefore, intelligent wheelchairs that combine mechatronics technology, sensor technology, and Internet of Things technology provide a travel option for special groups with a good travel experience in life. However, most existing intelligent wheelchairs need to modify the electric wheelchair, and install a host computer and a series of sensors (such as depth cameras, lidar, ultrasonic sensors, etc.) to achieve functions such as environmental perception and obstacle avoidance and path tracking. This will increase the total weight of the electric wheelchair, affect the power consumption of the electric wheelchair, and the modification and maintenance costs are high, resulting in a poor experience for users when using the electric wheelchair. Summary of the Invention

[0003] The present invention provides an electric wheelchair assisted driving system and method based on a mobile terminal, which is used to improve the technical problem that most existing electric wheelchairs need to be modified to achieve functions such as environmental perception and obstacle avoidance and path tracking, resulting in a poor experience for users when using the electric wheelchair.

[0004] An electric wheelchair assisted driving system based on a mobile terminal provided by the first aspect of the present invention includes a mobile terminal and a wheelchair controller;

[0005] The mobile terminal is mounted on the electric wheelchair, and the mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module, and a communication module connected in sequence;

[0006] The human-computer interaction module is used to obtain driving setting parameters;

[0007] The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information;

[0008] The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect real-time road condition images, and determine a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road condition images, and the current control instruction of the electric wheelchair;

[0009] The wheelchair controller is arranged on the electric wheelchair, and the wheelchair controller is communicatively connected with the communication module. The wheelchair controller is used to control the electric wheelchair to drive according to the control instruction.

[0010] Optionally, the navigation information acquisition module is specifically configured to:

[0011] Collect the current coordinates of the mobile terminal in real time;

[0012] Convert the destination and waypoints in the driving setting parameters into longitude and latitude coordinates, and correspondingly obtain the destination coordinates and waypoint coordinates;

[0013] Perform path planning based on the current coordinates, destination coordinates, and waypoint coordinates through a path planning API to generate navigation information.

[0014] Optionally, the perception and obstacle avoidance module is specifically configured to:

[0015] Call the rear camera of the mobile terminal to collect real-time road condition images;

[0016] Obtain the corresponding satellite map image according to the current coordinates, and convert the road backbone in the satellite map image into a road rasterized map;

[0017] Input the road condition image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and output a road segmentation bird's-eye view;

[0018] Extract the navigation action of the current coordinates from the navigation information, and perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder to output the future trajectory of the electric wheelchair;

[0019] Construct a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0020] Adopt a model predictive control algorithm to comprehensively perform path tracking based on the current coordinates, the future trajectory, the speed constraint, and the current control command of the electric wheelchair, and output a new control command for the electric wheelchair.

[0021] Optionally, the wheelchair controller is Bluetooth-connected to the communication module.

[0022] A method for assisting in driving an electric wheelchair based on a mobile terminal provided in the second aspect of the present invention includes:

[0023] Obtain driving setting parameters;

[0024] Collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information;

[0025] Call the rear camera of the mobile terminal to collect real-time road condition images, and determine a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road condition images, and the current control instruction of the electric wheelchair;

[0026] Control the driving of the electric wheelchair according to the control instruction.

[0027] Optionally, the method further includes: collecting the current coordinates of the mobile terminal in real time, and performing path planning based on the driving setting parameters and the current coordinates to generate navigation information, including:

[0028] Collect the current coordinates of the mobile terminal in real time;

[0029] Convert the destination and waypoints in the driving setting parameters into longitude and latitude coordinates, and correspondingly obtain the destination coordinates and waypoint coordinates;

[0030] Perform path planning based on the current coordinates, destination coordinates, and waypoint coordinates through a path planning API to generate navigation information.

[0031] Optionally, the step of calling the rear camera of the mobile terminal to collect real-time road condition images, and determining a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road condition images, and the current control instruction of the electric wheelchair includes:

[0032] Call the rear camera of the mobile terminal to collect real-time road condition images;

[0033] Obtain the corresponding satellite map image according to the current coordinates, and convert the road backbone in the satellite map image into a road rasterized map;

[0034] Input the road condition images and the road rasterized map into a preset visual bird's-eye view model for feature processing, and output a road segmentation bird's-eye view;

[0035] Extract the navigation action of the current coordinates from the navigation information, and perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder to output the future trajectory of the electric wheelchair;

[0036] Construct a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0037] Adopt a model predictive control algorithm to comprehensively perform path tracking based on the current coordinates, the future trajectory, the speed constraint, and the current control instruction of the electric wheelchair, and output a new control instruction for the electric wheelchair.

[0038] A computer device provided by the third aspect of the present invention includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to execute the steps of the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above.

[0039] A computer-readable storage medium provided by the fourth aspect of the present invention has a computer program stored thereon. When the computer program is executed, it implements the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above.

[0040] A computer program product provided by the fifth aspect of the present invention includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above.

[0041] From the above technical solutions, it can be seen that the present invention has the following advantages:

[0042] The first aspect of the above technical solution of the present invention provides an electric wheelchair assisted driving system based on a mobile terminal, including a mobile terminal and a wheelchair controller; the mobile terminal is mounted on the electric wheelchair, and the mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module, and a communication module connected in sequence; the human-computer interaction module is used to obtain driving setting parameters; the navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning based on the driving setting parameters and the current coordinates to generate navigation information; the perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect the real-time road condition image, and based on the driving setting parameters, the current coordinates, the navigation information, the road condition image, and the current control instruction of the electric wheelchair, determine the new control instruction of the electric wheelchair; the wheelchair controller is arranged on the electric wheelchair, and the wheelchair controller is communicatively connected with the communication module, and the wheelchair controller is used to control the electric wheelchair to travel according to the control instruction. Based on the above solution, tasks such as human-computer interaction, navigation information acquisition, environment perception, and driving planning are integrated and completed on the mobile terminal, and the electric wheelchair is controlled to travel by wireless communication. On the one hand, there is no need to additionally install devices such as sensors and a host computer, which will not cause additional impact on the power supply of the wheelchair and also reduces the user's usage cost. On the other hand, when the user replaces electric wheelchairs of different brands, only the updated communication protocol of the wheelchair controller provided by the manufacturer needs to be followed, and the applicability between different brands of wheelchairs is better. By giving full play to the advantages of the mobile terminal in the interaction and decision-making aspects, it helps more users travel easily and conveniently, and improves the user experience when using an electric wheelchair.

[0043] The second aspect of the above technical solution of the present invention provides an electric wheelchair assisted driving method based on a mobile terminal, including: obtaining driving setting parameters; collecting the current coordinates of the mobile terminal in real time, and performing path planning according to the driving setting parameters and the current coordinates to generate navigation information; calling the rear camera of the mobile terminal to collect real-time road condition images, and determining new control instructions for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road condition images, and the current control instructions of the electric wheelchair; controlling the electric wheelchair to drive according to the control instructions. Based on the above solution, tasks such as human-computer interaction, navigation information acquisition, environmental perception, and driving planning are integrated and completed on the mobile terminal to control the electric wheelchair to drive. By giving full play to the advantages of the mobile terminal in the interaction and decision-making ends, it helps more users travel easily and conveniently, and improves the user experience when using the electric wheelchair. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0045] Figure 1 It is a structural block diagram of an electric wheelchair assisted driving system based on a mobile terminal provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of generating navigation information by a path planning API provided by an embodiment of the present invention;

[0047] Figure 3 It is a structural schematic diagram of a visual bird's-eye view model provided by an embodiment of the present invention;

[0048] Figure 4 It is a step flowchart of an electric wheelchair assisted driving method based on a mobile terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] Embodiments of the present invention provide an electric wheelchair assisted driving system and method based on a mobile terminal, which are used to solve the technical problem that most existing electric wheelchairs need to be modified to achieve functions such as environmental perception and obstacle avoidance and tracking, resulting in a poor user experience when using the electric wheelchair.

[0050] To make the objectives, features, and advantages of the present invention more apparent and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0051] Please refer to Figure 1 , Figure 1 which is a structural block diagram of an electric wheelchair assisted driving system based on a mobile terminal provided by an embodiment of the present invention.

[0052] An electric wheelchair assisted driving system based on a mobile terminal provided by the present invention includes a mobile terminal and a wheelchair controller.

[0053] It should be noted that the mobile terminal includes, but is not limited to, devices such as smartphones, tablets, and laptop computers that can be carried around and have computing, communication, and data processing capabilities. Among them, smartphones are relatively preferred due to their good portability.

[0054] In an electric wheelchair, the wheelchair controller is the core component responsible for driving the electric wheelchair to move according to the path planning result, mainly including a motor controller, which can operate the motor of the electric wheelchair according to a control instruction, so as to realize the forward, backward, and turning of the wheelchair.

[0055] The mobile terminal is mounted on the electric wheelchair, and the mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module, and a communication module connected in sequence.

[0056] The human-computer interaction module is used to obtain driving setting parameters.

[0057] It should be noted that the user can specify driving setting parameters such as the destination, waypoints, wheelchair driving mode, and expected driving speed through the human-computer interaction module on the mobile terminal in a manual input or voice input manner. The wheelchair driving mode refers to the cruise (constant speed) / fast mode, and the expected driving speed can be understood as the expected driving gear; in a specific implementation, the human-computer interaction module can be the user interface (UI) of the mobile terminal.

[0058] The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning based on the driving setting parameters and the current coordinates to generate navigation information.

[0059] It should be noted that since the mobile terminal is mounted on the electric wheelchair, the navigation information acquisition module real-time acquires the current coordinates of the mobile terminal presented in the form of latitude and longitude coordinates, and uses this as the current position of the electric wheelchair. According to the driving setting parameters and the current coordinates, path planning can be performed to obtain navigation information to help the user move from the current coordinates to the destination.

[0060] In a specific implementation manner, the navigation information acquisition module is specifically used for:

[0061] Real-time collect the current coordinates of the mobile terminal;

[0062] Convert the destination and waypoints in the driving setting parameters into latitude and longitude coordinates, and correspondingly obtain the destination coordinates and waypoint coordinates;

[0063] Perform path planning based on the current coordinates, destination coordinates and waypoint coordinates through the path planning API to generate navigation information.

[0064] It should be noted that after the user completes the settings in the human-computer interaction module, the navigation information acquisition module can call the geocoding API of the navigation software Web service (taking Amap as an example) to convert the destination and waypoint names specified by the user into latitude and longitude coordinates, so as to obtain the destination coordinates and waypoint coordinates; according to the latitude and longitude coordinates of the current location, waypoints, and destination, call the path planning API to perform path planning to generate navigation information; since the main driving area of the electric wheelchair is the sidewalk, so pedestrian route planning can be performed, and the entire route can be divided into several sub-sections, such as Figure 2 As shown, the obtained navigation information may include: starting point latitude and longitude coordinates, walking instructions (such as walking xx meters along xx Road straight / ahead / turn left / turn right), segmented road names, sub-section lengths, navigation actions (including going straight, turning left / right, walking forward left / right), sub-section coordinate point strings (several latitude and longitude coordinates in the sub-section).

[0065] It can be understood that the path planning API can be an online path planning API or an offline path planning API, such as Baidu Map API, Amap API, etc.

[0066] The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to real-time collect the road condition images, and determine the new control instructions of the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, road condition images and the current control instructions of the electric wheelchair.

[0067] It should be noted that the mobile terminal can be mounted on the electric wheelchair by means of a fixed bracket or the like. In one implementation, the rear camera of the mobile terminal is kept facing the forward direction. At this time, the rear camera of the mobile terminal can be called to collect real-time road condition images. In another implementation, the front camera of the mobile terminal is kept facing the forward direction, and the front camera of the mobile terminal is called to collect real-time road condition images; preferably, the rear camera is used. In this way, the user can view the displayed content in real time on the other side of the mobile terminal opposite to the rear camera, that is, the user interface.

[0068] After collecting the real-time road condition images, the perception and obstacle avoidance module can output a new control instruction for the electric wheelchair based on the collected driving setting parameters, current coordinates, navigation information, real-time road condition images, and the current control instruction of the called electric wheelchair. Among them, the control instruction can be understood as an instruction with the left wheel speed and right wheel speed of the electric wheelchair as the content. The current control instruction of the electric wheelchair refers to the control instruction generated in the perception and obstacle avoidance module and sent to the wheelchair controller for driving control at the current moment, while the new control instruction refers to the control instruction for the next moment after updating the current control instruction.

[0069] In a specific implementation, the perception and obstacle avoidance module is specifically used for:

[0070] Call the rear camera of the mobile terminal to collect real-time road condition images;

[0071] Obtain the corresponding satellite map image according to the current coordinates, and convert the road backbone in the satellite map image into a road rasterized map;

[0072] Input the real-time road condition image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and output a road segmentation bird's-eye view;

[0073] Extract the navigation action of the current coordinates from the navigation information, and perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and output the future trajectory of the electric wheelchair;

[0074] Construct a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0075] Adopt a model predictive control algorithm to comprehensively perform path tracking on the current coordinates, future trajectory, speed constraint, and the current control instruction of the electric wheelchair, and output a new control instruction for the electric wheelchair.

[0076] It should be noted that the perception and obstacle avoidance module uses the trained deep learning model to perform real-scene map matching to obtain the real-time driving plan of the electric wheelchair. In the implementation mode where the mobile terminal preferably uses a smart phone, the deep learning model is deployed to run on the smart phone. The NPU on the smart phone can be scheduled to accelerate the deep learning calculation, significantly improving the model inference speed and efficiency. Compared with the CPU or GPU commonly used in tablet computers or laptops, the NPU consumes less energy when processing deep learning tasks, and realizes data operation and inference at the device end, enhancing data privacy and security;

[0077] When performing environment perception and obstacle avoidance through the deep learning model, as Figure 3 shown, first, according to the currently updated coordinates in real time, the corresponding satellite map image is obtained through the navigation software APP, and its road backbone is converted into a road rasterized map (Rasterized Map). The real-time collected road live image and the road rasterized map are used as the input of the preset visual bird's-eye view model. It can be understood that the visual bird's-eye view model can refer to the existing technology specifically and will not be elaborated here. In the visual bird's-eye view model, on the one hand, the live feature map of the road live image is extracted through a convolutional neural network and converted into a real-scene feature map from a bird's-eye view through the internal and external camera parameters. On the other hand, the road backbone feature map in the road rasterized map is extracted through a convolutional neural network to understand the topological structure information of the road. Then, the real-scene feature map and the road backbone feature map are fused and matched through cross-attention. Finally, a large-scale road segmentation bird's-eye view is obtained from the fused feature map through the decoder (BEV Decoder);

[0078] It can be understood that in the field of autonomous driving, BEV (Bird's Eye View) is an important visual perception technology. It converts the environment around the driving object into a two-dimensional plane image viewed from above, providing a global view without occlusion and perspective distortion, which helps to improve the accuracy of target detection and tracking. At the same time, BEV provides a unified coordinate framework suitable for path planning and driving object control, and can more effectively fuse information in the time dimension, allowing end-to-end optimization of perception, prediction, and planning tasks within the same spatial framework. In the BEV space, the algorithm can predict the occluded area based on prior knowledge, improving the robustness of the perception system and facilitating the parallel execution of multiple perception tasks such as obstacle detection, lane line detection, and drivable area estimation. The BEV technology is one of the key technologies for realizing environment perception in the field of autonomous driving, and it helps to improve the performance and safety of the autonomous driving system by providing a unified and intuitive view;

[0079] The bird's-eye view of road segmentation can be used for downstream tasks such as motor lane / pedestrian path division, lane line / road edge detection, pedestrian / vehicle detection, etc., to achieve the detection of the passable area of the current driving section where the electric wheelchair is located. First, extract the navigation action of the current coordinate from the navigation information. In specific implementation, use the last coordinate in each sub-section coordinate point string as the junction between this sub-section and the next sub-section, calculate the difference between the current coordinate and the longitude and latitude of each junction point. When the difference is less than a certain threshold, it is considered that the electric wheelchair has traveled to the junction of two sub-sections. At this time, the electric wheelchair should travel to the next section according to the navigation action (go straight / turn left / turn right / walk left / right front) in the navigation information. Then, use a preset multi-layer perceptron decoder to combine the bird's-eye view of road segmentation and the navigation action, and predict the trajectory according to the obstacles and the passable area to generate the future trajectory of the electric wheelchair driving in the current section, ensuring that the electric wheelchair does not deviate from the sidewalk area and does not collide with obstacles during the driving along this path. It can be understood that the future trajectory refers to a series of continuous positions passed by the electric wheelchair within a period of time, usually composed of a series of timestamps and corresponding path point coordinates.

[0080] Common electric wheelchairs are mainly driven by the rear wheels, and their kinematic model is a two-wheel differential model. By controlling the speed and steering of the left and right wheels respectively, the electric wheelchair can move in all directions. In this embodiment, a speed constraint is constructed according to the driving mode and driving speed in the driving setting parameters. Under the constraint of the speed constraint, through the model predictive control algorithm, based on the current coordinate, the future trajectory and the current control command of the electric wheelchair, calculate the speed and steering of the left and right wheels of the electric wheelchair to achieve trajectory tracking, and finally generate a new control command for the speed of the left and right wheels of the electric wheelchair.

[0081] It can be understood that the model predictive control (MPC) algorithm is a model-based control method that uses the dynamic model of the system to predict future behavior. Its basic idea is to generate control inputs by repeatedly solving optimization problems, and it is widely used in industrial automation, robotics, vehicle control and other fields. In specific implementation, first, establish a two-wheel differential kinematic state equation with the center point of the two wheels as the reference point:

[0082]

[0083] Among them, denote the state quantity as , and the control quantity as , the derivative of the state quantity with respect to time is , and solve the heading angle through the path point coordinates of the future trajectory;

[0084] In the formula, is the abscissa of the path point of the future trajectory, is the ordinate of the path point of the future trajectory, is the heading angle of the path point of the future trajectory, is the linear velocity of the electric wheelchair, is the angular velocity of the electric wheelchair, is the left wheel speed of the electric wheelchair, is the right wheel speed of the electric wheelchair, is the distance between the left and right wheels of the electric wheelchair, is the current moment , is the positioning abscissa of the electric wheelchair at the current moment, is the positioning ordinate of the electric wheelchair at the current moment;

[0085] Denote the state variables and control variables at the current moment as , , , and are known, where is is the heading angle of the electric wheelchair at the current moment, is the linear velocity of the electric wheelchair at the current moment, is the angular velocity of the electric wheelchair at the current moment; Given the path point coordinates on the future trajectory that the electric wheelchair needs to track obtained from the planning , perform linearization processing on the state equation through Taylor expansion:

[0086] ;

[0087] Among them, denote:

[0088]

[0089] Perform discretization transformation and simplification to obtain:

[0090]

[0091] In the formula, is the derivative of the state variable at the current moment with respect to time, is the deviation of the state variable at the current moment from the reference state variable, represents the control variable increment, is the linear velocity increment, is the angular velocity increment, is the first matrix, is the second matrix, is the first time-varying matrix, is the second time-varying matrix, is the time;

[0092] In each control cycle, the MPC solves an optimization problem with the goal of minimizing a cost function. In this embodiment, the cost function can be set as:

[0093] ;

[0094] where, represents the deviation of the state quantity at the current moment from the reference state quantity, is the control quantity increment, is the first weight coefficient, is the second weight coefficient, is the relaxation factor parameter to ensure that the equation has a solution;

[0095] Extract the driving mode and driving speed from the driving setting parameters, and construct speed constraints based on them, including:

[0096] Specify by driving mode range (in cruise mode and are smaller, making the speed change smoother; in fast mode and are larger, making the speed change more obvious):

[0097] ;

[0098] Specify by driving speed range (different driving gears specify different speed ranges):

[0099] ;

[0100] Solve the above optimization problem under the speed constraints to obtain the optimal , take the among them, and then calculate the left and right wheel speeds at the next moment according to the following method and :

[0101]

[0102] The wheelchair controller is set on the electric wheelchair. The wheelchair controller is communicatively connected to the communication module. The wheelchair controller is used to control the driving of the electric wheelchair according to the control instruction.

[0103] It should be noted that communication is established between the wheelchair controller and the communication module. When the control instruction newly generated by the obstacle avoidance module is sent to the wheelchair controller through the communication module, the wheelchair controller parses the control instruction and drives the motor of the electric wheelchair through hardware such as the motor controller according to the parsing result to realize the walking and turning of the wheelchair;

[0104] In specific implementation, the connection between the wheelchair controller and the communication module can be a wired communication connection or a wireless communication connection. Preferably, it is a wireless communication connection, which can reduce additional configuration work and has strong mobility; wireless communication includes methods such as WiFi connection and Bluetooth connection. Preferably, the wheelchair controller is Bluetooth-connected to the communication module, which has the advantages of less short-distance communication interference, lower power consumption, and better real-time control response;

[0105] When the mobile terminal establishes a wireless connection with the Bluetooth module of the wheelchair controller through the built-in communication module, pairing is the first step in establishing communication between the two, including processes such as pairing capability exchange, device authentication, key generation, connection encryption, and confidential information distribution; during the pairing process, both parties will generate a Long-Term Key (LTK). If both parties store the LTK, then the subsequent connection can skip the pairing process and directly use the LTK for encrypted communication; once the pairing is successful, a Bluetooth connection can be established, and data transmission is achieved through the Link Layer of Bluetooth; after the mobile terminal and the wheelchair controller establish a connection, they query the services and protocols supported by the wheelchair controller through the Service Discovery Protocol (SDP). Once the service is discovered, the mobile terminal and the wheelchair controller will use a specific communication protocol (Profile) to establish an upper-layer connection, such as the Serial Port Profile (SPP) or the Generic Attribute Profile (GATT) of Bluetooth Low Energy; the mobile terminal uses the Universally Unique Identifier (UUID) to identify and exchange data with the wheelchair controller; the communication module of the mobile terminal sends control commands for the left and right wheels of the electric wheelchair through the Bluetooth interface, and the control commands are transmitted to the wheelchair controller through the Bluetooth protocol stack.

[0106] In this implementation, tasks such as human-computer interaction, navigation information acquisition, environment perception, and driving planning are integrated on the mobile terminal, and the electric wheelchair is controlled to drive through wireless communication. On the one hand, there is no need to additionally install devices such as sensors and host computers, which will not cause additional impact on the power supply of the wheelchair and also reduce the user's usage cost. On the other hand, when the user replaces electric wheelchairs of different brands, they only need to update the communication protocol according to the wheelchair controller provided by the manufacturer, and the applicability between different brands of wheelchairs is better. At the same time, in the future, it will be more convenient to improve the driving intelligence of the electric wheelchair by updating the intelligent algorithm on the mobile terminal. This real-time adjustment and optimization ability is crucial for the autonomous navigation of the electric wheelchair in a complex environment. By giving full play to the advantages of the mobile terminal in the interaction and decision-making aspects, it helps more users travel easily and conveniently, and improves the user experience when using the electric wheelchair.

[0107] Please refer to Figure 4 , Figure 4 which is the flowchart of the steps of an electric wheelchair assisted driving method based on a mobile terminal provided by an embodiment of the present invention.

[0108] An electric wheelchair assisted driving method based on a mobile terminal provided by the present invention includes:[[]]

[0109] Step 101, obtain driving setting parameters.

[0110] Step 102, collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information.

[0111] Step 102 includes the following sub-steps:[[]]

[0112] Collect the current coordinates of the mobile terminal in real time;

[0113] Convert the destination and waypoints in the driving setting parameters into longitude and latitude coordinates, and correspondingly obtain the destination coordinates and waypoint coordinates;

[0114] Perform path planning based on the current coordinates, destination coordinates and waypoint coordinates through the path planning API to generate navigation information.

[0115] Step 103, call the rear camera of the mobile terminal to collect the real-time road condition image, and determine the new control instruction of the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, road condition image and the current control instruction of the electric wheelchair.

[0116] Step 103 includes the following sub-steps:[[]]

[0117] Call the rear camera of the mobile terminal to collect the real-time road condition image;

[0118] Obtain the corresponding satellite map image according to the current coordinates, and convert the road backbone in the satellite map image into a road rasterized map;

[0119] Input the road condition image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and output a road segmentation bird's-eye view;

[0120] Extract the navigation action of the current coordinates from the navigation information, and perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder to output the future trajectory of the electric wheelchair;

[0121] Construct a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0122] The model predictive control algorithm is used to comprehensively consider the current coordinates, future trajectory, speed constraints, and the current control command of the electric wheelchair for path tracking, and output a new control command for the electric wheelchair.

[0123] Step 104: Control the electric wheelchair to travel according to the control command.

[0124] In this embodiment, tasks such as human-computer interaction, navigation information acquisition, environment perception, and travel planning are integrated on the mobile terminal to communicate and control the travel of the electric wheelchair. By giving full play to the advantages of the mobile terminal in the interaction and decision-making aspects, it helps more users travel easily and conveniently, improving the user experience when using the electric wheelchair.

[0125] The embodiment of the present invention also provides a computer device, including a memory and a processor, and a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the electric wheelchair assisted driving method based on the mobile terminal in any of the above embodiments.

[0126] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program / instructions are stored, and when the computer program / instructions are executed by the processor, the steps of the electric wheelchair assisted driving method based on the mobile terminal in any of the above embodiments are implemented.

[0127] The embodiment of the present invention also provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by the processor, the steps of the electric wheelchair assisted driving method based on the mobile terminal in any of the above embodiments are implemented.

[0128] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the method described above can refer to the corresponding process in the foregoing system embodiment, and will not be elaborated here.

[0129] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0130] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0132] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0133] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. An electric wheelchair assisted driving system based on a mobile terminal, characterized in that, It includes a mobile terminal and a wheelchair controller; The mobile terminal is mounted on an electric wheelchair, and the mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module, and a communication module that are connected in sequence; The human-computer interaction module is used to obtain driving setting parameters; The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning based on the driving setting parameters and the current coordinates to generate navigation information; The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect real-time road condition images, and determine a new control instruction for the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, road condition images, and the current control instruction of the electric wheelchair; The wheelchair controller is arranged on the electric wheelchair, and the wheelchair controller is communicatively connected with the communication module, and the wheelchair controller is used to control the driving of the electric wheelchair according to the control instruction.

2. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that, The navigation information acquisition module is specifically used for: Collecting the current coordinates of the mobile terminal in real time; Performing longitude and latitude conversion on the destination and waypoints in the driving setting parameters to obtain the destination coordinates and waypoint coordinates correspondingly; Performing path planning based on the current coordinates, destination coordinates, and waypoint coordinates through a path planning API to generate navigation information.

3. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that, The perception and obstacle avoidance module is specifically used for: Calling the rear camera of the mobile terminal to collect real-time road condition images; Obtaining the corresponding satellite map image according to the current coordinates, and converting the road backbone in the satellite map image into a road rasterized map; Inputting the road condition image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and outputting a road segmentation bird's-eye view; Extracting the navigation action of the current coordinates from the navigation information, and performing trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder to output the future trajectory of the electric wheelchair; Constructing a speed constraint according to the driving mode and driving speed in the driving setting parameters; Adopting a model predictive control algorithm to comprehensively perform path tracking based on the current coordinates, the future trajectory, the speed constraint, and the current control instruction of the electric wheelchair, and output a new control instruction for the electric wheelchair.

4. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that, The wheelchair controller is connected to the communication module via Bluetooth.

5. An electric wheelchair assisted driving method based on a mobile terminal, characterized in that Obtaining driving setting parameters; Collecting the current coordinates of the mobile terminal in real time, and performing path planning based on the driving setting parameters and the current coordinates to generate navigation information; Calling the rear camera of the mobile terminal to collect real-time road condition images, and determining a new control instruction for the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, road condition images, and the current control instruction of the electric wheelchair; Controlling the driving of the electric wheelchair according to the control instruction.

6. The electric wheelchair assisted driving method based on a mobile terminal according to claim 5, wherein The collecting the current coordinates of the mobile terminal in real time, and performing path planning based on the driving setting parameters and the current coordinates to generate navigation information includes: Collecting the current coordinates of the mobile terminal in real time; Convert the destination and waypoints in the driving setting parameters into longitude and latitude coordinates, and correspondingly obtain the destination coordinates and waypoint coordinates. Based on the current coordinates, destination coordinates and waypoint coordinates, perform route planning through the route planning API to generate navigation information.

7. The electric wheelchair assisted driving method based on a mobile terminal according to claim 5, characterized in that, The method of calling the rear camera of the mobile terminal to collect real-time road condition images and determining the new control instruction of the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, road condition images and current control instruction of the electric wheelchair includes: Call the rear camera of the mobile terminal to collect real-time road condition images. Obtain the corresponding satellite map image according to the current coordinates, and convert the road backbone in the satellite map image into a road rasterized map. Input the road condition image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and output a road segmentation bird's-eye view. Extract the navigation action of the current coordinates from the navigation information, and perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder to output the future trajectory of the electric wheelchair. Construct a speed constraint according to the driving mode and driving speed in the driving setting parameters. Adopt a model predictive control algorithm to perform path tracking by integrating the current coordinates, the future trajectory, the speed constraint and the current control instruction of the electric wheelchair, and output the new control instruction of the electric wheelchair.

8. A computer device, characterized in that, It includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the steps of the method for assisting in driving an electric wheelchair based on a mobile terminal according to any one of claims 5-7.

9. A computer-readable storage medium having computer programs / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method for assisting in driving an electric wheelchair based on a mobile terminal according to any one of claims 5-7 are implemented.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method for assisting in driving an electric wheelchair based on a mobile terminal according to any one of claims 5-7 are implemented.

Images