Electric wheelchair auxiliary driving system and method based on mobile terminal

Abstract

The invention discloses an electric wheelchair auxiliary driving system and method based on a mobile terminal, and relates to the technical field of wheelchairs, and the system comprises that the mobile terminal is provided with a man-machine interaction module, a navigation information acquisition module, a sensing obstacle avoidance module and a communication module which are connected in sequence; the man-machine interaction module is used for acquiring driving setting parameters; the navigation information acquisition module is used for acquiring the current coordinates of the mobile terminal in real time, and performing path planning according to the driving setting parameters and the current coordinates to generate navigation information; the sensing obstacle avoidance module is used for calling a rear camera of the mobile terminal to collect a road real-time image in real time, and determining a new control instruction of the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the road real-time image and the current control instruction of the electric wheelchair; the wheelchair controller is in communication connection with the communication module and is used for controlling the electric wheelchair to run according to the control instruction. Based on the scheme, the experience feeling of the user when the user uses the electric wheelchair is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of wheelchairs, and in particular to an electric wheelchair auxiliary driving system and method based on a mobile terminal. Background Art

[0002] Electric wheelchairs are a common means of transportation for special groups of people who need assisted mobility, such as the elderly, the disabled and the patients. However, most electric wheelchairs on the market can only be manually controlled by the user. For special groups, traveling often requires both physical and mental tests. Therefore, smart wheelchairs that combine electromechanical technology, sensor technology and Internet of Things technology provide a travel option for special groups with a good life travel experience. However, most existing smart wheelchairs require modification of the electric wheelchair, and the installation of a host computer and a series of sensors (such as depth cameras, laser radars, ultrasonic sensors, etc.) to achieve environmental perception and obstacle avoidance tracking functions. This will increase the total weight of the electric wheelchair, affect the power consumption of the electric wheelchair, and the modification and maintenance costs are high, resulting in a poor user experience when using the electric wheelchair. Summary of the invention

[0003] The present invention provides an electric wheelchair assisted driving system and method based on a mobile terminal, which is used to improve the technical problem that most existing electric wheelchairs need to be modified to realize functions such as environmental perception and obstacle avoidance and tracking, resulting in a poor user experience when using the electric wheelchair.

[0004] A first aspect of the present invention provides an electric wheelchair assisted driving system based on a mobile terminal, comprising a mobile terminal and a wheelchair controller;

[0005] The mobile terminal is mounted on the electric wheelchair, and is provided with a human-computer interaction module, a navigation information acquisition module, a perception obstacle avoidance module and a communication module which are connected in sequence;

[0006] The human-computer interaction module is used to obtain driving setting parameters;

[0007] The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information;

[0008] The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect real-time road images, and determine new control instructions for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road images and the current control instructions of the electric wheelchair;

[0009] The wheelchair controller is arranged on the electric wheelchair, the wheelchair controller is in communication connection with the communication module, and the wheelchair controller is used for controlling the electric wheelchair to travel according to control instructions.

[0010] Optionally, the navigation information acquisition module is specifically used to:

[0011] Collecting the current coordinates of the mobile terminal in real time;

[0012] The destination and the en route point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the en route point respectively;

[0013] The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

[0014] Optionally, the obstacle avoidance module is specifically used for:

[0015] Calling the rear camera of the mobile terminal to collect real-time road images;

[0016] Acquire a corresponding satellite map image according to the current coordinates, and convert the main road trunks in the satellite map image into a road raster map;

[0017] Inputting the road real-time image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and outputting a road segmentation bird's-eye view;

[0018] Extracting the navigation action of the current coordinates from the navigation information, performing trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and outputting the future trajectory of the electric wheelchair;

[0019] Establishing a speed constraint according to the driving mode and the driving speed in the driving setting parameters;

[0020] A model predictive control algorithm is used to comprehensively analyze the current coordinates, the future trajectory, the speed constraint and the current control instruction of the electric wheelchair to perform path tracking, and a new control instruction of the electric wheelchair is output.

[0021] Optionally, the wheelchair controller is connected to the communication module via Bluetooth.

[0022] A second aspect of the present invention provides an electric wheelchair assisted driving method based on a mobile terminal, comprising:

[0023] Get driving setting parameters;

[0024] Collecting the current coordinates of the mobile terminal in real time, and performing path planning according to the driving setting parameters and the current coordinates to generate navigation information;

[0025] Calling the rear camera of the mobile terminal to collect a real-time road image, and determining a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instruction of the electric wheelchair;

[0026] The electric wheelchair is controlled to travel according to the control instructions.

[0027] Optionally, the real-time acquisition of the current coordinates of the mobile terminal, and the path planning according to the driving setting parameters and the current coordinates to generate navigation information include:

[0028] Collecting the current coordinates of the mobile terminal in real time;

[0029] The destination and the en route point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the en route point respectively;

[0030] The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

[0031] Optionally, calling the rear camera of the mobile terminal to collect a real-time road image, and determining a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instruction of the electric wheelchair, includes:

[0032] Calling the rear camera of the mobile terminal to collect real-time road images;

[0033] Acquire a corresponding satellite map image according to the current coordinates, and convert the main road trunks in the satellite map image into a road raster map;

[0034] Inputting the road real-time image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and outputting a road segmentation bird's-eye view;

[0035] Extracting the navigation action of the current coordinates from the navigation information, performing trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and outputting the future trajectory of the electric wheelchair;

[0036] Establishing a speed constraint according to the driving mode and the driving speed in the driving setting parameters;

[0037] A model predictive control algorithm is used to comprehensively analyze the current coordinates, the future trajectory, the speed constraint and the current control instruction of the electric wheelchair to perform path tracking, and a new control instruction of the electric wheelchair is output.

[0038] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above items.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above items is implemented.

[0040] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the electric wheelchair assisted driving method based on a mobile terminal as described in any one of the above.

[0041] It can be seen from the above technical solutions that the present invention has the following advantages:

[0042] The first aspect of the above technical solution of the present invention provides an electric wheelchair assisted driving system based on a mobile terminal, including a mobile terminal and a wheelchair controller; the mobile terminal is mounted on the electric wheelchair, and the mobile terminal is provided with a human-computer interaction module, a navigation information acquisition module, a perception and obstacle avoidance module and a communication module which are connected in sequence; the human-computer interaction module is used to obtain the driving setting parameters; the navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information; the perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect the real-time road image, and determine the new control instructions of the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instructions of the electric wheelchair; the wheelchair controller is arranged on the electric wheelchair, and the wheelchair controller is communicatively connected to the communication module, and the wheelchair controller is used to control the driving of the electric wheelchair according to the control instructions. Based on the above solution, tasks such as human-computer interaction, navigation information acquisition, environmental perception and driving planning are integrated on the mobile terminal, and the electric wheelchair is controlled by wireless communication. On the one hand, there is no need to install additional sensors, host computers and other equipment, which will not cause additional impact on the power supply of the wheelchair and reduce the user's usage cost. On the other hand, when users replace electric wheelchairs of different brands, they only need to update the communication protocol according to the manufacturer's wheelchair controller. It has good applicability between wheelchairs of different brands. By giving full play to the advantages of mobile terminals in interaction and decision-making, it helps more users travel easily and conveniently, and improves the user experience when using electric wheelchairs.

[0043] The second aspect of the above technical solution of the present invention provides an electric wheelchair assisted driving method based on a mobile terminal, including: obtaining driving setting parameters; collecting the current coordinates of the mobile terminal in real time, and planning the path according to the driving setting parameters and the current coordinates to generate navigation information; calling the rear camera of the mobile terminal to collect the real-time road image, and determine the new control instructions of the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instructions of the electric wheelchair; controlling the electric wheelchair to travel according to the control instructions. Based on the above solution, tasks such as human-computer interaction, navigation information acquisition, environmental perception and driving planning are integrated on the mobile terminal to control the driving of the electric wheelchair. By giving full play to the advantages of the mobile terminal in the interaction and decision-making end, it helps more users to travel easily and conveniently, and improves the user experience when using the electric wheelchair. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A structural block diagram of an electric wheelchair assisted driving system based on a mobile terminal provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of generating navigation information by a path planning API provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a visual bird's-eye view model provided by an embodiment of the present invention;

[0048] Figure 4 A flowchart of the steps of an electric wheelchair assisted driving method for a mobile terminal provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention provide an electric wheelchair assisted driving system and method based on a mobile terminal, which is used to improve the technical problem that most existing electric wheelchairs need to be modified to realize functions such as environmental perception and obstacle avoidance and tracking, resulting in a poor user experience when using the electric wheelchair.

[0050] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 , Figure 1 A structural block diagram of an electric wheelchair assisted driving system based on a mobile terminal provided in an embodiment of the present invention.

[0052] The present invention provides an electric wheelchair auxiliary driving system based on a mobile terminal, comprising a mobile terminal and a wheelchair controller.

[0053] It should be noted that mobile terminals include but are not limited to smart phones, tablet computers, laptop computers and other devices that can be carried around and have computing, communication and data processing capabilities, among which smart phones are relatively preferred due to their good portability.

[0054] In an electric wheelchair, the wheelchair controller is the core component responsible for driving the electric wheelchair to move according to the path planning results. It mainly includes a motor controller, which can operate the motor of the electric wheelchair according to control instructions, thereby realizing the forward, backward and steering of the wheelchair.

[0055] The mobile terminal is mounted on the electric wheelchair, and is provided with a human-computer interaction module, a navigation information acquisition module, a perception obstacle avoidance module and a communication module which are connected in sequence.

[0056] The human-computer interaction module is used to obtain driving setting parameters.

[0057] It should be noted that the user can specify the destination, route, wheelchair driving mode, expected driving speed and other driving setting parameters by manual input or voice input through the human-computer interaction module on the mobile terminal. The wheelchair driving mode refers to the cruise (constant speed) / fast mode, and the expected driving speed can be understood as the expected driving gear. In a specific implementation, the human-computer interaction module can be the user interface (UI) of the mobile terminal.

[0058] The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and to plan the route according to the driving setting parameters and the current coordinates to generate navigation information.

[0059] It should be noted that since the mobile terminal is mounted on the electric wheelchair, the navigation information acquisition module obtains the current coordinates of the mobile terminal in the form of latitude and longitude coordinates in real time, and uses this as the current position of the electric wheelchair. Path planning can be performed based on the driving setting parameters and the current coordinates, thereby obtaining navigation information to help the user move from the current coordinates to the destination.

[0060] In a specific implementation, the navigation information acquisition module is specifically used to:

[0061] Collect the current coordinates of the mobile terminal in real time;

[0062] The destination and the passing point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the passing point respectively;

[0063] The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

[0064] It should be noted that after the user completes the settings in the human-computer interaction module, the geocoding API of the navigation software Web service (taking Amap as an example) can be called through the navigation information acquisition module to convert the destination and the names of the en route points specified by the user into longitude and latitude coordinates, thereby obtaining the coordinates of the destination and the en route points; according to the longitude and latitude coordinates of the current position, the en route points, and the destination, the path planning API is called to perform path planning to generate navigation information; since the main driving area of ​​the electric wheelchair is the sidewalk, the walking route planning is performed, and the entire route can be divided into several sub-segments, such as Figure 2 As shown, the obtained navigation information may include: longitude and latitude coordinates of the starting point, walking instructions (e.g. walk xx meters along xx road and go straight / turn left / turn right), segmented road names, segment lengths, navigation actions (including going straight, turning left / right, walking to the left front / right front), and segment coordinate point strings (several longitude and latitude coordinates in a segment).

[0065] It is understandable that the path planning API can be an online path planning API or an offline path planning API, such as Baidu Map API, Amap API, etc.

[0066] The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect real-time road images, and determine new control instructions for the electric wheelchair based on driving setting parameters, current coordinates, navigation information, real-time road images and current control instructions of the electric wheelchair.

[0067] It should be noted that the mobile terminal can be mounted on the electric wheelchair by means of a fixed bracket or the like. In one implementation, the rear camera of the mobile terminal is kept facing the forward direction, and the rear camera of the mobile terminal can be called to collect the real-time image of the road. In another implementation, the front camera of the mobile terminal is kept facing the forward direction, and the front camera of the mobile terminal is called to collect the real-time image of the road. Preferably, the rear camera is used, in which the user can view the displayed content in real time on the other side of the mobile terminal opposite to the rear camera, that is, the user interface;

[0068] After collecting the real-time road image, the perception and obstacle avoidance module can output new control instructions for the electric wheelchair based on the collected driving setting parameters, current coordinates, navigation information, real-time road images and the current control instructions of the called electric wheelchair, wherein the control instructions can be understood as instructions with the left wheel speed and right wheel speed of the electric wheelchair as content, the current control instructions of the electric wheelchair refer to the control instructions generated in the perception and obstacle avoidance module and sent to the wheelchair controller for driving control at the current moment, and the new control instructions refer to the control instructions for the next moment after the current control instructions are updated.

[0069] In a specific implementation, the obstacle avoidance module is specifically used to:

[0070] Call the rear camera of the mobile terminal to collect real-time road images;

[0071] Obtain the corresponding satellite map image according to the current coordinates, and convert the road trunks in the satellite map image into a road raster map;

[0072] The road real-time image and the road rasterized map are input into the preset visual bird's-eye view model for feature processing, and the road segmentation bird's-eye view is output;

[0073] Extract the navigation action of the current coordinates from the navigation information, perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and output the future trajectory of the electric wheelchair;

[0074] Building a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0075] The model predictive control algorithm is used to integrate the current coordinates, future trajectory, speed constraints and the current control instructions of the electric wheelchair to perform path tracking and output new control instructions for the electric wheelchair.

[0076] It should be noted that the perception and obstacle avoidance module uses the trained deep learning model to perform real-scene map matching to obtain the real-time driving plan of the electric wheelchair. In the implementation mode where the mobile terminal preferably uses a smartphone, the deep learning model is deployed to the smartphone for operation, and the deep learning calculation can be accelerated by scheduling the NPU on the smartphone, significantly improving the speed and efficiency of model reasoning. Compared with the CPU or GPU commonly used in tablets or laptops, the NPU consumes less energy when processing deep learning tasks, and realizes data calculation and reasoning on the device side, which enhances data privacy and security.

[0077] When using deep learning models for environmental perception and obstacle avoidance, such as Figure 3 As shown, first, the corresponding satellite map image is obtained through the navigation software APP according to the current coordinates updated in real time, and its road trunk is converted into a road rasterized map (Rasterized Map), and the real-time collected road real-time image and road rasterized map are used as the input of the preset visual bird's-eye view model. It can be understood that the visual bird's-eye view model can refer to the existing technology for details, and will not be repeated here; in the visual bird's-eye view model, on the one hand, the real-time feature map of the real-time road image is extracted through the convolutional neural network, and it is converted into a real-scene feature map under a bird's-eye view perspective through the internal and external parameters of the camera. On the other hand, the road trunk feature map in the road rasterized map is extracted through the convolutional neural network to understand the topological structure information of the road, and then the real-scene feature map and the road trunk feature map are fused and matched through cross-attention, and finally a large-scale road segmentation bird's-eye view is obtained from the fused feature map through the decoder (BEV Decoder);

[0078] It is understandable that in the field of autonomous driving, BEV (Bird's Eye View) is an important visual perception technology. It converts the environment around the driving object into a two-dimensional plane image viewed from above, providing a global perspective without occlusion and perspective deformation, which helps to improve the accuracy of target detection and tracking. At the same time, BEV provides a unified coordinate framework suitable for path planning and driving object control, which can more effectively integrate information in the time dimension and allow end-to-end optimization of perception, prediction and planning tasks within the same spatial framework. In the BEV space, the algorithm can predict the occluded area based on prior knowledge, improve the robustness of the perception system, and help to perform multiple perception tasks such as obstacle detection, lane line detection and drivable area estimation in parallel. BEV technology is one of the key technologies for realizing environmental perception in the field of autonomous driving. It helps to improve the performance and safety of autonomous driving systems by providing a unified and intuitive perspective.

[0079] The road segmentation bird's-eye view can be used for downstream tasks such as motor vehicle lane / pedestrian sidewalk division, lane line / road edge detection, pedestrian / vehicle detection, etc., to detect the passable area of ​​the current electric wheelchair section. First, the navigation action of the current coordinates is extracted from the navigation information. In the specific implementation, the last coordinate in the coordinate point string of each sub-section is used as the intersection of the sub-section and the next sub-section, and the difference between the current coordinates and the longitude and latitude of each intersection point is calculated. When the difference is less than a certain threshold, it is considered that the electric wheelchair has reached the intersection of the two sub-sections. At this time, the electric wheelchair should follow the navigation The navigation action (going straight / turning left / turning right / walking to the left / right front) in the information is used to drive to the next road section; then, a preset multi-layer perceptron decoder is used to combine the road segmentation bird's-eye view and the navigation action to predict the trajectory according to obstacles and passable areas to generate the future trajectory of the electric wheelchair on the current road section, ensuring that the electric wheelchair does not deviate from the sidewalk area and does not collide with obstacles while driving along the path; it can be understood that the future trajectory refers to a series of continuous positions passed by the electric wheelchair within a period of time, usually consisting of a series of timestamps and corresponding path point coordinates;

[0080] Common electric wheelchairs are mainly driven by rear wheels, and their kinematic model is a two-wheel differential model. The electric wheelchair can move in various directions by controlling the speed and steering of the left and right wheels respectively. This embodiment builds speed constraints according to the driving mode and driving speed in the driving setting parameters. Under the constraints of the speed constraints, the speed and steering of the left and right wheels of the electric wheelchair are calculated based on the current coordinates, future trajectory and the current control instructions of the electric wheelchair through the model predictive control algorithm to achieve trajectory tracking, and finally generates new control instructions for the speed of the left and right wheels of the electric wheelchair.

[0081] It can be understood that the Model Predictive Control (MPC) algorithm is a model-based control method that uses the dynamic model of the system to predict future behavior. Its basic idea is to generate control inputs by repeatedly solving optimization problems. It is widely used in industrial automation, robotics, vehicle control and other fields. In specific implementation, first, the dual-wheel differential kinematic state equation is established with the dual-wheel center point as the reference point:

[0082]

[0083] Among them, the state quantity is , the control quantity is , the time derivative of the state quantity is , solve the heading angle through the path point coordinates of the future trajectory ;

[0084] In the formula, is the horizontal coordinate of the path point of the future trajectory, is the ordinate of the path point of the future trajectory, is the heading angle of the waypoint of the future trajectory, is the linear speed of the electric wheelchair, is the angular velocity of the electric wheelchair, is the left wheel speed of the electric wheelchair, is the right wheel speed of the electric wheelchair, The distance between the left and right wheels of the electric wheelchair is, For the current moment , is the current position coordinate of the electric wheelchair, The vertical coordinate of the electric wheelchair's current position;

[0085] Remember the current moment The state quantity and control quantity are , , , and It is known that, among them, is the heading angle of the electric wheelchair at the current moment, is the linear velocity of the electric wheelchair at the current moment, is the angular velocity of the electric wheelchair at the current moment; given the planned path point coordinates of the electric wheelchair on the future trajectory that needs to be tracked , the state equation is linearized by Taylor expansion:

[0086] ;

[0087] Among them, record:

[0088]

[0089] Discretization transformation and simplification can be obtained:

[0090]

[0091] In the formula, is the time derivative of the state quantity at the current moment, is the deviation of the current state quantity to the reference state quantity, represents the control quantity increment, is the linear velocity increment, is the angular velocity increment, is the first matrix, is the second matrix, is the first time-varying matrix, is the second time-varying matrix, For time;

[0092] In each control cycle, MPC solves an optimization problem with the goal of minimizing a cost function. In this embodiment, the cost function Can be set to:

[0093] ;

[0094] in, Indicates the deviation of the current state quantity from the reference state quantity. is the control quantity increment, is the first weight coefficient, is the second weight coefficient, is the relaxation factor parameter to ensure that the equation has a solution;

[0095] The driving mode and driving speed are extracted from the driving setting parameters and used to construct speed constraints, including:

[0096] Specify in driving mode Range (in cruise mode and Smaller, making the speed change more gradual; in fast mode and Larger, making the speed change more obvious):

[0097] ;

[0098] Specified by driving speed Range (different driving gears specify different speed ranges):

[0099] ;

[0100] Solve the above optimization problem under the speed constraint to obtain the optimal , take one of them Then, the next moment is obtained by solving the following method Left and right wheel speed and :

[0101]

[0102] The wheelchair controller is arranged on the electric wheelchair, the wheelchair controller is communicatively connected with the communication module, and the wheelchair controller is used to control the travel of the electric wheelchair according to the control instructions.

[0103] It should be noted that the wheelchair controller establishes communication with the communication module. When the control command newly generated by the perception and obstacle avoidance module is sent to the wheelchair controller through the communication module, the wheelchair controller parses the control command and drives the motor of the electric wheelchair through the motor controller and other hardware according to the parsing result to realize the walking and steering of the wheelchair.

[0104] In a specific implementation, the wheelchair controller and the communication module may be connected by wired communication or wireless communication, wherein wireless communication is preferred, which can reduce additional configuration work and has strong mobility flexibility; wireless communication includes WiFi connection, Bluetooth connection and the like, wherein Bluetooth connection between the wheelchair controller and the communication module is preferred, which has the effects of less short-distance communication interference, lower power consumption and better real-time control response;

[0105] When the mobile terminal establishes a wireless connection with the Bluetooth module of the wheelchair controller through the built-in communication module, pairing is the first step in establishing communication between the two, including pairing capability exchange, device authentication, key generation, connection encryption, and confidential information distribution. During the pairing process, both parties will generate a long-term key (Long-Term Key, LTK). If both parties store the LTK, the subsequent connection can skip the pairing process and directly use the LTK for encrypted communication. Once the pairing is successful, a Bluetooth connection can be established, and data transmission can be achieved through the Bluetooth link layer. After the mobile terminal establishes a connection with the wheelchair controller, the service discovery protocol (Service Discovery Protocol, SDP) is used to query the services and protocols supported by the wheelchair controller. Once the service is discovered, the mobile terminal and the wheelchair controller will use a specific communication protocol (Profile) to establish an upper-layer connection, such as the Serial Port Profile (Serial Port Profile, SPP) or the Bluetooth low energy GATT (Generic Attribute Profile). The mobile terminal exchanges data with the wheelchair controller through the UUID (Universally Unique Identifier) ​​identifier. The communication module of the mobile terminal sends control instructions for the left and right wheels of the electric wheelchair through the Bluetooth interface, and the control instructions are transmitted to the wheelchair controller through the Bluetooth protocol stack.

[0106] In this implementation, tasks such as human-computer interaction, navigation information acquisition, environmental perception and driving planning are integrated on the mobile terminal, and the electric wheelchair is controlled by wireless communication. On the one hand, there is no need to install additional sensors, host computers and other equipment, which will not cause additional impact on the power supply of the wheelchair and reduce the user's usage cost. On the other hand, when the user replaces an electric wheelchair of a different brand, he only needs to update the communication protocol according to the manufacturer's wheelchair controller. It has good applicability between wheelchairs of different brands. At the same time, in the future, it will be more convenient to improve the driving intelligence of electric wheelchairs by updating the intelligent algorithm on the mobile terminal. This real-time adjustment and optimization capability is crucial for the autonomous navigation of electric wheelchairs in complex environments. By giving full play to the advantages of mobile terminals in interaction and decision-making, it helps more users travel easily and conveniently, and improves the user experience when using electric wheelchairs.

[0107] See also Figure 4 , Figure 4 A flowchart of the steps of an electric wheelchair assisted driving method based on a mobile terminal provided in an embodiment of the present invention.

[0108] The present invention provides an electric wheelchair assisted driving method based on a mobile terminal, comprising:

[0109] Step 101: Obtain driving setting parameters.

[0110] Step 102: collect the current coordinates of the mobile terminal in real time, perform route planning based on the driving setting parameters and the current coordinates, and generate navigation information.

[0111] Step 102 includes the following sub-steps:

[0112] Collect the current coordinates of the mobile terminal in real time;

[0113] The destination and the passing point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the passing point respectively;

[0114] The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

[0115] Step 103: call the rear camera of the mobile terminal to collect the real-time road image, and determine the new control instruction of the electric wheelchair based on the driving setting parameters, current coordinates, navigation information, the real-time road image and the current control instruction of the electric wheelchair.

[0116] Step 103 includes the following sub-steps:

[0117] Call the rear camera of the mobile terminal to collect real-time road images;

[0118] Obtain the corresponding satellite map image according to the current coordinates, and convert the road trunks in the satellite map image into a road raster map;

[0119] The road real-time image and the road rasterized map are input into the preset visual bird's-eye view model for feature processing, and the road segmentation bird's-eye view is output;

[0120] Extract the navigation action of the current coordinates from the navigation information, perform trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and output the future trajectory of the electric wheelchair;

[0121] Building a speed constraint according to the driving mode and driving speed in the driving setting parameters;

[0122] The model predictive control algorithm is used to integrate the current coordinates, future trajectory, speed constraints and the current control instructions of the electric wheelchair to perform path tracking and output new control instructions for the electric wheelchair.

[0123] Step 104: Control the electric wheelchair to travel according to the control instruction.

[0124] In this embodiment, tasks such as human-computer interaction, navigation information acquisition, environmental perception and driving planning are integrated on the mobile terminal, and the driving of the electric wheelchair is controlled by communication. By giving full play to the advantages of the mobile terminal in interaction and decision-making, more users can travel easily and conveniently, and the user experience when using an electric wheelchair is improved.

[0125] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the electric wheelchair assisted driving method based on a mobile terminal as described in any of the above embodiments.

[0126] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the electric wheelchair assisted driving method based on a mobile terminal as in any of the above embodiments are implemented.

[0127] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the electric wheelchair assisted driving method based on a mobile terminal as in any of the above embodiments.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the method described above can refer to the corresponding process in the aforementioned system embodiment, and will not be repeated here.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0133] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An electric wheelchair assisted driving system based on a mobile terminal, characterized in that: Including mobile terminal and wheelchair controller; The mobile terminal is mounted on the electric wheelchair, and is provided with a human-computer interaction module, a navigation information acquisition module, a perception obstacle avoidance module and a communication module which are connected in sequence; The human-computer interaction module is used to obtain driving setting parameters; The navigation information acquisition module is used to collect the current coordinates of the mobile terminal in real time, and perform path planning according to the driving setting parameters and the current coordinates to generate navigation information; The perception and obstacle avoidance module is used to call the rear camera of the mobile terminal to collect real-time road images, and determine new control instructions for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road images and the current control instructions of the electric wheelchair; The wheelchair controller is arranged on the electric wheelchair, the wheelchair controller is in communication connection with the communication module, and the wheelchair controller is used for controlling the electric wheelchair to travel according to control instructions.

2. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that: The navigation information acquisition module is specifically used for: Collecting the current coordinates of the mobile terminal in real time; The destination and the en route point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the en route point respectively; The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

3. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that: The perception and obstacle avoidance module is specifically used for: Calling the rear camera of the mobile terminal to collect real-time road images; Acquire a corresponding satellite map image according to the current coordinates, and convert the main road trunks in the satellite map image into a road raster map; Inputting the road real-time image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and outputting a road segmentation bird's-eye view; Extracting the navigation action of the current coordinates from the navigation information, performing trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and outputting the future trajectory of the electric wheelchair; Establishing a speed constraint according to the driving mode and the driving speed in the driving setting parameters; A model predictive control algorithm is used to comprehensively analyze the current coordinates, the future trajectory, the speed constraint and the current control instruction of the electric wheelchair to perform path tracking, and a new control instruction of the electric wheelchair is output.

4. The electric wheelchair assisted driving system based on a mobile terminal according to claim 1, characterized in that: The wheelchair controller is connected to the communication module via Bluetooth.

5. An electric wheelchair assisted driving method based on a mobile terminal, characterized in that: Get driving setting parameters; Collecting the current coordinates of the mobile terminal in real time, and performing path planning according to the driving setting parameters and the current coordinates to generate navigation information; Calling the rear camera of the mobile terminal to collect a real-time road image, and determining a new control instruction for the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instruction of the electric wheelchair; The electric wheelchair is controlled to travel according to the control instructions.

6. The electric wheelchair assisted driving method based on a mobile terminal according to claim 5, characterized in that: The real-time acquisition of the current coordinates of the mobile terminal, and the path planning according to the driving setting parameters and the current coordinates to generate navigation information include: Collecting the current coordinates of the mobile terminal in real time; The destination and the en route point in the driving setting parameters are converted into longitude and latitude to obtain the coordinates of the destination and the en route point respectively; The path planning API is used to plan the path based on the current coordinates, destination coordinates, and en route coordinates to generate navigation information.

7. The electric wheelchair assisted driving method based on a mobile terminal according to claim 5, characterized in that: The calling of the rear camera of the mobile terminal to collect a real-time road image, and determining a new control instruction of the electric wheelchair based on the driving setting parameters, the current coordinates, the navigation information, the real-time road image and the current control instruction of the electric wheelchair, comprises: Calling the rear camera of the mobile terminal to collect real-time road images; Acquire a corresponding satellite map image according to the current coordinates, and convert the main road trunks in the satellite map image into a road raster map; Inputting the road real-time image and the road rasterized map into a preset visual bird's-eye view model for feature processing, and outputting a road segmentation bird's-eye view; Extracting the navigation action of the current coordinates from the navigation information, performing trajectory prediction based on the navigation action and the road segmentation bird's-eye view through a preset multi-layer perceptron decoder, and outputting the future trajectory of the electric wheelchair; Establishing a speed constraint according to the driving mode and the driving speed in the driving setting parameters; A model predictive control algorithm is used to comprehensively analyze the current coordinates, the future trajectory, the speed constraint and the current control instruction of the electric wheelchair to perform path tracking, and a new control instruction of the electric wheelchair is output.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the electric wheelchair assisted driving method based on a mobile terminal as described in any one of claims 5 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by the processor, the steps of the electric wheelchair assisted driving method based on a mobile terminal are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the steps of the electric wheelchair assisted driving method based on a mobile terminal are implemented.

Images