Multi-mode automatic following obstacle avoidance wheelchair control method, device, equipment and medium

Abstract

The invention relates to a multi-mode automatic following obstacle avoidance wheelchair control method and device, equipment and a medium, and the method comprises the steps: determining a target passenger to be in a seating state when detecting that an energy accumulation value does not exceed a preset energy accumulation threshold value, extracting an image key point in a target sidewalk image through employing a Sh i-Toma i algorithm, and obtaining a target sidewalk image; calculating a motion vector of each image key point between two continuous frames of target sidewalk images by adopting an optical flow method so as to determine a dynamic point proportion; and determining a linear velocity constraint of the velocity type to which the automatic following obstacle avoidance wheelchair belongs in each environment according to the dynamic point proportion, and performing path planning according to the linear velocity constraint by adopting an improved cost function corresponding to a DWA algorithm so as to determine an automatic obstacle avoidance track of the automatic following obstacle avoidance wheelchair in a second cost map. The environment adaptive capacity, the target tracking capacity, the dynamic obstacle avoidance capacity, the path planning capacity and the intelligent level of the wheelchair are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of wheelchair control, and in particular to a multi-mode automatic following and obstacle-avoiding wheelchair control method, corresponding device, electronic device and computer-readable storage medium. Background Art

[0002] As China enters an aging society, smart wheelchairs are rapidly developing and attracting widespread attention. Demand for wheelchairs is increasing among elderly users and those undergoing rehabilitation. However, traditional electric wheelchairs rely on manual control, making them difficult for those with limited mobility. Existing tracking wheelchairs often use a single sensor (such as UWB or RFID), resulting in poor environmental adaptability, inability to recognize user gestures and intentions, and a lack of dynamic obstacle avoidance. Furthermore, existing systems lack scenario-based intelligent speed control mechanisms, making it difficult to balance safety and efficiency in complex environments.

[0003] To sum up, in order to adapt to the problems in the existing technology, such as the lack of a scenario-based intelligent speed regulation mechanism in the existing system and the inability to balance safety and efficiency in a complex environment, the applicant has made corresponding explorations to solve this problem. Summary of the Invention

[0004] The purpose of this application is to solve the above problems and provide a multi-mode automatic following obstacle avoidance wheelchair control method, corresponding device, electronic device and computer-readable storage medium.

[0005] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0006] A multi-mode automatic following obstacle avoidance wheelchair control method is proposed to meet one of the purposes of this application, including:

[0007] Obtaining a target sidewalk image containing obstacles from a binocular vision module of the automatic following obstacle-avoiding wheelchair, performing obstacle detection on the target sidewalk image using a trained target detection model to determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, and integrating costs corresponding to the static obstacles, dynamic obstacles, and safe driving area into a first costmap to determine a second costmap;

[0008] Obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range; if it is detected that the energy accumulation value exceeds a preset energy accumulation threshold, determining that the target passenger in the automatic following obstacle-avoiding wheelchair is in an out-of-seat state, calling a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger, and using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait;

[0009] If it is detected that the energy accumulation value does not exceed a preset energy accumulation threshold, the target passenger is determined to be in a seated state, the Shi-Tomasi algorithm is used to extract image key points in the target sidewalk image, and the optical flow method is used to calculate the motion vector of each image key point between two consecutive frames of the target sidewalk image to determine the dynamic point ratio;

[0010] The linear speed constraint of the speed type of the automatic following and obstacle-avoiding wheelchair in each environment is determined according to the proportion of dynamic points, and the improved cost function corresponding to the DWA algorithm is used to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following and obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following and obstacle-avoiding wheelchair.

[0011] Optionally, the step of obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine the energy accumulation value within a preset time range includes:

[0012] Obtaining a two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor, dividing each element position point in the two-dimensional pressure matrix into a core element position point and a non-core element position point, and obtaining pressure values corresponding to the core element position point and the non-core element position point in the two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor;

[0013] Determining a first weight coefficient of the core element position point and a second weight coefficient of the non-core element position point in the two-dimensional pressure matrix, calculating and determining a first product between the first weight coefficient and the pressure value of the core element position point, and calculating and determining a second product between the second weight coefficient and the pressure value of the non-core element position point;

[0014] The first products corresponding to each core element position point are summed to determine a first sum value, the second products corresponding to each non-core element position point are summed to determine a second sum value, and the total pressure value at each moment in the array pressure-sensitive sensor is determined based on a third sum value between the first sum value and the second sum value.

[0015] Optionally, the step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes:

[0016] Calling the target detection model that has been trained to a converged state to perform target detection on a target sidewalk image containing a target passenger to determine key point positions of the target passenger, wherein the key point positions include hip key points, knee key points, and ankle key points, wherein the hip key points include a left hip key point and a right hip key point, the knee key points include a left knee key point and a right knee key point, and the ankle key points include a left ankle key point and a right ankle key point;

[0017] Get the first vector between the left hip keypoint and the left knee keypoint, the second vector between the left knee keypoint and the left ankle keypoint, the third vector between the right hip keypoint and the right knee keypoint, the fourth vector between the right knee keypoint and the right ankle keypoint, the first distance between the left hip keypoint and the left knee keypoint, the second distance between the left knee keypoint and the left ankle keypoint, the third distance between the right hip keypoint and the right knee keypoint, and the fourth distance between the right knee keypoint and the right ankle keypoint;

[0018] calculating and determining a third product between the first vector and the second vector, calculating and determining a fourth product between the first distance and the second distance, and using an arccosine function value of a first ratio between the third product and the fourth product as a hip, knee, and ankle joint angle of the left leg;

[0019] Calculate and determine the fifth product between the third vector and the fourth vector, calculate and determine the sixth product between the third distance and the fourth distance, and use the inverse cosine function value of the second ratio between the fifth product and the sixth product as the hip, knee and ankle joint angle of the right leg.

[0020] Optionally, the step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes:

[0021] Obtaining the target passenger's initial standing height, current standing height, hip-knee-ankle joint angles of the left leg, and hip-knee-ankle joint angles of the right leg, wherein the standing height represents the height from the top of the target passenger's head to the soles of their feet;

[0022] If it is detected that the hip-knee-ankle joint angle of the left leg and the hip-knee-ankle joint angle of the right leg are both less than a preset angle threshold, and a third ratio between the current standing height and the initial standing height is less than a preset ratio threshold, the target passenger is determined to be in a sitting state, and the automatic following obstacle-avoiding wheelchair is driven to stop and wait at a position parallel to the direction of the target passenger according to a preset following distance.

[0023] Optionally, the step of calling a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger includes:

[0024] When the target passenger is detected to be out of his seat, the StrongSORT algorithm is used to track the target passenger. Based on the state of the target passenger in the previous frame image, the Kalman filter is used to predict and correct the current target position. The ReID module is used to extract the appearance feature vector of the target passenger to calculate the cosine similarity with its historical trajectory to determine the current position of the target passenger.

[0025] Optionally, the step of using an improved cost function corresponding to the DWA algorithm to perform path planning according to the linear velocity constraint to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map includes:

[0026] Get the height cost, smoothing cost, distance cost and heading cost;

[0027] Constructing a trajectory evaluation function according to the height cost, the smoothness cost, the distance cost, and the heading cost;

[0028] The DWA algorithm is used to determine the optimal automatic obstacle avoidance trajectory of the automatic following obstacle avoidance wheelchair in the second cost map according to the improved cost function and the trajectory evaluation function, so as to complete the control of the multi-mode automatic following obstacle avoidance wheelchair.

[0029] Optionally, the basic network architecture of the target detection model includes a YOLOv8s target detection model.

[0030] A multi-mode automatic following obstacle avoidance wheelchair control device provided for another purpose of the present application includes:

[0031] an obstacle detection module configured to obtain a target sidewalk image containing obstacles from a binocular vision module of the automatic following obstacle-avoiding wheelchair, perform obstacle detection on the target sidewalk image using a target detection model that has been trained to a converged state, to determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, and incorporate costs corresponding to the static obstacles, the dynamic obstacles, and the safe driving area into a first costmap to determine a second costmap;

[0032] a target tracking module configured to obtain the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range, and upon detecting that the energy accumulation value exceeds a preset energy accumulation threshold, determine that a target passenger in the automatic following obstacle-avoiding wheelchair is in an out-of-seat state, invoke a preset StrongSORT algorithm to perform target tracking on the target passenger to determine the current position of the target passenger, and employ a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait;

[0033] a dynamic point ratio determination module configured to, upon detecting that the energy accumulation value does not exceed a preset energy accumulation threshold, determine the target passenger as being seated, extract image key points from the target sidewalk image using a Shi-Tomasi algorithm, and calculate a motion vector of each image key point between two consecutive frames of the target sidewalk image using an optical flow method to determine a dynamic point ratio;

[0034] The obstacle avoidance trajectory determination module is configured to determine the linear speed constraint of the speed type of the automatic following obstacle-avoiding wheelchair in each environment according to the proportion of the dynamic points, and use the improved cost function corresponding to the DWA algorithm to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following obstacle-avoiding wheelchair.

[0035] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the multi-mode automatic following obstacle avoidance wheelchair control method described in the present application.

[0036] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the multi-mode automatic following obstacle-avoiding wheelchair control method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0037] Compared with the existing technology, this application addresses the problems that the existing system lacks a scenario-based intelligent speed regulation mechanism and cannot balance safety and efficiency in complex environments. This application includes but is not limited to the following beneficial effects:

[0038] First, traditional electric wheelchairs often rely on a single sensor (such as UWB or RFID), which has poor environmental adaptability and is difficult to cope with changes in obstacles in complex environments. In contrast, this application uses a combination of a binocular vision module and an array of pressure-sensitive sensors to obtain real-time visual image information and pressure sensing data from the environment, improving the wheelchair's environmental perception capabilities, enabling more accurate identification of static and dynamic obstacles, and ensuring that the wheelchair can autonomously avoid obstacles in complex environments.

[0039] Secondly, traditional wheelchair systems often fail to identify the user's dynamic state when tracking a wheelchair, such as whether the user is unseated or moving. This application, by introducing the StrongSORT algorithm, enables real-time tracking of the target passenger upon detecting their unseated state. This target tracking method accurately determines the target passenger's position, ensuring that the wheelchair can dock accurately when needed, avoiding misoperation or misalignment.

[0040] Third, by combining a wheelchair-following strategy with target tracking technology, this application can intelligently drive a wheelchair to automatically follow the occupant based on their real-time location. When the target person's accumulated energy exceeds a preset threshold, the system determines that the user has left their seat, triggering the wheelchair to automatically follow and dock. This mechanism enables autonomous following and docking, improving the user experience and is particularly user-friendly for users with limited mobility.

[0041] Fourth, by using the DWA algorithm and an improved cost function, combined with the Shi-Tomasi algorithm and optical flow, this application can calculate speed constraints based on the proportion of dynamic points in the target sidewalk image and accurately plan paths. This allows the wheelchair to dynamically adjust its path in complex environments based on real-time perception information, avoiding obstacles and automatically adjusting its speed based on environmental speed constraints, ensuring a balance between safety and efficiency.

[0042] Fifth, this application uses a continuous feedback mechanism to dynamically adjust the wheelchair's driving state based on actual conditions. When the user is in the "seated state," the system uses image key point extraction and motion vector calculation to determine the proportion of dynamic obstacles in the environment, and then determines the appropriate movement speed and trajectory planning. This not only improves the wheelchair's operating safety, but also ensures that it can smoothly adapt to changing environmental conditions during driving.

[0043] Sixth, traditional wheelchairs rely on manual control by the user, and there is a certain operational threshold. The intelligent features of this system (such as automatic obstacle avoidance, automatic following, intelligent docking, etc.) greatly reduce dependence on users, especially for the elderly and patients with limited mobility, reducing the difficulty of operation and improving the convenience and safety of use. Through a multi-modal perception and control mechanism, this application enables the automatic following obstacle avoidance wheelchair to autonomously understand and respond to different scenarios and user needs, which not only enhances the intelligence level of the wheelchair, but also provides a personalized user experience to adapt to the needs of different environments and scenarios.

[0044] Furthermore, this application effectively improves the wheelchair's environmental adaptability, target tracking ability, dynamic obstacle avoidance ability, path planning ability and intelligence level, thus having broad application prospects in an aging society and rehabilitation field, especially for patients with limited mobility, greatly improving the convenience, safety and comfort of using the wheelchair. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 This is an exemplary architecture used by the multi-mode automatic following obstacle avoidance wheelchair in the embodiments of the present application;

[0047] Figure 2 This is a flow chart of a multi-mode automatic following obstacle avoidance wheelchair control method according to an embodiment of the present application;

[0048] Figure 3 This is a functional block diagram of a multi-mode automatic following obstacle avoidance wheelchair control device in an embodiment of the present application;

[0049] Figure 4 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0051] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0053] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0054] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0055] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0056] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0057] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0058] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0059] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0060] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0061] See also Figure 1The automatic following obstacle-avoiding wheelchair of the present application includes a sensor component 1, which consists of a binocular vision module 8, a millimeter-wave radar 11, an ultrasonic module 12, an array pressure-sensitive sensor 13, a data output module 15, a voice recognition module 14 and a data output module 15, wherein the binocular vision module 8 is composed of a binocular camera 9 and an infrared module 10. The data acquisition module 7 controls the connection to the binocular vision module 8 and outputs to the data output module 15; the data acquisition module 7 controls the connection to the millimeter wave radar module 11 and outputs to the data output module 15; the data acquisition module 7 controls the connection to the ultrasonic module 12 and outputs to the data output module 15; the data acquisition module 7 controls the connection to the array pressure-sensitive sensor 13 and outputs to the data output module 15; the data acquisition module 7 controls the connection to the voice recognition module 14 and outputs to the data output module 15; the real-time environment information component 2 is composed of an environment information collection module 21, an environment information processing module 22 and an environment information output module 23, wherein the data output module 15 is connected to the environment information collection module 21 to control the connection to the environment information processing module 22, and the environment information processing module 22 controls the connection to the environment information output module 23; the obstacle avoidance path planning component is composed of an obstacle information collection module 21, an environment information processing module 22 and an environment information output module 23. The target passenger posture capture and following component consists of a target passenger information receiving module 16, a posture information processing module 18, a posture information output module 18, a position and speed information processing module 19, and a position and speed prediction output module 20. The data output module 15 is connected to the target passenger information receiving module 16 and controls the connection to the posture information processing module 17 and the position and speed information processing module 19, respectively. The posture information processing module 17 controls the connection to the posture information output module 18, and the position and speed information processing module 19 controls the position and speed prediction output module 20. The multi-algorithm fusion processing module 6 receives information from the data output module 15, the environmental information output module 23, the posture information output module 18, the position and speed output module 20, and the obstacle avoidance path output module 26, performs multi-algorithm fusion processing, and then controls the drive module 5.

[0062] In a further embodiment, multiple sensors are used to collect environmental data and perform data fusion. A data acquisition module controls the start and stop of data collection by multiple sensor modules. A binocular camera collects RGB images of the environment with a resolution of 1920×1080 and a frame rate of 60fps to obtain visual information. A millimeter-wave radar module collects point cloud data to obtain three-dimensional information about the environment. An infrared module and an ultrasonic module serve as auxiliary modules for the binocular camera to detect visual blind spots. A voice recognition module serves as the main module for human-computer interaction to identify commands from the target passenger. An array of pressure-sensitive sensors obtains pressure sensations to detect wheelchair occupancy. The data and output module outputs the information collected by the sensor modules.

[0063] Based on the above example scenario, please refer to Figure 2 In one embodiment, the multi-mode automatic following obstacle avoidance wheelchair control method of the present application includes:

[0064] Step S10: Obtain a target sidewalk image containing obstacles from the binocular vision module of the automatic following obstacle-avoiding wheelchair, perform obstacle detection on the target sidewalk image using a target detection model that has been trained to a converged state, determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, and incorporate the costs corresponding to the static obstacles, dynamic obstacles, and safe driving area into a first costmap to determine a second costmap.

[0065] The automatic following and obstacle-avoiding wheelchair control system can obtain a target sidewalk image containing obstacles from the wheelchair's binocular vision module, perform obstacle detection on the target sidewalk image using a trained, converged object detection model, and identify static obstacles, dynamic obstacles, and safe driving areas within the target sidewalk image. The costs corresponding to the static obstacles, dynamic obstacles, and safe driving areas are then incorporated into a first cost map to determine a second cost map. The object detection model's underlying network architecture includes a YOLOv8s object detection model. Static obstacles include fire hydrants, flower beds, and plastic bottles, among others; dynamic obstacles include pedestrians or animals. The cost map is a two-dimensional image or data structure that represents the "cost" of each location in the wheelchair's path, representing the degree of impact of an obstacle on the safe driving area. Each pixel in the cost map represents a specific spatial location. Lower cost values indicate a safer location and allow the wheelchair to pass smoothly. Higher cost values indicate dense obstacles or significant obstruction, preventing the wheelchair from passing. The first cost map is an initial cost map, and the second cost map is a cost map determined after integrating the costs corresponding to the static obstacles, the dynamic obstacles, and the safe driving area into the initial cost map.

[0066] In some embodiments, binocular cameras can be used to collect environmental road information for different road types in different seasons, weather conditions, time periods, and lighting conditions. Epipolar correction and disparity maps can be performed on each image to construct a diverse dataset of road obstacles. Data augmentation techniques (such as random rotation, translation, and scaling) can then be used to perform the same enhancement operations on both images to ensure geometric consistency.

[0067] The target detection model of this application takes the YOLOv8s target detection model as an example, which does not constitute a limitation to this application. The LabelImg annotation tool is used to annotate the dataset images with names and select locations through a visual operation interface. The annotation types include fire hydrants, flower beds, plastic bottles, pedestrians, or animals. The dataset is divided according to the ratio of training sets, validation sets, and test sets. Multiple rounds of training are performed to achieve the expected detection accuracy, and the bounding boxes and category probabilities of the detected objects are output. Once the YOLOv8s target detection model is trained to a convergent state, it can be used to detect static obstacles, dynamic obstacles, and safe driving areas in the target sidewalk image. Among them, the bounding box coordinates of the detected object are expressed as:

[0068] Output={(x min ,y min ,x max ,y max ,p class )},

[0069] Among them, x min Indicates the horizontal coordinate of the upper left corner of the target detection box, y min Indicates the upper left corner ordinate of the target detection box, x max Indicates the horizontal coordinate of the lower right corner of the target detection box, y max Indicates the lower right corner ordinate of the target detection box, p class Indicates the confidence of the target detection box;

[0070] Furthermore, the target detection frame corresponding to the obstacle is projected into the point cloud to calculate the three-dimensional position of the obstacle, which is expressed as:

[0071]

[0072] Among them, x i ,y i ,z i Represents the three-dimensional coordinates of each point in the point cloud within the target detection frame; N represents the number of points in the point cloud belonging to the obstacle; X obj ,Y obj ,Z obj Represents the three-dimensional centroid coordinates of the obstacle.

[0073] In a further embodiment, the cost functions corresponding to the static obstacle, the dynamic obstacle, and the safe driving area are expressed as:

[0074]

[0075] Where λ is the penalty coefficient for dynamic obstacles; if it is a static obstacle, such as a wall, fixed public facilities, etc., it will be directly marked as impassable, with an infinite cost, which is expressed as C sem (x,y)=∞;

[0076] If it is a dynamic obstacle, such as an animal or a pedestrian, the confidence level of the target detection frame is p class Dynamically adjust the cost, high confidence obstacles need to be avoided first, and different types of obstacles can be given different weights, which is expressed as C sem (x,y)=λ·P class ;

[0077] If it is a safe area, it is represented by C sem (x,y)=0.

[0078] The costs corresponding to the static obstacles, the dynamic obstacles, and the safe driving area are integrated into the first cost map to determine a second cost map.

[0079] Step S20: Obtain the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range. If it is detected that the energy accumulation value exceeds a preset energy accumulation threshold, the target passenger in the automatic following obstacle-avoiding wheelchair is determined to be in an out-of-seat state, call a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger, and use a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait.

[0080] Obtain a target sidewalk image containing obstacles from a binocular vision module of an automatic following obstacle-avoiding wheelchair, perform obstacle detection on the target sidewalk image using a target detection model that has been trained to a convergent state to determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, integrate the costs corresponding to the static obstacles, the dynamic obstacles, and the safe driving area into a first cost map to determine a second cost map, obtain the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range, and if it is detected that the energy accumulation value exceeds a preset energy accumulation threshold, determine that the target passenger in the automatic following obstacle-avoiding wheelchair is in an out-of-seat state, call a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger, and use a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait;

[0081] In some embodiments, the binocular vision module 8 in the automatic following obstacle avoidance wheelchair is controlled in coordination with the array pressure-sensitive sensor 13. The array pressure-sensitive sensor 13 detects in real time whether the target passenger is in a seated state and transmits the detection value to the data output module 15 for seat-leaving judgment. The calculated energy accumulation value S(t) is compared with the set energy accumulation threshold ε p For comparison, if S(t)>ε p , indicating that the energy accumulation value within the preset time range exceeds the preset energy accumulation threshold, wherein the preset time range can be 5 seconds. When the energy accumulation value within 5 seconds exceeds the preset energy accumulation threshold, the seat-leaving state is triggered, and it is considered that the target passenger has left the seat. The binocular camera 9 is started to perform three-dimensional reconstruction and obstacle avoidance, and target tracking and posture capture of the target passenger are performed; if S(t)≤ε p , it is considered that the leaving seat state has not been triggered and the target passenger is still in the seat, so the vehicle switches to the automatic obstacle avoidance mode with environment-adaptive speed control.

[0082] In a specific embodiment, the array pressure-sensitive sensor seat-leave determination model is expressed as:

[0083]

[0084] Where p(t)(p(τ)) is the two-dimensional pressure matrix, the total pressure value at time t obtained from the array pressure-sensitive sensor, and the scalar value obtained by summing the elements of the two-dimensional matrix, which reflects the pressure value of each point on the sensor surface; p threshold =25kg, which represents the threshold for seat leaving judgment, ε p =10 3 kg 2 ·s, which represents the energy accumulation threshold.

[0085] p(t)(p(τ)) is expressed as:

[0086]

[0087] Among them, P ij (t) represents the element in the two-dimensional pressure matrix of the array pressure sensor at time t, i and j are the row and column indices of the matrix, P ij (t) specifically represents the pressure value at the element location.

[0088] In some embodiments, the step of obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine the energy accumulation value within a preset time range includes:

[0089] Step S201: obtaining a two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor, dividing each element position point in the two-dimensional pressure matrix into a core element position point and a non-core element position point, and obtaining pressure values corresponding to the core element position point and the non-core element position point in the two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor;

[0090] Step S202: determining a first weight coefficient of the core element position point and a second weight coefficient of the non-core element position point in the two-dimensional pressure matrix, calculating and determining a first product between the first weight coefficient and the pressure value of the core element position point, and calculating and determining a second product between the second weight coefficient and the pressure value of the non-core element position point;

[0091] Step S203: sum the first products corresponding to each core element position point to determine a first sum value, sum the second products corresponding to each non-core element position point to determine a second sum value, and determine the total pressure value at each moment in the array pressure-sensitive sensor based on a third sum value between the first sum value and the second sum value.

[0092] In some embodiments, the step of calling a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger includes:

[0093] When the target passenger is detected to be out of his seat, the StrongSORT algorithm is used to track the target passenger. Based on the state of the target passenger in the previous frame image, the Kalman filter is used to predict and correct the current target position. The ReID module is used to extract the appearance feature vector of the target passenger to calculate the cosine similarity with its historical trajectory to determine the current position of the target passenger.

[0094] Specifically, StrongSORT (Simple Online and Realtime Tracking with Strong Re-identification) is an efficient target tracking algorithm based on deep learning. It improves on the classic SORT (Simple Online and Realtime Tracking) algorithm by incorporating target appearance features (through Re-identification) to enhance target tracking accuracy and robustness. StrongSORT is capable of handling multi-target tracking and provides real-time, accurate target tracking results in dynamic environments.

[0095] The Kalman filter predicts the likely location of the target in the current frame based on the target's position and state (such as velocity and acceleration) in the previous frame. This prediction is based on physical model assumptions, such as uniform velocity or acceleration. The Re-identification (Re-ID) module extracts the appearance features of the target passenger. These feature vectors, typically extracted using deep learning models (such as convolutional neural networks (CNNs), represent the target's appearance characteristics and have strong discriminative capabilities, enabling identification of different individuals. Appearance feature vectors are generated based on the target's visual information in the video frame, such as color, shape, and texture. The appearance feature vectors of the current frame are compared with the appearance features of the target in previous frames, and their cosine similarity is calculated. Cosine similarity measures the angular similarity between two vectors. Values closer to 1 indicate greater similarity, while values closer to -1 indicate less similarity. In this way, the system can match the target in the current frame with targets in previous trajectories to confirm the current target's location. The current location of the target is ultimately determined by combining the position predicted by the Kalman filter with the appearance feature matching of the Re-ID module. The Kalman filter provides a prediction based on physical motion, while the ReID module corrects or confirms the prediction by matching appearance features.

[0096] In summary, the current position of the target passenger is determined by the dynamic prediction of the Kalman filter and the appearance feature similarity matching of the ReID module. This method combines the motion information and visual features of the target and can more accurately track and locate the target.

[0097] In a further embodiment, the step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes:

[0098] Step S2001: Calling the target detection model that has been trained to a converged state to perform target detection on a target sidewalk image containing a target passenger to determine key point positions of the target passenger, wherein the key point positions include hip key points, knee key points, and ankle key points, wherein the hip key points include a left hip key point and a right hip key point, the knee key points include a left knee key point and a right knee key point, and the ankle key points include a left ankle key point and a right ankle key point;

[0099] Step S2002: Obtain a first vector between the left hip key point and the left knee key point, a second vector between the left knee key point and the left ankle key point, a third vector between the right hip key point and the right knee key point, a fourth vector between the right knee key point and the right ankle key point, a first distance between the left hip key point and the left knee key point, a second distance between the left knee key point and the left ankle key point, a third distance between the right hip key point and the right knee key point, and a fourth distance between the right knee key point and the right ankle key point;

[0100] Step S2003: Calculate and determine a third product between the first vector and the second vector, calculate and determine a fourth product between the first distance and the second distance, and use an arccosine function value of a first ratio between the third product and the fourth product as the hip, knee, and ankle joint angle of the left leg;

[0101] Step S2004: Calculate and determine the fifth product between the third vector and the fourth vector, calculate and determine the sixth product between the third distance and the fourth distance, and use the inverse cosine function value of the second ratio between the fifth product and the sixth product as the hip, knee, and ankle joint angle of the right leg.

[0102] In a further embodiment, the step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes:

[0103] Step S2001: Obtaining the target passenger's initial standing height, current standing height, hip-knee-ankle joint angles of the left leg, and hip-knee-ankle joint angles of the right leg, wherein the standing height represents the height from the top of the target passenger's head to the soles of their feet;

[0104] Step S2002: If it is detected that the hip-knee-ankle joint angle of the left leg and the hip-knee-ankle joint angle of the right leg are both less than a preset angle threshold, and a third ratio between the current standing height and the initial standing height is less than a preset ratio threshold, the target passenger is determined to be in a sitting state, and the automatic following obstacle-avoiding wheelchair is driven to stop and wait at a position parallel to the direction of the target passenger according to a preset following distance.

[0105] Specifically, based on the above steps, the three-dimensional coordinates (x target ,y target ,z target ), and define three key point positions at the same time, wherein the key point positions include the hip key point, the knee key point and the ankle key point, wherein the hip key point H includes the left hip key point H L And the right hip key point H R The knee key point K includes the left knee key point K L And the right knee key point K R , the ankle key point A includes the left ankle key point A L and right ankle key point A R ;

[0106] Get the left hip key point H L , right hip key point H R , left knee key point K L , right knee key point K R , left ankle key point A L and right ankle key point A R Then, the hip-knee-ankle joint angle θ of the left leg can be calculated based on the first vector between the left hip key point and the left knee key point, the second vector between the left knee key point and the left ankle key point, the second distance between the left knee key point and the left ankle key point, the third distance between the right hip key point and the right knee key point, and the fourth distance between the right knee key point and the right ankle key point. L , which is expressed as:

[0107]

[0108] Among them, θ L H represents the hip, knee and ankle joint angles of the left leg. L Indicates the left hip key point, K L Indicates the left knee key point, A L Represents the left ankle key point.

[0109] Similarly, the hip-knee-ankle joint angles of the right leg are similar to those of the left leg, and are not described in detail here. sitIt can be 110°, and it is detected that the hip-knee-ankle joint angle of the left leg and the hip-knee-ankle joint angle of the right leg are both less than the preset angle threshold, and the current standing height H current With the initial standing height H stand The third ratio between If the target passenger is less than a preset ratio threshold η, wherein the preset ratio threshold η can be 0.7, the target passenger is determined to be in a sitting state, and the automatic following obstacle avoidance wheelchair is driven to follow the preset following distance D. set Stop and wait in the direction parallel to the target passenger, where if θ L <θ sit ,θ R <θ sit And the current standing height H current With the initial standing height H stand When the third ratio between the target passenger and the target passenger is less than η, it is determined to be sitting down. When the target passenger is in the sitting state, the following distance D is set. set =D min = 40cm and finally stop in a direction parallel to the target passenger direction, otherwise, D set =D default =1m.

[0110] Step S30: If it is detected that the energy accumulation value does not exceed a preset energy accumulation threshold, the target passenger is determined to be in a seated state, the Shi-Tomasi algorithm is used to extract image key points in the target sidewalk image, and the optical flow method is used to calculate the motion vector of each image key point between two consecutive frames of the target sidewalk image to determine the dynamic point ratio;

[0111] Obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range, and if the energy accumulation value does not exceed a preset energy accumulation threshold, determining the target occupant as seated, extracting image key points from the target sidewalk image using the Shi-Tomasi algorithm, and calculating the motion vector of each image key point between two consecutive frames of the target sidewalk image using the optical flow method to determine the dynamic point ratio;

[0112] Specifically, the Shi-Tomasi algorithm is a classic computer vision algorithm used to detect corners (also known as keypoints) in images. Corner points are typically locations with significant structural features in an image, such as object corners and edge intersections. In this step, the Shi-Tomasi algorithm extracts keypoints from the target sidewalk image. These keypoints become the basis for the subsequent optical flow method to calculate motion vectors.

[0113] Optical flow is a technique used to estimate the motion of objects in images. It is often used to calculate the motion vector of pixels between two consecutive image frames. The motion vector represents the displacement of each keypoint between the two frames. Specifically, the optical flow method calculates the amount of motion (i.e., velocity) of each keypoint in the target sidewalk image from the first frame to the second frame. This information can be used to estimate whether these keypoints are moving and their speed.

[0114] After calculating the motion vector of each key point, a threshold can be determined based on the size of the motion vector (usually the speed). If the motion vector of a key point exceeds this threshold, the point can be considered "dynamic". The proportion of dynamic points refers to which points have larger motion vectors among all the extracted key points, representing the movement of these points between the two frames of the image. The higher the proportion of dynamic points, the more active the movement in the scene, and vice versa, it may indicate that the scene is relatively static.

[0115] Step S40: Determine the linear speed constraint of the speed type of the automatic following obstacle-avoiding wheelchair in each environment according to the proportion of dynamic points, and use the improved cost function corresponding to the DWA algorithm to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following obstacle-avoiding wheelchair.

[0116] If it is detected that the energy accumulation value does not exceed the preset energy accumulation threshold, the target passenger is determined to be in a seated state, the Shi-Tomasi algorithm is used to extract image key points in the target sidewalk image, and the optical flow method is used to calculate the motion vector of each image key point between two consecutive frames of target sidewalk images to determine the dynamic point ratio. Then, the linear speed constraint of the automatic following obstacle-avoiding wheelchair in the speed type of each environment is determined according to the dynamic point ratio, and the improved cost function corresponding to the DWA algorithm is used to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following obstacle-avoiding wheelchair, wherein the speed type of the environment includes a high-speed zone, a medium-speed zone and a low-speed zone.

[0117] Specifically, when the pressure-sensitive array sensor detects that the target person is seated, the wheelchair can detect environmental information through the binocular vision module and determine the speed of environmental adaptation.

[0118] During the automatic driving of the wheelchair, the binocular vision module collects the speed information of the reference objects on the road and measures the linear speed v of the wheelchair through the encoder. ego and angular velocity ω ego , the optical flow method is used for motion compensation to obtain the true speed information of the reference object, thereby constructing a dynamic environment classification model:

[0119] Improved potential function after compensation:

[0120] Among them, v i represents the speed of the i-th moving object, represents the object position, and ε represents the smoothing factor Φ comp (x,y) represents the potential function, which is used to describe the influence of other reference objects on the wheelchair movement in the dynamic environment after compensation by the optical flow method. N represents the total number of reference objects in the environment. Indicates the sum of the contribution values of all reference objects.

[0121] Obtained by velocity gradient in accordance with The travel environment is divided into high-speed (High), medium-speed (Mid), and low-speed (Low) areas. By linearly scaling the speed gradient and adaptively adjusting it in combination with the DWA algorithm, the linear speed constraint is improved, which can be expressed as:

[0122]

[0123] Add an angular velocity constraint, which is expressed as:

[0124]

[0125] Among them, ω max Expressed as angular velocity constraint, v max represents the linear velocity constraint, and L represents the turning radius of the wheelchair.

[0126] The improved cost function is expressed as:

[0127]

[0128] Among them, Cost represents the improved cost function, α represents the speed preference coefficient, β represents the obstacle avoidance sensitivity, and γ represents the environmental dynamic coefficient. represents the potential function Φ comp Gradient growth.

[0129] In some embodiments, the step of performing path planning based on the linear velocity constraint using an improved cost function corresponding to the DWA algorithm to determine an automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair on the second cost map includes:

[0130] Step S401, obtaining height cost, smoothing cost, distance cost and heading cost;

[0131] Step S402: constructing a trajectory evaluation function according to the height cost, the smoothing cost, the distance cost, and the heading cost;

[0132] Step S403: Using the DWA algorithm to determine the optimal automatic obstacle avoidance trajectory of the automatic following obstacle avoidance wheelchair in the second cost map according to the improved cost function and the trajectory evaluation function, so as to complete the control of the multi-mode automatic following obstacle avoidance wheelchair.

[0133] Specifically, the DWA (Dynamic Window Approach) algorithm is an algorithm commonly used in robot path planning and obstacle avoidance. It is based on the robot's dynamic model and selects the optimal trajectory for execution by evaluating a series of possible motion trajectories under the robot's motion constraints. The core idea of the DWA algorithm is to limit the robot's feasible speed and acceleration range through a "dynamic window", and then evaluate the cost of each feasible trajectory based on these restrictions, and finally select the optimal trajectory. Construct a trajectory evaluation function based on the height cost, the smoothness cost, the distance cost, and the heading cost;

[0134] The DWA algorithm is used to determine the optimal automatic obstacle avoidance trajectory of the automatic following obstacle avoidance wheelchair in the second cost map according to the improved cost function and the trajectory evaluation function, so as to complete the control of the multi-mode automatic following obstacle avoidance wheelchair.

[0135] The height cost is expressed as:

[0136]

[0137] Among them, h t Indicates the actual height of t trajectory points, h safe Indicates the preset safety height value.

[0138] This formula indicates that when the height safety threshold is exceeded, a height cost is generated, otherwise it is 0; The height risk for all trajectories is cumulative.

[0139] The smoothing cost is expressed as:

[0140]

[0141] Among them, ω k represents the angular velocity of the Kth trajectory point; ω k-1 represents the angular velocity of the k-1th trajectory point; Δt represents the time interval between adjacent trajectory points; represents angular acceleration; represents the sum of the squares of the angular acceleration along the entire trajectory.

[0142] The distance cost is the minimum distance from the trajectory to the nearest obstacle, which is expressed as:

[0143]

[0144] Among them, (x t ,y t ) represents the horizontal and vertical coordinates of the t-th track point of the wheelchair; (x obs ,y obs ) represents the horizontal and vertical coordinates of the t-th trajectory point of the obstacle; σ represents a very small positive number to avoid the denominator being 0;

[0145] The minimum straight-line distance between the wheelchair and the obstacle at the t-th trajectory point

[0146] The heading cost is the deviation between the trajectory endpoint and the target orientation, which is expressed as:

[0147] J heading =1-cos(θ t -θ goal ),

[0148] Among them, θ t represents the instantaneous heading angle of the wheelchair at the t-th trajectory point; θ goal Indicates the heading angle of the target point.

[0149] The trajectory evaluation function is expressed as:

[0150] J(v,ω)=α1J dist +α2J heading +α3J height +α4J smooth ;

[0151] Among them, α1, α2, α3, and α4 represent weight coefficients, which adjust the priority of each cost item.

[0152] In some embodiments, a Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module. The training data of the model includes speech data sets of Mandarin and English, with a total length of 1,000 hours. In the recognition process, the speech signal is first processed, the Mel-frequency cepstral coefficients (MFCC) are extracted, and the BeamSearch algorithm is used to decode and output text; users can control functions such as wheelchair parking by voice, and subsequent developers can also use the speech recognition module to develop more human-computer interaction instructions to provide humanized services. When the difference in obstacle detection results between the binocular vision module and the millimeter-wave radar module is greater than 15cm, emergency braking is triggered and the ultrasonic module is started for secondary verification. If the ultrasonic wave detects an obstacle at a distance less than 0.5m, it confirms the existence of the obstacle and maintains the braking state. If the ultrasonic wave does not detect an obstacle, it is determined to be a false alarm of the sensor and enters the manual intervention mode.

[0153] As can be seen from the above embodiments, compared with the prior art, this application addresses the problems in the prior art, such as the lack of a scenario-based intelligent speed regulation mechanism in existing systems and the inability to balance safety and efficiency in complex environments. This application includes but is not limited to the following beneficial effects:

[0154] First, traditional electric wheelchairs often rely on a single sensor (such as UWB or RFID), which has poor environmental adaptability and is difficult to cope with changes in obstacles in complex environments. In contrast, this application uses a combination of a binocular vision module and an array of pressure-sensitive sensors to obtain real-time visual image information and pressure sensing data from the environment, improving the wheelchair's environmental perception capabilities, enabling more accurate identification of static and dynamic obstacles, and ensuring that the wheelchair can autonomously avoid obstacles in complex environments.

[0155] Secondly, traditional wheelchair systems often fail to identify the user's dynamic state when tracking a wheelchair, such as whether the user is unseated or moving. This application, by introducing the StrongSORT algorithm, enables real-time tracking of the target passenger upon detecting their unseated state. This target tracking method accurately determines the target passenger's position, ensuring that the wheelchair can dock accurately when needed, avoiding misoperation or misalignment.

[0156] Third, by combining a wheelchair-following strategy with target tracking technology, this application can intelligently drive a wheelchair to automatically follow the occupant based on their real-time location. When the target person's accumulated energy exceeds a preset threshold, the system determines that the user has left their seat, triggering the wheelchair to automatically follow and dock. This mechanism enables autonomous following and docking, improving the user experience and is particularly user-friendly for users with limited mobility.

[0157] Fourth, by using the DWA algorithm and an improved cost function, combined with the Shi-Tomasi algorithm and optical flow, this application can calculate speed constraints based on the proportion of dynamic points in the target sidewalk image and accurately plan paths. This allows the wheelchair to dynamically adjust its path in complex environments based on real-time perception information, avoiding obstacles and automatically adjusting its speed based on environmental speed constraints, ensuring a balance between safety and efficiency.

[0158] Fifth, this application uses a continuous feedback mechanism to dynamically adjust the wheelchair's driving state based on actual conditions. When the user is in the "seated state," the system uses image key point extraction and motion vector calculation to determine the proportion of dynamic obstacles in the environment, and then determines the appropriate movement speed and trajectory planning. This not only improves the wheelchair's operating safety, but also ensures that it can smoothly adapt to changing environmental conditions during driving.

[0159] Sixth, traditional wheelchairs rely on manual control by the user, and there is a certain operational threshold. The intelligent features of this system (such as automatic obstacle avoidance, automatic following, intelligent docking, etc.) greatly reduce dependence on users, especially for the elderly and patients with limited mobility, reducing the difficulty of operation and improving the convenience and safety of use. Through a multi-modal perception and control mechanism, this application enables the automatic following obstacle avoidance wheelchair to autonomously understand and respond to different scenarios and user needs, which not only enhances the intelligence level of the wheelchair, but also provides a personalized user experience to adapt to the needs of different environments and scenarios.

[0160] Furthermore, this application effectively improves the wheelchair's environmental adaptability, target tracking ability, dynamic obstacle avoidance ability, path planning ability and intelligence level, thus having broad application prospects in an aging society and rehabilitation field, especially for patients with limited mobility, greatly improving the convenience, safety and comfort of using the wheelchair.

[0161] See also Figure 3A multi-mode automatic following and obstacle avoidance wheelchair control device is provided to meet one of the purposes of the present application, including an obstacle detection module 1100, a target tracking module 1200, a dynamic point ratio determination module 1300, and an obstacle avoidance trajectory determination module 1400. The obstacle detection module 1100 is configured to obtain a target sidewalk image containing obstacles from a binocular vision module of the automatic following and obstacle avoidance wheelchair, perform obstacle detection on the target sidewalk image using a target detection model that has been trained to a convergent state, and determine static obstacles, dynamic obstacles, and safe driving areas in the target sidewalk image. The costs corresponding to the static obstacles, dynamic obstacles, and safe driving areas are incorporated into a first cost map to determine a second cost map. The target tracking module 1200 is configured to obtain the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range. If the energy accumulation value exceeds a preset energy accumulation threshold, the target passenger in the automatic following and obstacle avoidance wheelchair is determined to be out of the seat, and a preset StrongSORT algorithm is invoked to track the target passenger to determine the target passenger. The current position of the seated person is determined by a preset wheelchair following strategy, and the automatic following obstacle-avoiding wheelchair is driven to stop at the target passenger to wait. The dynamic point ratio determination module 1300 is configured to determine the target passenger as being in a seated state when detecting that the energy accumulation value does not exceed a preset energy accumulation threshold, extract image key points in the target sidewalk image using the Shi-Tomasi algorithm, and calculate the motion vector of each image key point between two consecutive frames of the target sidewalk image using the optical flow method to determine the dynamic point ratio. The obstacle avoidance trajectory determination module 1400 is configured to determine the linear speed constraint of the speed type of the automatic following obstacle-avoiding wheelchair in each environment according to the dynamic point ratio, and perform path planning according to the linear speed constraint using an improved cost function corresponding to the DWA algorithm to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map, thereby completing the control of the multi-mode automatic following obstacle-avoiding wheelchair.

[0162] Based on any embodiment of this application, please refer to Figure 4 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 4As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence, and when the computer-readable instructions are executed by the processor, the processor may implement a multi-mode automatic following obstacle avoidance wheelchair control method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the multi-mode automatic following obstacle avoidance wheelchair control method of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0163] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the specific functions of each module in the multi-mode automatic following obstacle avoidance wheelchair control device. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules in the multi-mode automatic following obstacle avoidance wheelchair control device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0164] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-mode automatic following obstacle-avoiding wheelchair control method described in any embodiment of the present application.

[0165] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the multi-mode automatic following obstacle-avoiding wheelchair control method described in any embodiment of the present application.

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multi-mode automatic following obstacle avoidance wheelchair control method, characterized in that: include: Obtaining a target sidewalk image containing obstacles from a binocular vision module of the automatic following obstacle-avoiding wheelchair, performing obstacle detection on the target sidewalk image using a trained target detection model to determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, and integrating costs corresponding to the static obstacles, dynamic obstacles, and safe driving area into a first costmap to determine a second costmap; Obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range; if it is detected that the energy accumulation value exceeds a preset energy accumulation threshold, determining that the target passenger in the automatic following obstacle-avoiding wheelchair is in an out-of-seat state, calling a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger, and using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait; If it is detected that the energy accumulation value does not exceed a preset energy accumulation threshold, the target passenger is determined to be in a seated state, the Shi-Tomasi algorithm is used to extract image key points in the target sidewalk image, and the optical flow method is used to calculate the motion vector of each image key point between two consecutive frames of the target sidewalk image to determine the dynamic point ratio; The linear speed constraint of the speed type of the automatic following and obstacle-avoiding wheelchair in each environment is determined according to the proportion of dynamic points, and the improved cost function corresponding to the DWA algorithm is used to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following and obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following and obstacle-avoiding wheelchair.

2. The multi-mode automatic following obstacle avoidance wheelchair control method according to claim 1, characterized in that: The step of obtaining the total pressure value of the array pressure-sensitive sensor at each moment to determine the energy accumulation value within a preset time range includes: Obtaining a two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor, dividing each element position point in the two-dimensional pressure matrix into a core element position point and a non-core element position point, and obtaining pressure values corresponding to the core element position point and the non-core element position point in the two-dimensional pressure matrix at each moment in the array pressure-sensitive sensor; Determining a first weight coefficient of the core element position point and a second weight coefficient of the non-core element position point in the two-dimensional pressure matrix, calculating and determining a first product between the first weight coefficient and the pressure value of the core element position point, and calculating and determining a second product between the second weight coefficient and the pressure value of the non-core element position point; The first products corresponding to each core element position point are summed to determine a first sum value, the second products corresponding to each non-core element position point are summed to determine a second sum value, and the total pressure value at each moment in the array pressure-sensitive sensor is determined based on a third sum value between the first sum value and the second sum value.

3. The multi-mode automatic following obstacle avoidance wheelchair control method according to claim 1, characterized in that: The step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes: Calling the target detection model that has been trained to a converged state to perform target detection on a target sidewalk image containing a target passenger to determine key point positions of the target passenger, wherein the key point positions include hip key points, knee key points, and ankle key points, wherein the hip key points include a left hip key point and a right hip key point, the knee key points include a left knee key point and a right knee key point, and the ankle key points include a left ankle key point and a right ankle key point; Get the first vector between the left hip keypoint and the left knee keypoint, the second vector between the left knee keypoint and the left ankle keypoint, the third vector between the right hip keypoint and the right knee keypoint, the fourth vector between the right knee keypoint and the right ankle keypoint, the first distance between the left hip keypoint and the left knee keypoint, the second distance between the left knee keypoint and the left ankle keypoint, the third distance between the right hip keypoint and the right knee keypoint, and the fourth distance between the right knee keypoint and the right ankle keypoint; calculating and determining a third product between the first vector and the second vector, calculating and determining a fourth product between the first distance and the second distance, and using an arccosine function value of a first ratio between the third product and the fourth product as a hip, knee, and ankle joint angle of the left leg; Calculate and determine the fifth product between the third vector and the fourth vector, calculate and determine the sixth product between the third distance and the fourth distance, and use the inverse cosine function value of the second ratio between the fifth product and the sixth product as the hip, knee and ankle joint angle of the right leg.

4. The multi-mode automatic following obstacle avoidance wheelchair control method according to claim 3, characterized in that: The step of using a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to stop at the target passenger and wait includes: Obtaining the target passenger's initial standing height, current standing height, hip-knee-ankle joint angles of the left leg, and hip-knee-ankle joint angles of the right leg, wherein the standing height represents the height from the top of the target passenger's head to the soles of their feet; If it is detected that the hip-knee-ankle joint angle of the left leg and the hip-knee-ankle joint angle of the right leg are both less than a preset angle threshold, and a third ratio between the current standing height and the initial standing height is less than a preset ratio threshold, the target passenger is determined to be in a sitting state, and the automatic following obstacle-avoiding wheelchair is driven to stop and wait at a position parallel to the direction of the target passenger according to a preset following distance.

5. The multi-mode automatic following obstacle avoidance wheelchair control method according to claim 1, characterized in that: The step of calling a preset StrongSORT algorithm to track the target passenger to determine the current position of the target passenger includes: When the target passenger is detected to be out of his seat, the StrongSORT algorithm is used to track the target passenger. Based on the state of the target passenger in the previous frame image, the Kalman filter is used to predict and correct the current target position. The ReID module is used to extract the appearance feature vector of the target passenger to calculate the cosine similarity with its historical trajectory to determine the current position of the target passenger.

6. The multi-mode automatic following obstacle avoidance wheelchair control method according to claim 1, characterized in that: The step of performing path planning according to the linear velocity constraint using an improved cost function corresponding to the DWA algorithm to determine an automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair on the second cost map includes: Get the height cost, smoothing cost, distance cost and heading cost; Constructing a trajectory evaluation function according to the height cost, the smoothness cost, the distance cost, and the heading cost; The DWA algorithm is used to determine the optimal automatic obstacle avoidance trajectory of the automatic following obstacle avoidance wheelchair in the second cost map according to the improved cost function and the trajectory evaluation function, so as to complete the control of the multi-mode automatic following obstacle avoidance wheelchair.

7. The multi-mode automatic following obstacle avoidance wheelchair control method according to claims 1 to 6, characterized in that: The basic network architecture of the target detection model includes a YOLOv8s target detection model.

8. A multi-mode automatic following obstacle avoidance wheelchair control device, characterized in that: include: an obstacle detection module configured to obtain a target sidewalk image containing obstacles from a binocular vision module of the automatic following obstacle-avoiding wheelchair, perform obstacle detection on the target sidewalk image using a target detection model that has been trained to a converged state, to determine static obstacles, dynamic obstacles, and a safe driving area in the target sidewalk image, and incorporate costs corresponding to the static obstacles, the dynamic obstacles, and the safe driving area into a first costmap to determine a second costmap; a target tracking module configured to obtain the total pressure value of the array pressure-sensitive sensor at each moment to determine an energy accumulation value within a preset time range, and upon detecting that the energy accumulation value exceeds a preset energy accumulation threshold, determine that a target passenger in the automatic following obstacle-avoiding wheelchair is in an out-of-seat state, invoke a preset StrongSORT algorithm to perform target tracking on the target passenger to determine the current position of the target passenger, and employ a preset wheelchair following strategy to drive the automatic following obstacle-avoiding wheelchair to park at the target passenger to wait; a dynamic point ratio determination module configured to, upon detecting that the energy accumulation value does not exceed a preset energy accumulation threshold, determine the target passenger as being seated, extract image key points from the target sidewalk image using a Shi-Tomasi algorithm, and calculate a motion vector of each image key point between two consecutive frames of the target sidewalk image using an optical flow method to determine a dynamic point ratio; The obstacle avoidance trajectory determination module is configured to determine the linear speed constraint of the speed type of the automatic following obstacle-avoiding wheelchair in each environment according to the proportion of the dynamic points, and use the improved cost function corresponding to the DWA algorithm to perform path planning according to the linear speed constraint to determine the automatic obstacle avoidance trajectory of the automatic following obstacle-avoiding wheelchair in the second cost map, so as to complete the control of the multi-mode automatic following obstacle-avoiding wheelchair.

9. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images