Wheelchair real-time obstacle prediction and avoidance system and method based on multi-target tracking

Abstract

The invention belongs to the technical field of intelligent wheelchairs, particularly relates to a wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking, and aims to solve the problem that the performance and application of a wheelchair in a complex dynamic environment are limited in the prior art. The system comprises a data acquisition and processing module used for calculating the distance between an obstacle and a reference point; the obstacle detecting and tracking module is used for generating a suspected area of the obstacle based on the multi-dimensional features; screening out a determined area, then calculating the similarity of suspected dynamic obstacles in the determined area, screening out the dynamic obstacles based on the similarity, and screening out the determined obstacles based on credibility scores; the decision control module is used for determining an optional path based on the path evaluation function; and then a trajectory evaluation preferential function is used to score a trajectory corresponding to path planning, and the trajectory with the highest stability is selected. According to the invention, high-efficiency obstacle detection and avoidance can be realized on the premise of low calculation amount.

Description

Background Art

[0002] With the continuous development of society, the problem of population aging and the needs of people with disabilities have gradually become prominent, and the quality of life and safety of the elderly and people with disabilities have attracted widespread attention from the society. They face problems such as inconvenience in movement and reduced ability to take care of themselves. These problems not only affect their physical and mental health and quality of life, but also bring additional burdens to their families and society. With the rapid advancement of science and technology, professionals in the field of rehabilitation are committed to seeking innovative methods and technologies to promote the rehabilitation process of people with limited movement, such as people with disabilities and the elderly, and improve their quality of life. The development and application of assistive devices have become key research directions, among which intelligent driving wheelchairs have huge potential application prospects.

[0003] However, smart driving wheelchairs face multiple challenges in practical applications, such as complex environment perception and obstacle recognition. The current smart wheelchair obstacle avoidance system relies on the collaborative decision-making and calculation of multiple sensors. It usually uses related sensors including lidar, GPS positioning module, gyroscope, speedometer, etc. to obtain environmental information and posture information in real time, and processes sensor information through computers to obtain obstacle information and make obstacle avoidance decisions. However, more sensors not only increase the cost, but also increase the computational complexity and power consumption of the system, reducing the feasibility of mass application of the system. In addition, the existing lidar-based obstacle detection system cannot use the motion information of the obstacle to predict the motion trajectory of the dynamic obstacle, which further limits the performance and application potential of the wheelchair in complex dynamic environments. Summary of the invention

[0004] In order to solve the above-mentioned problem in the prior art, that is, the obstacle detection system based on laser radar cannot use the motion information of the obstacle to predict the motion trajectory of the dynamic obstacle, which further limits the performance and application potential of the wheelchair in a complex dynamic environment, the present invention provides a real-time obstacle prediction and avoidance system for a wheelchair based on multi-target tracking, the system comprising:

[0005] A data acquisition and processing module, used to capture images and collect scene data, and calculate the distance between the obstacle and the reference point based on the scene data;

[0006] The obstacle detection and tracking module is used to generate multi-dimensional features based on the captured image, generate a suspected area of ​​the obstacle based on the multi-dimensional features; filter out the determined area based on the prediction deviation of the suspected area, and then calculate the similarity of the dynamic obstacles in the determined area to filter out the dynamic obstacles, and then calculate the credibility score for each obstacle, and determine the obstacle based on the credibility score of the obstacle;

[0007] A decision control module is used to define a path evaluation function based on obstacles and distances between obstacles and reference points, and to determine an optional path based on the path evaluation function; to limit a reasonable speed range using two feasible speed sets; and to score the trajectory corresponding to each optional path using a trajectory evaluation optimization function under the limitation of the two feasible speed sets, and to select the trajectory with the highest smoothness and stability as the optimal trajectory for the wheelchair movement;

[0008] Interaction module, used to provide visual interface interaction and user operation;

[0009] The communication module is used for communication among various modules based on the communication mechanism.

[0010] In this embodiment, calculating the distance between the obstacle and the reference point based on the scene data includes:

[0011] in, is the distance between the obstacle and the target point, is the pixel distance function, is the pixel matching function, α′ and β′ are both dynamic compensation factors, is the first constraint function, which constrains the smooth transition of adjacent pixels. is the second constraint function, i represents the pixel index in the image, Δ i is the matching displacement of pixel i, Δ j is the matching displacement of pixel j, j is the neighboring pixel of pixel i, is the conditional function, It is the compensation item for environmental complexity.

[0012] In this embodiment, the prediction deviation of the suspected area includes:

[0013] Among them, k represents the number of samples, s i Indicates the position deviation between the actually detected obstacle area and the actual obstacle area. Represents the predicted obstacle coordinate information deviation value, P′ i is the weight coefficient for obstacle feature similarity adjustment, W(·) is the error weight function, E represents the predicted deviation value, s x is the horizontal coordinate deviation of the obstacle area center, s y is the vertical coordinate deviation of the obstacle area center, s w is the obstacle area width deviation, s h is the height deviation of the obstacle area, s x ρ is the horizontal coordinate deviation of the center of the obstacle area in the auxiliary view, sρ w It is the auxiliary view obstacle area width deviation.

[0014] In this embodiment, calculating the similarity of dynamic obstacles in the determined area includes:

[0015]

[0016] Among them, α, β, γ, and δ are weight coefficients adjusted over time, and C det With C pre are the center coordinates of the detected 3D target frame and the predicted 3D target frame, respectively. det With A pre are the acceleration vectors of the detected 3D target frame and the predicted 3D target frame, respectively, det With γ pre are the orientation angles of the detected 3D target frame and the predicted 3D target frame, V det With V pre They represent the volume of the detected 3D target box and the predicted 3D target box respectively, ∈(t) is the environment complexity parameter, T det With T pre are the trajectory vectors of the detected target and the predicted target, respectively, N is the number of trajectory points, τ(t) is the time decay factor, and D centre is the target center matching similarity, D angle is the target angle matching similarity, D volume is the target volume matching similarity, D traj Match the target trajectory similarity.

[0017] In this embodiment, calculating the credibility score of the obstacle includes:

[0018] Among them, S det To consider the similarity of obstacle shapes, P(c) is the probability of obstacle detection, S motion Measures the stability of the target's motion state; S is the credibility score of the obstacle, They are all weight coefficients for dynamically adjusting the credibility of different factors.

[0019] In this embodiment, the path evaluation function is:

[0020]

[0021] in, is the path evaluation function, is the cumulative score from the starting point to the current node, and Inversely proportional to the speed of the vehicle, ensuring that the path planning actively avoids close obstacles. is the target distance estimate from the current node to the target node, is the path curvature evaluation function, (x cur ,y cur ) is the current node coordinate, (x goal ,y goal ) is the target node coordinate, μ(t) is the environment adaptive adjustment coefficient, δ is the balance bending weight adjustment coefficient, k is the environment complexity adjustment parameter, and ε(t) is the obstacle density and node distribution complexity function.

[0022] In this embodiment, the two feasible speed sets are:

[0023] Among them, V m is the driving speed limit set, v min and v max Represent the minimum and maximum linear speeds, ω min and ω max represents the minimum angular velocity and the maximum angular velocity, v is the linear velocity, ω is the angular velocity, and τ is the safety time threshold; V a is the safe speed limit set, dis(v,ω) is the shortest distance between the trajectory corresponding to the moving wheelchair speed and the obstacle, are the maximum linear acceleration and the maximum angular acceleration.

[0024] In this embodiment, under the limitation of two feasible speed sets, the trajectory evaluation optimization function is used to score the trajectory corresponding to each speed path planning, including:

[0025] Among them, σ is a smoothing function, α, β, and γ are all adaptive adjustment weights of each score, Align(v,ω) is a function that measures the consistency between the wheelchair's driving direction and the target direction; Gap(v,ω) represents the minimum safe gap between the current trajectory and the nearest obstacle, based on the minimum obstacle distance The angle θ with the velocity direction i , avoiding trajectories approaching obstacles at high speed, Stability(v,ω) reflects the stability of the trajectory, is the optimal function for trajectory evaluation.

[0026] In this embodiment, the decision control module also includes real-time obstacle avoidance and navigation control of the wheelchair in real-time operation. The formula for obstacle avoidance and navigation control is:

[0027] Among them, e(t) represents the deviation between the current position and the expected path, is the deviation change rate, K p With K dare the adjustment coefficient and differential adjustment coefficient respectively, ΔK is the dynamic adjustment of control strength, K f is the feedforward adjustment coefficient, p(t) is the feedforward prediction factor, R(t) is the real-time obstacle information, and u(t) is the control input, which is used to measure the control signal of the wheelchair in real-time obstacle avoidance and navigation control.

[0028] In a second aspect of the present invention, a real-time obstacle prediction and avoidance method for a wheelchair based on multi-target tracking is proposed, the method comprising:

[0029] Calculate the distance between the obstacle and the reference point based on the scene data;

[0030] Generate a suspected area of ​​an obstacle based on multi-dimensional features; select a determined area based on the prediction deviation of the suspected area, and then calculate the similarity of dynamic obstacles in the determined area to select dynamic obstacles, and then calculate the credibility score for each obstacle, and determine the obstacle based on the credibility score of the obstacle;

[0031] A path evaluation function is defined based on the determined obstacles and the distance between the obstacles and the reference point, and an optional path is determined based on the path evaluation function; two feasible speed sets are used to limit a reasonable speed range; under the limitation of the two feasible speed sets, the trajectory corresponding to each optional path is scored using a trajectory evaluation optimization function, and the trajectory with the highest smoothness and stability is selected as the optimal trajectory for wheelchair movement;

[0032] Real-time obstacle avoidance and navigation control of wheelchairs in real-time operation.

[0033] Beneficial effects of the present invention:

[0034] (1) The present invention constructs a real-time obstacle monitoring and avoidance system for an intelligent wheelchair. Through reasonable algorithm design, the system can achieve efficient obstacle detection and avoidance with low computational complexity, can be deployed on a microcomputer, and has high practicality and economy.

[0035] (2) The present invention solves the problems of high R&D cost, high computational power consumption and insufficient dynamic obstacle prediction of intelligent driving wheelchairs. The present invention enriches the obstacle avoidance function of the intelligent driving wheelchair by using only a visual perception device and a motion state detection device, and ensures the safety of the intelligent driving wheelchair user with high real-time, accuracy and foresight.

[0036] (3) The present invention can accurately locate the position of obstacles and reduce the false alarm rate by generating multi-dimensional features and screening suspected areas. By using similarity calculation and credibility scoring mechanism, dynamic obstacles can be effectively separated from static backgrounds, thereby improving the adaptability and robustness of the system. By scoring the credibility of each obstacle, the existence of the obstacle and its threat level can be further confirmed, so as to formulate a safer movement strategy. Based on the determined obstacles and the distance between the obstacles and the target point, the path evaluation function is defined, and a variety of feasible paths can be automatically generated, increasing the flexibility and diversity of path selection. The speed of the path is adjusted using two feasible speed sets, and multiple speed path planning schemes are generated to ensure that the wheelchair can find the optimal speed combination under different circumstances. The trajectory corresponding to each speed path planning is scored by the trajectory evaluation optimization function, and finally the trajectory with the highest smoothness and stability is selected as the optimal trajectory for the wheelchair movement, thereby improving the comfort and safety of the riding experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 is a schematic diagram of a wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to an embodiment of the present invention;

[0039] Figure 2 is a detailed schematic diagram of a wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of an obstacle detection and tracking module according to an embodiment of the present invention;

[0041] Figure 4 It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] The present invention provides a wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking, the system comprising:

[0045] A data acquisition and processing module, used to capture images and collect scene data, and calculate the distance between the obstacle and the reference point based on the scene data;

[0046] The obstacle detection and tracking module is used to generate multi-dimensional features based on the captured image, generate suspected areas of obstacles based on the multi-dimensional features; filter out the determined areas based on the prediction deviation of the suspected areas, then calculate the similarity of the suspected dynamic obstacles in the determined areas, and filter out the dynamic obstacles based on the similarity, then calculate the credibility score for each obstacle, and filter out the determined obstacles based on the credibility score;

[0047] A decision control module is used to define a path evaluation function based on obstacles and distances between obstacles and reference points, and to determine an optional path based on the path evaluation function; to limit a reasonable speed range using two feasible speed sets; and to score the trajectory corresponding to each optional path using a trajectory evaluation optimization function under the limitation of the two feasible speed sets, and to select the trajectory with the highest smoothness and stability as the optimal trajectory for the wheelchair movement;

[0048] Interaction module, used to provide visual interface interaction and user operation;

[0049] The communication module is used for communication among various modules based on the communication mechanism.

[0050] In order to more clearly explain the wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking of the present invention, the following is combined with Figure 1 Each module in the embodiment of the present invention is described in detail.

[0051] The first embodiment of the present invention is a wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking, and each module is described in detail as follows:

[0052] The present invention mainly includes an intelligent decision-making host computer and an intelligent control lower computer for control. The intelligent decision-making host computer is responsible for receiving environmental perception information and motion perception information, and predicting the position and motion state of obstacles in real time through multi-target tracking technology. Based on this information, the host computer makes intelligent decisions, including path planning and the generation of obstacle avoidance strategies. Subsequently, the decision information is transmitted to the intelligent control lower computer through the communication module. After receiving the instructions from the host computer, the intelligent control lower computer controls the wheelchair motor to achieve obstacle avoidance operations.

[0053] The system includes modules such as Figure 2As shown in the figure, it mainly includes S01 data acquisition and processing module, S02 obstacle detection and tracking module, S03 decision control module, S04 interaction module and S05 communication module. Among them, S02 obstacle detection and tracking module is implemented based on multi-target tracking algorithm, and S03 decision control module global environment modeling and real-time situation awareness collaborative mechanism builds a two-level navigation architecture to respond to changes in the environment around the smart wheelchair in real time.

[0054] The specific implementation plan and process of this system are introduced in detail below.

[0055] A data acquisition and processing module, used to capture images and collect scene data, and calculate the distance between the obstacle and the reference point based on the scene data;

[0056] The data acquisition and processing module includes an environmental information capture stage and a multi-source data optimization stage. Through the collaborative work of a dual-channel visual perception device and a motion state detection device, high-precision spatial modeling and equipment status monitoring are achieved. The two perception devices complement each other in data type and time and space dimensions to build a closed-loop system for three-dimensional environmental reconstruction and motion trajectory analysis. The dual-channel vision system captures three-dimensional environmental information. In the image quality enhancement stage, the present invention first uses a multi-point calibration scheme to optimize the environmental parameter compensation of the dual-channel vision device, and then corrects the distortion of the real image. An adaptive adjustment factor is introduced when calculating depth information to improve adaptability to complex scenes.

[0057] In this embodiment, calculating the distance between the obstacle and the reference point based on the scene data includes:

[0058] in, is the distance between the obstacle and the target point, is the pixel distance function, is the pixel matching function, α′ and β′ are both dynamic compensation factors, is the first constraint function, which constrains the smooth transition of adjacent pixels. is the second constraint function, which constrains the case of large matching changes to avoid mutations, i represents the pixel index in the image, Δ i is the matching displacement of pixel i, Δ j is the matching displacement of pixel j, j is the neighboring pixel of pixel i, It is the compensation item for environmental complexity.

[0059] Then, the depth information of the scene is calculated based on the pixel distance function and the configuration parameters of the dual-channel visual perception device. The depth information can reflect the actual distance from the object in the scene to the imaging plane of the dual-channel visual perception device, thereby detecting the distance, position and outline of the obstacle. In addition, the size and center position of the obstacle can also be calculated.

[0060] In the information preprocessing stage, the present invention uses dynamic data purification to process the motion state detection device signal, introduces an error compensation mechanism, and improves the stability of long-term measurement. The specific method is: Θ(t i )=Θ(t i-1 )+∫Ω(τ)dτ·Ψ′(Δt); V(t i )=V(t i-1 )+∫Α(τ)dτ·Λ′(ξ);

[0061] Among them, t i represents the time step, Θ(t i ) is the time t i Downward attitude angle, Θ(t i-1 ) is the time t i-1 The attitude angle under the condition, Ω(τ) is the angular velocity, V(t i ) is t i The speed under the i-1 ) is t i-1 The speed under the condition of t, Α(τ) is i The acceleration under i-1 ) is t i-1 The speed under the i ) is the time t i The displacement under i-1 ) is the time t i-1 The displacement under the condition of Ψτ′(Δt) is the time delay attenuation correction term, Λ′(ξ) is the attitude adjustment correction term, It is the progress impediment modifier.

[0062] In the data fusion part, a dual-loop architecture of state evolution prediction and real-time correction is adopted. The movement trend in a short period of time is calculated through state evolution prediction, and the data weight is dynamically adjusted in combination with the environmental information correction mechanism to improve the real-time tracking capability:

[0063] in, is the predicted state vector at the current time k, Yes, A describes the evolution of the system state over time, and B represents how the control instructions affect the system state. is the estimated state at the previous moment k-1, u k is the control input at the current time k, Υ′(∈) is the dynamic weight adjustment, Represents the dynamic weight of data fusion, U pred is the state uncertainty assessment factor, is the environmental noise assessment, is the data quality factor, z k is the current sensor measurement value.

[0064] Obstacle detection and tracking module, such as Figure 3 As shown, the obstacle detection and tracking module is divided into a detection algorithm module and a tracking algorithm module. In the detection algorithm module, the present invention uses a multi-level feature parsing architecture to extract rich features and construct an adaptive space perception network. The image input captured by the dual-channel visual perception device is processed by the basic feature extraction architecture to generate a multi-dimensional feature combination. The obstacle detection and tracking module is used to generate multi-dimensional features based on the captured image, and generate suspected areas of obstacles based on the multi-dimensional features; predict the deviation of the suspected area to filter out the determined area, and then calculate the similarity of the dynamic obstacles in the determined area to filter out the dynamic obstacles, and then calculate the credibility score for each obstacle, and determine the obstacle based on the credibility score of the obstacle;

[0065] In this embodiment, the prediction deviation of the suspected area includes:

[0066] Among them, k represents the number of samples, s i Indicates the position deviation between the actually detected obstacle area and the actual obstacle area. Represents the predicted obstacle coordinate information deviation value, P′ i is the weight coefficient for obstacle feature similarity adjustment, W(·) is the error weight function, E represents the predicted deviation value, s x is the horizontal coordinate deviation of the obstacle area center, s y is the vertical coordinate deviation of the obstacle area center, s w is the obstacle area width deviation, s h is the height deviation of the obstacle area, s x ′ is the horizontal coordinate deviation of the center of the obstacle area in the auxiliary view, s′ w is the width deviation of the obstacle area in the auxiliary view. The dual-channel visual perception device will obtain two views, one as the main view and the other as the auxiliary view.

[0067] The tracking algorithm module of the present invention can generate the motion trajectory of the obstacle according to the detection results of the continuous frames, and realize the continuous tracking of the obstacle. The specific process is as follows: Figure 3 In the detection algorithm module, the state description set of each obstacle is extracted in the form of: [x,y,z,γ,w,h,l,v x ,v y ,v z ,v θ ,a x ,ay ], where x, y, z, γ are the location parameters of the obstacle, w, h, l are the size parameters, and v x ,v y ,v z ,v θ is the speed of movement and rotation of the obstacle in each direction, a x ,a y Represents acceleration information. Based on the above state vector, the system uses a motion trajectory-based adaptive adjustment method to predict the future motion state of the obstacle, and integrates the prediction result with the detection result to achieve real-time update of the target position.

[0068] Since there may be multiple different obstacles, it is necessary to distinguish and associate different targets when implementing multi-target tracking. In this embodiment, calculating the similarity of dynamic obstacles in a determined area includes:

[0069] Among them, α, β, γ, and δ are weight coefficients adjusted over time, and C det With C pre are the center coordinates of the detected 3D target frame and the predicted 3D target frame, respectively. det With A pre are the acceleration vectors of the detected 3D target frame and the predicted 3D target frame, respectively, det With γ pre are the orientation angles of the detected 3D target frame and the predicted 3D target frame, V det With V pre They represent the volume of the detected 3D target box and the predicted 3D target box respectively, ∈(t) is the environment complexity parameter, T det With T pre are the trajectory vectors of the detected target and the predicted target, respectively, N is the number of trajectory points, τ(t) is the time decay factor, and D centre is the target center matching similarity, D angle is the target angle matching similarity, D volume is the target volume matching similarity, D traj Match the target trajectory similarity.

[0070] In this embodiment, calculating the credibility score of the obstacle includes:

[0071] Among them, S det To consider the similarity of obstacle shapes, P(c) is the probability of obstacle detection, S motion Measures the stability of the target's motion state; S is the credibility score of the obstacle, They are all weight coefficients for dynamically adjusting the credibility of different factors.

[0072] In this embodiment, the suspected area whose prediction deviation of the suspected area is less than the first threshold is the determined area, and the suspected dynamic obstacle whose similarity of the suspected dynamic obstacle in the determined area is greater than the second threshold is the dynamic obstacle.

[0073] In this embodiment, after a potential obstacle obtains a high credibility score, its continuous moment scores are confirmed. If multiple consecutive frames meet the criteria, trajectory stability verification is performed. Finally, the system marks it as a confirmed obstacle and starts to display and update the trajectory. Finally, based on the target matching results, the system fuses the obstacle detection and obstacle tracking results of each frame to obtain the continuous motion trajectory of the obstacle, and uses this trajectory information to predict the movement trend of the obstacle in the short term in the future.

[0074] A decision control module is used to define a path evaluation function based on obstacles and distances between obstacles and reference points, and to determine an optional path based on the path evaluation function; to limit a reasonable speed range using two feasible speed sets; and to score the trajectory corresponding to each optional path using a trajectory evaluation optimization function under the limitation of the two feasible speed sets, and to select the trajectory with the highest smoothness and stability as the optimal trajectory for the wheelchair movement;

[0075] The decision-making and control module is divided into intelligent decision-making and intelligent control parts. The intelligent decision-making part builds a two-level navigation architecture based on the collaborative mechanism of global environment modeling and real-time situational awareness to generate obstacle avoidance decisions. Global environment modeling is performed before the smart wheelchair starts to move. A complete path from the current position to the target position is planned based on the current environment, and the global map of the environment is considered. Real-time situational awareness is to plan short-distance paths and handle dynamic obstacles during the actual movement process based on the environmental information perceived by the sensor in real time.

[0076] In this embodiment, the path evaluation function is:

[0077] in, is the path evaluation function, is the cumulative score from the starting point to the current node, and Inversely proportional to the speed of the vehicle, ensuring that the path planning actively avoids close obstacles. is the target distance estimate from the current node to the target node, is the path curvature evaluation function, (x cur ,y cur ) is the current node coordinate, (x goal ,y goal) is the target node coordinate, μ(t) is the environment adaptive adjustment coefficient, δ is the balance bending weight adjustment coefficient, κ is the environment complexity adjustment parameter, and ε(t) is the obstacle density and node distribution complexity function.

[0078] During the actual driving process, the system collects environmental information in real time through sensors. Real-time situational awareness adopts a dynamic constrained speed optimization model. By introducing a dynamic feasibility factor, the speed sampling of the smart wheelchair is constrained and optimized, and the trajectory of the data sample is simulated and evaluated. Under the premise of ensuring safety, the optimal speed command is obtained to meet the dynamic obstacle avoidance requirements. The smart wheelchair can move in the target direction while avoiding obstacles. In order to enable a moving smart wheelchair to safely avoid obstacles on the premise of identifying obstacles, it is necessary to consider whether it can stop within a safe range at the current speed after discovering the obstacle to avoid a collision. To ensure that it can stop in time when an obstacle is detected, the following two feasible speed sets are defined. In this embodiment, the two feasible speed sets are:

[0079] V m is the driving speed limit set, v min and v max Represent the minimum and maximum linear speeds, ω min and ω max represents the minimum angular velocity and the maximum angular velocity, v is the linear velocity, ω is the angular velocity, and τ is the safety time threshold; V a is the safe speed limit set, dis(v,ω) is the shortest distance between the trajectory corresponding to the moving wheelchair speed and the obstacle, and dis(v,ω) is less than the distance from the obstacle to the reference point calculated based on the scene data. are the maximum linear acceleration and the maximum angular acceleration.

[0080] In order to ensure that the smart wheelchair can stop in time when dealing with obstacles, the speed constraints must be met:

[0081]

[0082] T safe (v,w) is the safe time to reach the obstacle at the current speed;

[0083] In this embodiment, the trajectory corresponding to each speed path planning is scored using a trajectory evaluation optimization function within the speed range limited by the two feasible speed sets, including:

[0084] Among them, σ is a smoothing function, α, β, and γ are all adaptive adjustment weights of each score, Align(v,ω) is a function that measures the consistency between the wheelchair's driving direction and the target direction; Gap(v,ω) represents the minimum safe gap between the current trajectory and the nearest obstacle, based on the minimum obstacle distance The angle θ with the velocity direction i , avoiding trajectories approaching obstacles at high speed, Stability(v,ω) reflects the stability of the trajectory, is the optimal function for trajectory evaluation.

[0085] This algorithm can effectively evaluate the trajectories generated by the smart wheelchair executing different combinations of linear speeds and angular speeds. The system finally selects the trajectory with the highest score from all predicted trajectories as the local optimal motion plan, realizing dynamic obstacle avoidance of the smart wheelchair and ensuring that the smart wheelchair reaches the target position safely and quickly.

[0086] The obstacle avoidance decision generation stage first uses global environment modeling to generate the optimal path from the starting point to the target point on the map. Subsequently, the path of the global environment modeling is decomposed into a series of local path segments with real-time situational awareness. The node with the lowest score on each local path segment is selected as the local target point, and real-time motion planning is performed using a dynamic constraint speed optimization model. Real-time motion planning updates obstacle information in real time based on visual sensor data and dynamically updates the global environment modeling. Under the guidance of real-time motion planning, the smart wheelchair tracks the latest path of the global environment modeling. Combined with the real-time updated visual information, the above-mentioned local path planning method is used to continuously adjust the driving trajectory until it finally reaches the target point. The intelligent control module receives the speed command information from the intelligent decision-making module, and realizes real-time obstacle avoidance and navigation control of the smart wheelchair by controlling the motor of the smart wheelchair.

[0087] In this embodiment, the decision control module also includes real-time obstacle avoidance and navigation control of the wheelchair in real-time operation. The formula for obstacle avoidance and navigation control is:

[0088] Among them, e(t) represents the deviation between the current position and the expected path, is the deviation change rate, K p With K d are the adjustment coefficient and differential adjustment coefficient respectively, ΔK is the dynamic adjustment of control strength, K f is the feedforward adjustment coefficient, p(t) is the feedforward prediction factor, which is used to amplify the influence of the prediction factor and improve the system's response speed to emergencies. This control strategy ensures that when encountering dynamic obstacles, the movement state can be quickly adjusted to maintain safe and smooth driving. R(t) is the real-time obstacle information, and u(t) is the control input, which is used to measure the control signal of the wheelchair in real-time obstacle avoidance and navigation control.

[0089] The interactive module is used to provide visual interface interaction and user operation; the interactive module includes the visual interface interaction and user operation control module, which aims to provide users with a friendly operation interface and convenient control methods to ensure the ease of use and efficiency of the system. The visual interface interaction module is responsible for the real-time display of the system status and the display of user interaction information, including system status display, obstacle detection and warning, and path planning. The system status display provides real-time status information of the smart wheelchair, such as the surrounding environment information, current position, target position, current wheelchair driving status, etc. Users can understand the real-time working status of the wheelchair through this interface. The obstacle detection and warning dynamically displays the obstacle position and movement trajectory, and predicts the obstacle's future short-term movement direction. With this information, users can timely understand the obstacles around the wheelchair to ensure safe obstacle avoidance. Path planning will display and update the safe driving path from the current wheelchair position to the target location in real time. Users can intuitively view the recommended driving path and monitor the real-time changes of the path to ensure that the wheelchair moves along the optimal route. The user interaction part allows users to set the target position, adjust system parameters (such as speed), start / stop system functions, and other operations through the graphical interface. This part provides a convenient operation method that allows users to easily control the system and adjust the settings to meet specific needs.

[0090] The communication module is used to communicate between modules based on the communication mechanism. The communication module is designed and implemented based on the communication mechanism in the ROS2 system. The time synchronization mechanism of ROS2 ensures the consistency and timeliness of data between different modules, and can ensure the efficiency and reliability of data transmission and information exchange between modules of the system. Through the publish / subscribe mechanism of ROS2, each functional module can exchange data through the defined message type, so as to support the seamless integration and efficient collaboration between the modules within the intelligent wheelchair system, thereby realizing the overall performance optimization and function enhancement of the system. Through this module, the intelligent wheelchair system can realize efficient, low-latency and reliable information transmission between modules, ensure the coordinated linkage of each functional unit, and thus improve the overall intelligence level.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0092] It should be noted that the wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not regarded as improper limitations of the present invention.

[0093] A second embodiment of the present invention provides a method for real-time wheelchair obstacle prediction and avoidance based on multi-target tracking, the method comprising:

[0094] Calculate the distance between the obstacle and the reference point based on the scene data;

[0095] Generate a suspected area of ​​an obstacle based on multi-dimensional features; select a determined area based on the prediction deviation of the suspected area, and then calculate the similarity of dynamic obstacles in the determined area to select dynamic obstacles, and then calculate the credibility score for each obstacle, and determine the obstacle based on the credibility score of the obstacle;

[0096] A path evaluation function is defined based on the determined obstacles and the distance between the obstacles and the reference point, and an optional path is determined based on the path evaluation function; two feasible speed sets are used to limit a reasonable speed range; under the limitation of the two feasible speed sets, the trajectory corresponding to each optional path is scored using a trajectory evaluation optimization function, and the trajectory with the highest smoothness and stability is selected as the optimal trajectory for wheelchair movement;

[0097] Real-time obstacle avoidance and navigation control of wheelchairs in real-time operation.

[0098] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0099] An electronic device according to the third embodiment of the present invention comprises: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned wheelchair real-time obstacle prediction and avoidance method based on multi-target tracking.

[0100] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned wheelchair real-time obstacle prediction and avoidance method based on multi-target tracking.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the storage device and processing device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0102] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0103] Reference below Figure 4 , which shows a schematic diagram of the structure of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 4 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Figure 4 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0107] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0110] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0111] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking, characterized in that: The system comprises: A data acquisition and processing module, used to capture images and collect scene data, and calculate the distance between the obstacle and the reference point based on the scene data; The obstacle detection and tracking module is used to generate multi-dimensional features based on the captured image, generate suspected areas of obstacles based on the multi-dimensional features; filter out the determined areas based on the prediction deviation of the suspected areas, then calculate the similarity of the suspected dynamic obstacles in the determined areas, and filter out the dynamic obstacles based on the similarity, then calculate the credibility score for each obstacle, and filter out the determined obstacles based on the credibility score; A decision control module is used to define a path evaluation function based on obstacles and distances between obstacles and reference points, and to determine an optional path based on the path evaluation function; to limit a reasonable speed range using two feasible speed sets; and to score the trajectory corresponding to each optional path using a trajectory evaluation optimization function under the limitation of the two feasible speed sets, and to select the trajectory with the highest smoothness and stability as the optimal trajectory for the wheelchair movement; Interaction module, used to provide visual interface interaction and user operation; The communication module is used for communication among various modules based on the communication mechanism.

2. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 1 is characterized in that: Calculating the distance between an obstacle and a reference point based on scene data includes: in, is the distance between the obstacle and the target point, is the pixel distance function, is the pixel matching function, is the first constraint function, is the second constraint function, α′ and β′ are both dynamic compensation factors, i represents the pixel index in the image, Δ i is the matching displacement of pixel i, Δ j is the matching displacement of pixel j, j is the neighboring pixel of pixel i, It is the compensation item for environmental complexity.

3. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 2 is characterized in that: Prediction deviations for suspected areas include: Among them, k represents the number of samples, s i Indicates the position deviation between the actually detected obstacle area and the actual obstacle area. Represents the predicted obstacle coordinate information deviation value, P′ i is the weight coefficient for obstacle feature similarity adjustment, W(·) is the error weight function, E represents the predicted deviation value, s x is the horizontal coordinate deviation of the obstacle area center, s y is the vertical coordinate deviation of the obstacle area center, s w is the obstacle area width deviation, s h is the height deviation of the obstacle area, s x ′ is the horizontal coordinate deviation of the center of the auxiliary view obstacle area, s′ w It is the auxiliary view obstacle area width deviation.

4. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 3 is characterized in that: The similarity of dynamic obstacles in the determined area is calculated by: D=α·D centre +β·D angle +γ·D volume +δ·D traj ; D centre =||C det -C pre ||2+η·||A det -TO pre ||2: Among them, α, β, γ, and δ are weight coefficients adjusted over time, and C det With C pre are the center coordinates of the detected 3D target frame and the predicted 3D target frame, respectively. det With A pre are the acceleration vectors of the detected 3D target frame and the predicted 3D target frame, respectively, det With γ pre are the orientation angles of the detected 3D target frame and the predicted 3D target frame, V det With V pre They represent the volumes of the detected 3D target box and the predicted 3D target box, v(t) is the environment complexity parameter, and T det With T pre are the trajectory vectors of the detected target and the predicted target, respectively, N is the number of trajectory points, τ(t) is the time decay factor, and D centre is the target center matching similarity, D angle is the target angle matching similarity, D volume is the target volume matching similarity, D traj Match the target trajectory similarity.

5. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 4 is characterized in that: Calculation of the obstacle credibility score includes: S det =D volume ·P(c); Among them, S det To consider the similarity of obstacle shapes, P(c) is the probability of obstacle detection, S motion is used to measure the stability of the target's motion state; S is the credibility score of the obstacle, They are all weight coefficients for dynamically adjusting the credibility of different factors.

6. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 5, characterized in that: The path evaluation function is: in, is the path evaluation function, is the cumulative score from the starting point to the current node, and Inversely proportional to the speed of the vehicle, ensuring that the path planning actively avoids close obstacles. is the target distance estimate from the current node to the target node, is the path curvature evaluation function, (x cur ,y cur ) is the current node coordinate, (x goal ,y goal ) is the target node coordinate, μ(t) is the environment adaptive adjustment coefficient, δ is the balance bending weight adjustment coefficient, κ is the environment complexity adjustment parameter, and ε(t) is the obstacle density and node distribution complexity function.

7. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 6, characterized in that: The two possible speed sets are: V m ={(v,ω)|v∈[v min ,v max ],ω∈[ω min ,ω max ],T safe (v,w)>τ}; Among them, V m is the driving speed limit set, v min and v max Represent the minimum and maximum linear speeds, ω min and ω max represents the minimum angular velocity and the maximum angular velocity, v is the linear velocity, ω is the angular velocity, and τ is the safety time threshold; V a is the safe speed limit set, dis(v,ω) is the shortest distance between the trajectory corresponding to the moving wheelchair speed and the obstacle, and are the maximum linear acceleration and the maximum angular acceleration.

8. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 7, characterized in that: Under the limitation of two feasible speed sets, the trajectory evaluation optimization function is used to score the trajectory corresponding to each speed path planning, including: Among them, σ is a smoothing function, α, β, and γ are all adaptive adjustment weights of each score, Align(v,ω) is a function that measures the consistency between the wheelchair's driving direction and the target direction; Gap(v,ω) represents the minimum safe gap between the current trajectory and the nearest obstacle, based on the minimum obstacle distance The angle θ with the velocity direction i , avoiding trajectories approaching obstacles at high speed, Stability(v,ω) reflects the stability of the trajectory, is the optimal function for trajectory evaluation.

9. The wheelchair real-time obstacle prediction and avoidance system based on multi-target tracking according to claim 8, characterized in that: The decision control module also includes real-time obstacle avoidance and navigation control of the wheelchair in real-time operation. The formula for obstacle avoidance and navigation control is: Among them, e(t) represents the deviation between the current position and the expected path, is the deviation change rate, K p With K d are the adjustment coefficient and differential adjustment coefficient respectively, ΔK is the dynamic adjustment of control strength, K f is the feedforward adjustment coefficient, p(t) is the feedforward prediction factor, R(t) is the real-time obstacle information, and u(t) is the control input, which is used to measure the control signal of the wheelchair in real-time obstacle avoidance and navigation control.

10. A wheelchair real-time obstacle prediction and avoidance method based on multi-target tracking, characterized in that: The method comprises: Calculate the distance between the obstacle and the reference point based on the scene data; Generate a suspected area of ​​an obstacle based on multi-dimensional features; select a determined area based on the prediction deviation of the suspected area, and then calculate the similarity of dynamic obstacles in the determined area to select dynamic obstacles, and then calculate the credibility score for each obstacle, and determine the obstacle based on the credibility score of the obstacle; A path evaluation function is defined based on the determined obstacles and the distance between the obstacles and the reference point, and an optional path is determined based on the path evaluation function; two feasible speed sets are used to limit a reasonable speed range; under the limitation of the two feasible speed sets, the trajectory corresponding to each optional path is scored using a trajectory evaluation optimization function, and the trajectory with the highest smoothness and stability is selected as the optimal trajectory for wheelchair movement; Real-time obstacle avoidance and navigation control of wheelchairs in real-time operation.

Images