A wheelchair intelligent control method, device and equipment based on neural network optimization

Abstract

This application relates to a wheelchair intelligent control method, device, and equipment based on neural network optimization. The method includes acquiring an RGB image and a depth image of the environment in which the mobile device is located, as well as acquiring the pose information of the mobile device; processing the RGB image and depth image using lane line detection, passable area detection, and trajectory prediction neural networks to obtain an occupancy grid image; processing the pose information and the occupancy grid image using global path planning algorithms and local path planning algorithms to obtain motion planning data for the mobile device; and controlling the operation of the mobile device based on the motion planning data. This method obtains an occupancy grid image by processing the acquired RGB image and depth image of the mobile device's environment using lane line detection, passable area detection, and trajectory prediction neural networks, respectively. The motion planning path planned using the occupancy grid image can improve the safety of the mobile device's operation; and solves the technical problem that the driving safety of existing wheelchair intelligent control technologies cannot be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a method, device and equipment for intelligent control of a wheelchair based on neural network optimization. Background Art

[0002] The existing intelligent control technology of smart wheelchairs may shuttle between lanes or even drive in opposite lanes in order to avoid obstacles in multi-lane scenarios, resulting in dangerous behavior of the smart wheelchairs.

[0003] The existing intelligent control technology of intelligent wheelchairs is difficult to distinguish between passable and impassable areas on the same plane during walking. For example, when an intelligent wheelchair is walking in a park scene, the passable park road and impassable grass on the same plane, although some areas (such as park grass) do not constitute obstacles in terms of height, the wheelchair is not suitable for driving in these areas.

[0004] The existing intelligent control technology of smart wheelchairs is difficult to deal with dynamic obstacles whose motion state is prone to significant changes. For example, when a pedestrian is moving forward, he may suddenly change direction. In the face of such dynamic obstacles, the existing intelligent control technology of smart wheelchairs may require extremely high computing power to make correct real-time responses, which will greatly increase the cost of the wheelchair.

[0005] The intelligent control technology design of existing smart wheelchairs is deeply bound to the structure of the wheelchair itself. When the wheelchair structure changes, the original intelligent control technology solution will no longer be used normally, which makes the intelligent control technology solution of existing smart wheelchairs not scalable, which will increase the design cost of the smart wheelchair. Summary of the invention

[0006] The embodiments of the present application provide a method, device and equipment for intelligent control of a wheelchair based on neural network optimization, which are used to solve the technical problem that the driving safety of the existing wheelchair intelligent control technology cannot be guaranteed.

[0007] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0008] On the one hand, a wheelchair intelligent control method based on neural network optimization is provided, comprising the following steps:

[0009] Obtaining an RGB image and a depth image of the environment in which the mobile device is located, and obtaining position information of the mobile device;

[0010] The RGB image and the depth image are processed by lane detection, drivable area detection and trajectory prediction neural network to obtain an occupancy grid image;

[0011] Using a global path planning algorithm and a local path planning algorithm on the posture information and the occupancy grid map to obtain movement planning data of the mobile device;

[0012] The mobile device is controlled to operate according to the mobile planning data.

[0013] Preferably, the RGB image and the depth image are processed by lane line detection, passable area detection and trajectory prediction neural network to obtain an occupancy grid image, including:

[0014] Using a lane detection neural network to perform lane detection on the RGB image to obtain lane information;

[0015] Using a traversable area detection neural network to perform area detection on the RGB image and the depth image to obtain a traversable area map;

[0016] Using a trajectory prediction neural network to perform dynamic obstacle detection on the RGB image and the depth image to obtain motion trajectory information of the dynamic obstacles;

[0017] An initial occupancy grid map is constructed according to the RGB map and the depth map, and the initial occupancy grid map is updated according to the lane information, the drivable area map and the motion trajectory information to obtain an occupancy grid map.

[0018] Preferably, the content of the lane line detection neural network includes:

[0019] Setting lane types and setting a mapping keypoint on each of said lane types;

[0020] Introducing a hybrid anchor point mechanism and a detection branch into the fully connected layer of the lane line detection neural network, anchoring the lane lines of the RGB image through the hybrid anchor point mechanism, and obtaining key points corresponding to each lane line;

[0021] Detecting all the key points in the RGB image by the detection branch to determine whether there is a lane line, and comparing the key points corresponding to the detected lane line with the mapped key points of each lane type to obtain the lane type of the lane line;

[0022] All the lane lines and their corresponding lane types are detected to construct the lane information corresponding to the RGB.

[0023] Preferably, the content of the traversable area detection neural network includes:

[0024] Obtaining normal information of each pixel from the depth map;

[0025] Using a deep neural network RGB encoder and a surface normal encoder to extract a feature map of each pixel from the RGB image and each piece of normal information, respectively, to obtain an RGB feature map and a surface normal feature map corresponding to each pixel;

[0026] Using the fusion device of the deep neural network to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel point element by element, so as to obtain a fused feature map corresponding to each pixel point;

[0027] The fused feature map is converted using the Sigmoid layer of the deep neural network to obtain a probability map;

[0028] Probability data is obtained from the probability map, and whether an area corresponding to the probability map is marked as a passable area is determined based on the comparison between the probability data and a set probability threshold, so as to obtain a passable area map.

[0029] Preferably, the content of the trajectory prediction neural network includes:

[0030] Constructing a two-dimensional coordinate system and a three-dimensional coordinate system according to the RGB image, and using a recognition model of a deep neural network to perform dynamic obstacle recognition on the RGB image to obtain a state detection value corresponding to each dynamic obstacle in the environment where the mobile device is located;

[0031] The state detection value is estimated by using a Kalman filter to obtain a state information prediction value corresponding to the dynamic obstacle; a position of the dynamic obstacle is estimated by using a Kalman filter algorithm according to the state information prediction value and the state detection value to obtain an optimal estimated position information corresponding to the dynamic obstacle;

[0032] Acquire depth data of each of the dynamic obstacles from the depth map, construct a corresponding obstacle model in the three-dimensional coordinate system according to the depth data of each of the dynamic obstacles, and acquire optimal three-dimensional information of each of the dynamic obstacles in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information;

[0033] Acquire updated optimal estimated position information of each of the dynamic obstacles in real time and record it as the latest status information;

[0034] The optimal estimated position information and the latest state information of two adjacent objects are associated and matched using IOU similarity and Re-ID re-identification to obtain dynamic tracking information of each dynamic obstacle;

[0035] The dynamic tracking information of each dynamic obstacle is predicted by using a three-dimensional Kalman filter to obtain a plurality of predicted trajectory points corresponding to each dynamic obstacle, and all the predicted trajectory points are connected to obtain the motion trajectory information corresponding to the dynamic obstacle.

[0036] Preferably, the position information and the occupancy grid map are processed using a global path planning algorithm and a local path planning algorithm to obtain the mobile planning data of the mobile device, including:

[0037] Acquire the current location and the end location of the mobile device in the occupancy grid map, as well as the road network information and obstacle information between the current location and the end location;

[0038] Taking the current location as the starting point and the end location as the end point, performing a path search in the occupancy grid map using the A star algorithm as a global path planning algorithm according to the road network information and the obstacle information, and obtaining an optimal running path for the mobile device;

[0039] The optimal running path is optimized using a Bezier curve to obtain an optimal optimization path;

[0040] Discretize the optimal optimization path to obtain discrete points; connect all the discrete points to form a preliminary elastic band;

[0041] The preliminary elastic band is optimized by using the TEB algorithm to obtain an optimized elastic band and the optimized elastic band is used as the movement planning data of the mobile device.

[0042] Preferably, the wheelchair intelligent control method based on neural network optimization includes: acquiring in real time an updated RGB map and an updated depth map of the environment in which the mobile device is located after movement; updating the occupancy grid map according to the updated RGB map and the updated depth map; updating the optimal optimization path according to the updated occupancy grid map, and then updating the movement planning data of the mobile device.

[0043] Preferably, the content of acquiring the position information of the mobile device includes:

[0044] Obtaining grayscale images of the previous and next two frames of the environment in which the mobile device is located and initial posture information;

[0045] The grayscale image of each frame is processed using the FAST feature point detection algorithm and the Harris corner point detection algorithm to obtain feature point data corresponding to the grayscale image of each frame;

[0046] Performing feature description on each feature point in the feature point data to obtain a feature point descriptor corresponding to each feature point; performing feature point matching on all the feature point descriptors of the previous and next two frames using the Hamming distance to obtain a feature point pair set;

[0047] The EPnP algorithm is used to process each pair of feature points in the feature point set to obtain relative posture information between the previous and next two frames of the mobile device corresponding to the feature point; the initial posture information is superimposed with all the relative posture information to obtain the current posture information of the mobile device.

[0048] Preferably, the EPnP algorithm is used to process each pair of feature points in the feature point pair set, and the relative posture information between the previous and next frames of the mobile device corresponding to the feature point is obtained, including:

[0049] Acquire the number of feature point pairs and all feature points corresponding to the grayscale image of the previous frame and the grayscale image of the next frame from the feature point pair set and record them as the first key feature point and the second key feature point respectively, and construct the coordinate system of the previous frame and the coordinate system of the next frame according to the grayscale images of the previous and next frames;

[0050] Obtaining the three-dimensional coordinates of each of the first key feature point and the second key feature point in the front frame coordinate system and the rear frame coordinate system and recording them as first three-dimensional coordinates and second three-dimensional coordinates respectively; obtaining the first control three-dimensional coordinates of the first front frame control point according to all the first three-dimensional coordinates; obtaining the second control three-dimensional coordinates of the first rear frame control point according to all the second three-dimensional coordinates;

[0051] According to all the first three-dimensional coordinates and the three-dimensional coordinates of the first front frame control point, a third control three-dimensional coordinate of the front frame control point corresponding to the three main directions of the first front frame control point is calculated; according to all the second three-dimensional coordinates and the three-dimensional coordinates of the first rear frame control point, a fourth control three-dimensional coordinate of the rear frame control point corresponding to the three main directions of the first rear frame control point is calculated;

[0052] Obtaining a front frame conversion coefficient and a rear frame conversion coefficient of each front frame control point and rear frame control point; calculating based on all the front frame conversion coefficients, the first control three-dimensional coordinates and all the third control three-dimensional coordinates, obtaining first three-dimensional coordinate representative data corresponding to all the first key feature points of the grayscale image of the front frame; calculating based on all the rear frame conversion coefficients, the second control three-dimensional coordinates and all the fourth control three-dimensional coordinates, obtaining second three-dimensional coordinate representative data corresponding to all the second key feature points of the grayscale image of the rear frame;

[0053] Iterative optimization is performed using a singular value decomposition method according to the number of feature point pairs, the first three-dimensional coordinate representative data, and the second three-dimensional coordinate representative data to obtain a rotation matrix and a translation matrix with a minimum error value;

[0054] The position information includes a rotation matrix and a translation matrix.

[0055] In yet another aspect, there is provided a wheelchair intelligent control device based on neural network optimization, comprising a data acquisition module, a data processing module, a path planning module and a control execution module;

[0056] The data acquisition module is used to acquire an RGB image and a depth image of the environment in which the mobile device is located, and to acquire position information of the mobile device;

[0057] The data processing module is used to process the RGB image and the depth image using lane line detection, passable area detection and trajectory prediction neural network to obtain an occupancy grid image;

[0058] The path planning module is used to use a global path planning algorithm and a local path planning algorithm to perform motion planning data of the mobile device on the posture information and the occupancy grid map;

[0059] The control execution module is used to control the operation of the mobile device according to the mobile planning data.

[0060] Preferably, the data processing module includes a lane detection submodule, a traffic area detection submodule, an obstacle detection submodule and a grid map generation submodule;

[0061] The lane detection submodule is used to perform lane detection on the RGB image using a lane line detection neural network to obtain lane information;

[0062] The passable area detection submodule is used to perform area detection on the RGB image and the depth image using a passable area detection neural network to obtain a passable area map;

[0063] The obstacle detection submodule is used to perform dynamic obstacle detection on the RGB image and the depth image using a trajectory prediction neural network to obtain motion trajectory information of the dynamic obstacles;

[0064] The grid map generation submodule is used to construct an initial occupancy grid map according to the RGB map and the depth map, and update the initial occupancy grid map according to the lane information, the passable area map and the motion trajectory information to obtain an occupancy grid map;

[0065] The content of the lane line detection neural network includes: setting the lane type and setting a mapping key point on each lane type; introducing a hybrid anchor point mechanism and a detection branch in the fully connected layer of the lane line detection neural network, anchoring the lane line of the RGB image through the hybrid anchor point mechanism, and obtaining the key point corresponding to each lane line; detecting all the key points in the RGB image through the detection branch to determine whether there is a lane line, comparing the key point corresponding to the detected lane line with the mapping key point of each lane type to obtain the lane type of the lane line; constructing the lane information corresponding to the RGB by detecting all the lane lines and their corresponding lane types;

[0066] The content of the passable area detection neural network includes: obtaining normal information of each pixel from the depth map; using the RGB encoder and the surface normal encoder of the deep neural network to extract the feature map of each pixel from the RGB map and each normal information respectively, to obtain the RGB feature map and the surface normal feature map corresponding to each pixel; using the fusion device of the deep neural network to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel element by element, to obtain the fused feature map corresponding to each pixel; using the Sigmoid layer of the deep neural network to transform the fused feature map to obtain a probability map; obtaining probability data from the probability map, and determining whether the area corresponding to the probability map is marked as a passable area according to the comparison between the probability data and the set probability threshold, to obtain a passable area map;

[0067] The content of the trajectory prediction neural network includes: constructing a two-dimensional coordinate system and a three-dimensional coordinate system according to the RGB image, using a recognition model of a deep neural network to identify dynamic obstacles on the RGB image, and obtaining state detection values ​​corresponding to each dynamic obstacle in the environment where the mobile device is located; using Kalman filter estimation to estimate the state detection value, and obtaining a state information prediction value corresponding to the dynamic obstacle; using a Kalman filter algorithm to estimate the position of the dynamic obstacle according to the state information prediction value and the state detection value, and obtaining the optimal estimated position information corresponding to the dynamic obstacle; obtaining the depth data of each dynamic obstacle from the depth map, and calculating the optimal estimated position information of each dynamic obstacle in the three-dimensional coordinate system according to the depth data of each dynamic obstacle. The method comprises the following steps: constructing a corresponding obstacle model in the coordinate system, and obtaining the optimal three-dimensional information of each of the dynamic obstacles in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information; obtaining the updated optimal estimated position information of each of the dynamic obstacles in real time and recording it as the latest state information; performing association matching on all two adjacent optimal estimated position information and the latest state information by using IOU similarity and Re-ID re-identification to obtain the dynamic tracking information of each of the dynamic obstacles; predicting the dynamic tracking information of each of the dynamic obstacles by using a three-dimensional Kalman filter to obtain a plurality of predicted trajectory points corresponding to each of the dynamic obstacles, and connecting all the predicted trajectory points to obtain the motion trajectory information corresponding to the dynamic obstacle.

[0068] In another aspect, a terminal device is provided, comprising a processor and a memory;

[0069] The memory is used to store program codes and transmit the program codes to the processor;

[0070] The processor is used to execute the above-mentioned wheelchair intelligent control method based on neural network optimization according to the instructions in the program code.

[0071] The method, device and equipment for intelligent wheelchair control based on neural network optimization, the method includes obtaining an RGB image and a depth image of the environment in which the mobile device is located, and obtaining the posture information of the mobile device; using lane line detection, passable area detection and trajectory prediction neural network processing on the RGB image and the depth image to obtain an occupancy grid map; using a global path planning algorithm and a local path planning algorithm on the posture information and the occupancy grid map to obtain the mobile planning data of the mobile device; and controlling the operation of the mobile device according to the mobile planning data. It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: the method for intelligent wheelchair control based on neural network optimization obtains an RGB image and a depth image of the environment in which the mobile device is located, and uses lane line detection, passable area detection and trajectory prediction neural network processing to obtain an occupancy grid map. The mobile planning path planned by the occupancy grid map can improve the driving safety of the mobile device; it solves the technical problem that the driving safety of the existing wheelchair intelligent control technology cannot be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0073] Figure 1 This is a flowchart of the steps of the wheelchair intelligent control method based on neural network optimization described in the embodiment of the present application;

[0074] Figure 2 A schematic diagram of the framework of the intelligent wheelchair control device based on neural network optimization described in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a terminal device described in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0078] In the embodiments of the present application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0079] As society ages, the travel and care needs of the elderly continue to increase. As an assistive device that can help the elderly move independently and improve their quality of life, smart wheelchairs have been increasingly widely used.

[0080] With the continuous advancement of science and technology, the technology of smart wheelchairs is also constantly developing and improving. Smart wheelchairs not only have the functions of traditional wheelchairs, but also combine knowledge from multiple fields such as sensor technology, computer vision technology, path planning algorithms and control systems. They can realize multiple functions such as automatic navigation, obstacle avoidance, safety warning, etc., providing users with a more convenient, safe and comfortable use experience. Smart wheelchairs are also suitable for other people who need assisted movement, such as the disabled, the injured and the sick. When these people use traditional wheelchairs, they often have problems such as difficulty in operation and safety hazards. Smart wheelchairs can help them better achieve autonomous movement and self-care through intelligent control and navigation.

[0081] The application of smart wheelchairs is also in line with the trend and needs of social development. With the development of society, people's requirements for quality of life are constantly improving, and the demand for assistive devices is also increasing. As a high-tech, high-value-added assistive device, smart wheelchairs not only meet people's basic needs, but also bring people a better life experience and more convenience. Therefore, the application scenarios of smart wheelchairs are very wide. It can not only meet the travel and care needs of the elderly and disabled, but also meet the trend and needs of social development. With the continuous advancement of technology and the continuous expansion of application fields, smart wheelchairs will surely play a more important role and bring more convenience and value to people's lives.

[0082] The intelligent control technology of existing smart wheelchairs collects environmental data through a variety of sensors. The collected environmental data cannot be directly used for path planning. It is necessary to extract features from the environmental data. After obtaining the environmental features, the control system of the smart wheelchair can perform path planning. After obtaining the path planning results, the control system will drive the wheelchair to move. Sensors include ultrasonic sensors, infrared sensors, laser rangefinders, etc., which can sense the distance, direction and other information of objects in the surrounding environment. In addition, image information of the surrounding environment can be obtained through visual sensors such as cameras to provide data sources for subsequent computer vision processing. Environmental feature extraction refers to converting environmental data collected by sensors into environmental features that can be used for path planning. Environmental features can include information such as the location, size, shape, etc. of obstacles, and can also include location information of landmarks or reference objects. Path planning refers to planning a path that can avoid obstacles and reach the target location safely based on the current position and target position. Commonly used path planning algorithms include rule-based algorithms (such as A* algorithm, Dijkstra algorithm, etc.), search-based algorithms (such as best-first search, depth-first search, etc.) and learning-based algorithms (such as reinforcement learning algorithms, etc.). The control system generally includes a motor controller, a steering gear controller, etc., which can drive the motor and steering gear of the wheelchair according to the control instructions to realize the movement and steering of the wheelchair. At the same time, the control system also needs to monitor the status of the wheelchair in real time, such as speed, direction, etc., to ensure that the wheelchair can move safely according to the planned path.

[0083] During the actual operation of the smart wheelchair, the automatic navigation system of the smart wheelchair needs to be continuously adjusted and optimized based on real-time feedback data. For example, when there is a large deviation between the actual walking path and the planned path, the control system needs to adjust the control strategy of the motor and servo in time to get the wheelchair back on the correct path. At the same time, through continuous learning and training, the automatic navigation system of the smart wheelchair can gradually improve the efficiency and accuracy of its path planning.

[0084] The embodiments of the present application provide a method, device and equipment for intelligent control of a wheelchair based on neural network optimization, which solves the technical problem that the driving safety of the existing wheelchair intelligent control technology cannot be guaranteed.

[0085] Embodiment 1:

[0086] Figure 1 This is a flowchart of the steps of the wheelchair intelligent control method based on neural network optimization described in an embodiment of the present application.

[0087] like Figure 1 As shown, the embodiment of the present application provides a wheelchair intelligent control method based on neural network optimization, comprising the following steps:

[0088] S1. Obtain an RGB image and a depth image of the environment in which the mobile device is located, and obtain the position information of the mobile device.

[0089] It should be noted that in step S1, first, an RGB image and a depth image of the environment in which the mobile device is located can be obtained through a camera with an RGB-D model, and second, the position information of the mobile device can be obtained. In this embodiment, the mobile device can be a smart wheelchair. In other embodiments, the mobile device can be a device or apparatus that requires intelligent control of movement.

[0090] S2. Lane detection, drivable area detection and trajectory prediction neural network are used to process the RGB image and depth image to obtain an occupancy grid map.

[0091] It should be noted that in step S2, the RGB image and depth image obtained in step S1 are processed by lane line detection, passable area detection and trajectory prediction neural network to obtain data to construct an occupancy grid map, which provides a basis for subsequent path planning.

[0092] S3. A global path planning algorithm and a local path planning algorithm are used for the posture information and the occupancy grid map to obtain the mobile planning data of the mobile device.

[0093] It should be noted that in step S3, path planning is performed based on the posture information obtained in step S1 and the occupancy grid map obtained in step S2 to obtain movement planning data of the mobile device.

[0094] S4. Control the operation of the mobile device according to the mobile planning data.

[0095] It should be noted that in step S4, the mobile planning data obtained in step S3 is compiled into binary code in the form of CAN communication coding, and the compiled binary code is sent to the control module of the mobile device through the CAN bus, thereby controlling the operation of the mobile device.

[0096] In an embodiment of the present application, an intelligent control system that operates according to the intelligent control method for wheelchairs based on neural network optimization is provided on a mobile device. The intelligent control system includes a perception module, a positioning module, a planning module, and an instruction conversion module that are connected in sequence, and the output end of the instruction conversion module is connected to the mobile device. An optimization module is connected between the perception module and the planning module. The intelligent control system adopts a modular design, which is convenient for subsequent intelligent upgrades and improvements to the intelligent control system, and also convenient for intelligent upgrades and modifications to mobile devices. This allows the intelligent control method for wheelchairs based on neural network optimization to be adapted to various mobile devices on the market.

[0097] It should be noted that the perception module and the positioning module are used to implement the technical solution of step S1. The planning module is used to implement the technical solution of step S3. The optimization module is used to implement the technical solution of step S2. Among them, the perception module uses visual perception to obtain the RGB image and depth image of the environment in which the mobile device is located, avoiding the use of high-cost perception components such as lidar, and effectively reducing the cost of mobile devices. The optimization module introduces neural networks to optimize the existing intelligent technical solutions for mobile devices, greatly improving the safety of automatic navigation of mobile devices.

[0098] In an embodiment of the present application, the intelligent control method for a wheelchair based on neural network optimization obtains an RGB image and a depth image of the environment in which the mobile device is located, and uses lane line detection, passable area detection and trajectory prediction neural network processing to obtain an occupancy grid map. The mobile planning path planned by the occupancy grid map can limit the movement of the mobile device to the set lane line, thereby improving the safety of the mobile device. It can also accurately determine the passable area, so that the mobile device no longer travels in all obstacle-free areas, but travels in the passable area selected in the obstacle-free area. While improving the safety of the mobile device, the travel of the mobile device is more in line with social rules. It can also obtain the motion trajectory information of dynamic obstacles, and apply the future trajectory to path planning, thereby improving the mobile device's ability to respond to emergencies, thereby improving the safety of the mobile device.

[0099] The present application provides a method for intelligent wheelchair control based on neural network optimization, the method comprising obtaining an RGB image and a depth image of the environment in which the mobile device is located, and obtaining the posture information of the mobile device; processing the RGB image and the depth image using a lane line detection, a passable area detection and a trajectory prediction neural network to obtain an occupancy grid image; processing the posture information and the occupancy grid image using a global path planning algorithm and a local path planning algorithm to obtain mobile planning data of the mobile device; and controlling the operation of the mobile device according to the mobile planning data. The method for intelligent wheelchair control based on neural network optimization obtains an RGB image and a depth image of the environment in which the mobile device is located, and processes them using a lane line detection, a passable area detection and a trajectory prediction neural network to obtain an occupancy grid image. The mobile planning path planned by the occupancy grid image can improve the driving safety of the mobile device; and solves the technical problem that the driving safety of the existing wheelchair intelligent control technology cannot be guaranteed.

[0100] In one embodiment of the present application, lane line detection, drivable area detection and trajectory prediction neural network processing are used on the RGB image and the depth image to obtain an occupancy grid image including:

[0101] The lane detection neural network is used to detect lanes in the RGB image to obtain lane information;

[0102] The traversable area detection neural network is used to perform area detection on the RGB image and the depth image to obtain a traversable area map.

[0103] The trajectory prediction neural network is used to detect dynamic obstacles on the RGB image and depth image to obtain the motion trajectory information of dynamic obstacles;

[0104] An initial occupancy grid map is constructed according to the RGB image and the depth map, and the initial occupancy grid map is updated according to the lane information, the passable area map and the motion trajectory information to obtain the occupancy grid map.

[0105] It should be noted that the intelligent wheelchair control method based on neural network optimization obtains lane information of lane line positions according to the RGB map through the lane line detection neural network, thereby limiting the movement of the mobile device within the set lane line, improving the safety of the mobile device. The intelligent wheelchair control method based on neural network optimization trains and learns the depth map and RGB map through the pre-deep neural network in the passable area detection neural network, and can accurately determine the passable area, so that the mobile device no longer drives in all obstacle-free areas, but drives in the obstacle-free area. The driving safety of the mobile device is improved, and the driving of the mobile device is more in line with social rules. The intelligent wheelchair control method based on neural network optimization obtains the future motion trajectory information of dynamic obstacles through the deep neural network model of the trajectory prediction neural network, and applies the future trajectory to path planning, which can improve the mobile device's ability to respond to emergencies, thereby improving the safety of the mobile device.

[0106] In an embodiment of the present application, during the construction of the initial occupancy grid map, various map parameters are first set according to the RGB map and the depth map. Subsequently, the camera device on the mobile device is started and various hardware parameters of the camera device are set, the left and right eye RGB maps and depth maps of the camera device, and the pose information of the visual odometer transformation of the camera device are obtained, and the lane line detection, passable area detection and trajectory prediction neural network are used to process the RGB map and the depth map to complete 3D target detection and tracking, trajectory prediction, detection of passable areas and lane lines, detection of road edges, camera device (car body) positioning information fusion and other tasks. At the same time, the obstacle point cloud generated by the camera device is obtained, the point cloud is filtered and only the point cloud within a given range is retained to update the obstacle information on the initial occupancy grid map.

[0107] It should be noted that the parameters include resolution, width, height, initial offset and rotation, etc. The camera device may be a camera of an RGB-D model.

[0108] In the embodiment of the present application, during the process of updating the initial occupancy grid map, the coordinates of the point clouds such as the motion trajectory information, lane information, and passable area map of the 3D target and its dynamic obstacles are uniformly converted to the coordinate system of the mobile device. With the position of the camera device as the origin, a small window is constructed on the initial occupancy grid map, and the range of all the above points in the coordinate system of the mobile device is divided according to the map resolution, and projected onto the window area accordingly, and the local state of the initial occupancy grid map is updated in real time according to the information on the window area. Among them, the grid state corresponding to the point clouds such as the motion trajectory information, lane information, and passable area map of the dynamic obstacles on the initial occupancy grid map is set to be occupied, indicating that there is an obstacle at this position on the initial occupancy grid map; the grid state corresponding to the predicted trajectory of the 3D target is set to possible existence or impending obstacle; the grid state of the remaining positions is set to unoccupied, indicating that there is no obstacle at this position on the initial occupancy grid map. After the point cloud projection is completed, the grids adjacent to the grids with obstacles are also set to be occupied to update the initial occupancy grid map, so as to avoid the mobile device from being too close to the obstacle when driving along the path obtained by path planning, causing safety accidents.

[0109] It should be noted that point cloud projection means that all z-axis coordinates of the point cloud are set to 0, so that the three-dimensional point cloud can be projected onto a two-dimensional plane.

[0110] In one embodiment of the present application, the content of the lane detection neural network includes:

[0111] Set the lane types and set a mapping keypoint on each lane type;

[0112] A hybrid anchor mechanism and a detection branch are introduced into the fully connected layer of the lane detection neural network. The hybrid anchor mechanism is used to anchor the lanes in the RGB image and obtain the key points corresponding to each lane.

[0113] The detection branch detects all key points in the RGB image to determine whether there is a lane line, and compares the key points corresponding to the detected lane line with the mapped key points of each lane type to obtain the lane type of the lane line;

[0114] All lane lines and their corresponding lane types will be detected to construct lane information corresponding to RGB.

[0115] It should be noted that the lane types include left lane, center lane and right lane. The key points on each lane line need to be classified into one of these three types. In the lane line detection neural network, lane line detection is regarded as a classification problem, so the category of each lane line key point can be mapped by the number of categories. The lane line detection neural network also adds a detection branch to determine whether the lane line exists at that location. The data output by the detection branch is 1 or 0, 1 represents the existence of the lane line, and 0 represents the non-existence of the lane line. A hybrid anchor point mechanism is designed in the fully connected layer of the lane line detection neural network to enhance the global perception ability of the lane line detection neural network. In order to represent the lane line, the lateral anchor point is first introduced, and the lane line is regarded as a set of key points of the lateral anchor point. In this embodiment, the hybrid anchor point mechanism refers to the use of lateral anchor points to describe the lane line with a large horizontal angle in the middle, and the use of longitudinal anchor points to describe the lane lines with small horizontal angles on both sides. The key point represents a lane line, and the key point can be represented by a set of normalized coordinates. For example, each lane line is described as a set of normalized coordinate values ​​relative to a reference point, and this set of coordinate values ​​is usually within a certain range, such as between 0 and 1. For example, the normalized coordinates of a lane line are (0.3, 0.7), which represent the position of the lane line at a certain reference point, where the first value 0.3 is the normalized horizontal coordinate and the second value 0.7 is the normalized vertical coordinate; the normalized coordinates allow lane lines of different lengths or shapes to be represented in a relatively consistent way. In general, using normalized coordinates can help the lane line detection neural network better understand and learn the position of the lane line without being affected by the specific image size or camera device viewing angle.

[0116] In one embodiment of the present application, the content of the traversable area detection neural network includes:

[0117] Get the normal information of each pixel from the depth map;

[0118] The RGB encoder and surface normal encoder of the deep neural network are used to extract the feature map of each pixel from the RGB image and each normal information respectively, and the RGB feature map and surface normal feature map corresponding to each pixel are obtained;

[0119] A deep neural network fuser is used to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel point element by element to obtain a fused feature map corresponding to each pixel point;

[0120] The Sigmoid layer of the deep neural network is used to transform the fused feature map to obtain a probability map;

[0121] Probability data is obtained from the probability map, and whether the area corresponding to the probability map is marked as a passable area is determined based on the comparison between the probability data and the set probability threshold, thereby obtaining a passable area map.

[0122] It should be noted that the traversable area detection neural network can be understood as a road traversable area detection method that uses a CNN to integrate normal information, infer surface normal information from dense disparity / depth depth maps, and use a deep neural network learning model to merge RGB and surface normal information into semantic road segmentation. The normal information includes x, y, and z direction components, and the x direction component is recorded as n x , the z-direction component is recorded as n z , the y-direction component is recorded as n y .

[0123] In an embodiment of the present application, obtaining the normal information of each pixel point includes: using the Sobel operator to calculate the horizontal and vertical gradients to obtain the x and y direction components of the normal information; using the KD tree to select 8 neighboring points to estimate the z direction component of the normal information.

[0124] It should be noted that the Sobel operator in the horizontal direction is:

[0125]

[0126] The vertical Sobel operator is:

[0127]

[0128] Use the Sobel operator to calculate the horizontal gradient G of the depth map x and the vertical gradient G y Horizontal gradient G x The calculation formula is: x =d*M x , where d is the depth of the depth map. Vertical gradient G y The calculation formula is: y =d*M y . Then the x-direction component n of the normal information is x The calculation formula is: x =-df x g x ; The y-direction component n of the normal information y The calculation formula is: y =-df y g y , where f x and f yThe focal length of the camera device in pixels respectively. Starting from the root node of the KD tree in the depth map, select an axis for division; the axis is usually selected in the dimension of the current depth; along the selected axis, compare point P with the point on the current node; if the coordinate of P on the axis is less than the coordinate on the current node, move to the left subtree, otherwise move to the right subtree. The above process is recursively performed in the selected subtree. During the backtracking process, check whether the current node is closer to P. If it is closer, add the current node to the list of candidate neighbor points. Next, it is necessary to check whether another subtree may have a closer point. If the distance from the current node to P is less than the known shortest distance, then it is necessary to enter another subtree for recursive search. Repeat the above steps until the nearest 8 neighboring points are found or the entire tree has been traversed. The z-direction component n of the normal information z The calculation formula is:

[0129]

[0130] Where n zi is the z-direction component of the node at the i-th location. ΔX i , ΔY i , ΔZ i are the x-, y-, and z-axis coordinates of the node at the i-th position relative to point P.

[0131] In the embodiment of the present application, the content of fusing RGB and surface normal information to realize passable area detection includes: using an RGB encoder and a surface normal encoder to extract an RGB feature map and a surface normal feature map from the RGB map and the obtained normal information respectively; adding the extracted RGB feature map and the surface normal feature map pixel by pixel to merge the information of the two; then fusing the feature map again in the fuser through a densely connected jump connection to restore the resolution of the feature map to obtain a fused feature map. Finally, a Sigmoid layer is used to generate a probability map for semantic driving scene segmentation.

[0132] In one embodiment of the present application, the content of the trajectory prediction neural network includes:

[0133] A two-dimensional coordinate system and a three-dimensional coordinate system are constructed based on the RGB image, and a recognition model of a deep neural network is used to identify dynamic obstacles in the RGB image to obtain state detection values ​​corresponding to each dynamic obstacle in the environment where the mobile device is located;

[0134] The state detection value is estimated by using Kalman filter estimation to obtain the state information prediction value of the corresponding dynamic obstacle; the position of the corresponding dynamic obstacle is estimated by using Kalman filter algorithm according to the state information prediction value and the state detection value to obtain the optimal estimated position information corresponding to the dynamic obstacle;

[0135] Obtaining depth data of each dynamic obstacle from the depth map, constructing a corresponding obstacle model in a three-dimensional coordinate system according to the depth data of each dynamic obstacle, and obtaining optimal three-dimensional information of each dynamic obstacle in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information;

[0136] Obtain the updated optimal estimated position information of each dynamic obstacle in real time and record it as the latest status information;

[0137] The optimal estimated position information and the latest state information of two adjacent objects are associated and matched using IOU similarity and Re-ID re-identification to obtain the dynamic tracking information of each dynamic obstacle.

[0138] A three-dimensional Kalman filter is used to predict the dynamic tracking information of each dynamic obstacle to obtain several predicted trajectory points corresponding to each dynamic obstacle. All predicted trajectory points are connected to obtain the motion trajectory information corresponding to the dynamic obstacle.

[0139] It should be noted that the trajectory prediction neural network can be understood as a method of target detection, tracking and prediction based on binocular vision. This method uses a well-trained deep neural network learning model to detect dynamic obstacles on real roads or common dynamic targets such as pedestrians and cars. The moving dynamic obstacles in the two-dimensional image of the RGB image are projected into the three-dimensional space, and the target matching between the two frames is performed using the confidence obtained by the IOU matching based on the position of the moving object and the re-identification matching weighted based on the appearance features. This method can reduce the frequency of false detection and missed detection, and effectively realize the tracking of dynamic obstacles with nonlinear motion in three dimensions; finally, the Kalman filter is used to predict the possible future location of the dynamic obstacle.

[0140] In an embodiment of the present application, a pre-trained deep neural network learning model is used to implement dynamic obstacle detection in the RGB image, and the detection result is returned to the two-dimensional coordinate system of the target observation to obtain the two-dimensional coordinates of the dynamic obstacle as the state detection value of the dynamic obstacle.

[0141] It should be noted that when using the pre-trained deep neural network learning model to detect dynamic obstacles in the RGB image, the detection frame of multiple dynamic obstacles as the detection target is first obtained. The detection will return the coordinate values ​​of the upper left corner and the lower right corner of the detection frame. The coordinates of the center point of the dynamic obstacle can be calculated as the two-dimensional coordinates of the dynamic obstacle. The dynamic obstacle can be regarded as a particle, and the state information of the dynamic obstacle is the coordinate information of the particle in each frame. The motion vector of the dynamic obstacle can also be obtained by connecting the particles of the two frames.

[0142] In the embodiment of the present application, according to the state detection value of the dynamic obstacle in the 1~t-1 frame, the Kalman filter is used to estimate the state information prediction value of the dynamic obstacle in the current frame and the state detection value of the dynamic obstacle in the current frame, and the optimal estimation value of the current frame is updated by the former two as the actual state information of each target. The Kalman filter algorithm is used to optimally estimate the position of the moving target, and the Kalman filter can be divided into three stages: prediction stage, observation stage and update stage.

[0143] It should be noted that the Kalman filter algorithm can predict the coordinates of the detection frame of the target in the t-th frame through the coordinates of the detection frame obtained in the previous t-1 frame, which is called the state information prediction value; plus the target detection will be performed in the t-th frame, the coordinates of the detection frame obtained by this detection are called the state detection value; however, due to the influence of the camera equipment process and the environment, the detection value has errors, so the third step of the Kalman filter algorithm is to fuse the detection value and the state information prediction value to obtain the most likely value as the optimal estimate of the dynamic obstacle in the detection frame of the t-th frame, and obtain the optimal estimated position information. The position estimation of the dynamic obstacle can be understood as: now to obtain the coordinates of a dynamic obstacle in the current frame, then the previous t-1 frame is known, assuming that the coordinates predicted by the Kalman filter based on the known are (3, 5), and the detection value of the target detection is (3, 3), then the optimal estimate is to fuse the detection value and the prediction value, and the result obtained by the Kalman filter algorithm may be (3, 4).

[0144] In an embodiment of the present application, during the process of constructing an obstacle model, each dynamic obstacle is three-dimensionally modeled according to its depth data, and the three-dimensional motion state of each dynamic obstacle relative to the mobile device is obtained in the three-dimensional coordinate system according to the optimal estimated position information and the obstacle model as the optimal three-dimensional information of the corresponding dynamic obstacle.

[0145] It should be noted that the contents of building the obstacle model include:

[0146] The camera device is mounted on the mobile device and moves with it. To obtain the movement trajectory of the moving obstacle, we must first establish a two-dimensional coordinate system with the optical center of the left camera device as the origin. According to the right-hand rule, the X-axis of the two-dimensional coordinate system is defined to be parallel to the horizontal plane, perpendicular to the forward direction of the mobile device and right positive, the Y-axis is parallel to the horizontal plane, positive in the forward direction of the mobile device, and the Z-axis is perpendicular to the horizontal direction. At this time, the three-dimensional coordinates of each dynamic obstacle relative to the camera device can be obtained. The movement trajectory of the dynamic obstacle is also the relative trajectory of the dynamic obstacle relative to the camera device. Secondly, the upper left corner of the image obtained by the camera device is defined as the origin of the pixel uv coordinate system, the horizontal right is the u axis, and the vertical downward is the v axis; and the object principal point (u0, v0) is also the x-axis and y-axis in the same direction as the u axis and v axis, respectively, and the internal parameters of the camera device are known. The above three are used together to convert the two-dimensional coordinates of the pixels of the dynamic obstacle to the three-dimensional coordinates of the camera device. The first formula is satisfied for the conversion of the uv coordinate system and the xy coordinate system. The first formula is:

[0147]

[0148] According to the central projection feature, the coordinate relationship of the pixel points satisfies the proportional relationship, so the camera coordinate system and the xy coordinate system satisfy the second formula, which is:

[0149]

[0150] Combining the first formula and the second formula, the conversion relationship formula between the two-dimensional coordinates of the pixel and the three-dimensional coordinates of the camera device can be obtained as follows:

[0151]

[0152] Where (u, v) is the center coordinate of a dynamic obstacle in a two-dimensional coordinate system, and is the optimal estimated coordinate value (such as the optimal estimated position information) of the dynamic obstacle in the current frame obtained by Kalman filtering; (u 0 , v 0 ) is the coordinate of the image principal point; f is the focal length of the camera; d x ,d y Represents the physical size of the pixel in the horizontal and vertical directions; (X C , Y C , Z C ) is the coordinate of the dynamic obstacle in the three-dimensional coordinate system, where Z C is the depth data, that is, the distance from the camera. It can be understood as follows: after obtaining the RGB image, (u, v) is the two-dimensional coordinate of a dynamic obstacle in the RGB image, and then according to the depth data and the focal length f of the camera, the three-dimensional coordinate of the dynamic obstacle (X C , YC , Z C ).

[0153] In an embodiment of the present application, in the process of obtaining the dynamic tracking information of each dynamic obstacle, the updated optimal estimated position information is first obtained and recorded as the latest state information, and then the two adjacent updated state information and the optimal estimated position information are matched for data association through IOU similarity and Re-ID re-identification. It can be understood that the optimal estimated position information of the current frame of the dynamic obstacle after the update is matched with the optimal estimated position information of the previous frame of the dynamic obstacle to obtain the dynamic tracking information of the corresponding dynamic obstacle.

[0154] It should be noted that the calculation of IOU similarity is to calculate the overlapping area of ​​the detection frame of the dynamic obstacle in the previous frame and the current frame. The similarity calculation formula is:

[0155]

[0156] In the formula, area is the area, B is p , B gt The detection box area of ​​the previous frame and the detection box area of ​​the current frame are defined as d and y, respectively, and the measured distribution and predicted distribution of the dynamic obstacle are defined as S, and the Mahalanobis distance deviation between the measured distribution and the predicted distribution is calculated as d (1) (i, j), the formula for calculating the Mahalanobis distance deviation is:

[0157]

[0158] Where i is the i-th dynamic obstacle and j is the j-th detection box. At the same time, a deep appearance feature descriptor is designed for each dynamic obstacle, which is a 128-dimensional unit feature vector of the Re-ID network trained offline on the pedestrian re-identification dataset. For each tracker, retain its last 100 appearance feature descriptor sets R that are successfully associated with the detection box and calculate their minimum cosine distance formula with the detection box:

[0159]

[0160] In the formula, r k 、r j are the feature vectors of the k-th track and the j-th detection box, respectively, used to represent the deep appearance; R i It is a collection of appearances that stores the feature vector of the successfully tracked i-th track. In order to establish matching association, the weighted sum is used to combine the two indicators to obtain the association influence coefficient. The association influence coefficient can be controlled by hyperparameters to control the influence of each indicator on the matching. The formula for calculating the association influence coefficient is:

[0161] ci，j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0162] Where, d (1) (i, j) and d (2) (i, j) is the Mahalanobis distance and cosine distance between the i-th track and the j-th detection box, which have been calculated above. λ is a hyperparameter in [0, 1] and can be set or adjusted according to needs. i,j is the weighted sum of the two distances.

[0163] In the embodiment of the present application, the running trajectory information of the dynamic obstacle is obtained, including three-dimensional Kalman filtering prediction based on dynamic tracking information, predicting the position of the dynamic obstacle in the next few frames, and connecting each point as the motion trajectory information of the dynamic obstacle.

[0164] It should be noted that according to the optimal three-dimensional information of dynamic obstacles (X C ,Y C ,Z C ), the current frame is recorded as the tth frame, and the state information of the tth frame is the optimal three-dimensional information. The optimal three-dimensional information of each dynamic obstacle follows a Gaussian distribution, and the variance is recorded as P k , the state transfer matrix of the optimal three-dimensional information between two frames is recorded as F k , the error is considered as Gaussian distribution w k ~N(w k , Q k ), then the state prediction equation is:

[0165]

[0166]

[0167] Where P k-1 is the covariance of the dynamic obstacle distribution in the previous frame. k is the state of the dynamic obstacle in the current frame, which includes the three-dimensional coordinates p and the moving speed v in the direction of the three-dimensional coordinate axis. Assume that the current state of the dynamic obstacle has a specific relationship with the sensor data H k , then the prediction result of frame t+1 is: In the formula, the mean of the Gaussian distribution of the dynamic obstacle predicted in the next frame is μ 0 , the covariance is ∑ 0 The prediction result of frame t+2 is used as the pseudo optimal prediction of frame t+1. Similarly, the motion trajectory of the next few frames can be predicted by using the three-dimensional Kalman filter, so as to obtain several predicted trajectory points corresponding to each dynamic obstacle. All predicted trajectory points are connected to obtain the motion trajectory information corresponding to the dynamic obstacle.

[0168] In one embodiment of the present application, the global path planning algorithm and the local path planning algorithm are used to perform motion planning data of the mobile device on the posture information and the occupancy grid map, and the obtained motion planning data includes:

[0169] Obtaining the current location of the mobile device in the occupancy grid map, the destination location, and the road network information and obstacle information between the current location and the destination location;

[0170] Taking the current location as the starting point and the end point as the end point, the Astar algorithm is used as the global path planning algorithm to search for paths in the occupancy grid map according to the road network information and obstacle information, and the best running path for the mobile device is obtained;

[0171] The best running path is optimized using Bezier curve to obtain the best optimized path;

[0172] Discretize the best optimization path to obtain discrete points; connect all discrete points to form a preliminary elastic band;

[0173] The TEB algorithm is used to optimize the preliminary elastic band, and the optimized elastic band is obtained and used as the mobile planning data of the mobile device;

[0174] Among them, an updated RGB map and an updated depth map of the environment in which the mobile device is located after moving are obtained in real time; the occupancy grid map is updated according to the updated RGB map and the updated depth map; the best optimization path is updated according to the updated occupancy grid map, and then the mobile planning data of the mobile device is updated.

[0175] It should be noted that global path planning for mobile devices is a complex process that requires comprehensive consideration of map data, path planning algorithms and vehicle dynamics. Factors that need to be considered when selecting a path planning algorithm for mobile planning data of a mobile device include: ensuring that the path does not intersect with obstacles to avoid collisions; optimizing the path length as much as possible to reduce the driving distance; optimizing the path to minimize the energy consumption of the vehicle. Taking these factors into consideration, the Astar algorithm is selected for path search. In this embodiment, the content of using the Astar algorithm for global path planning includes: obtaining an occupancy grid map as map data for path planning, and the map data contains the current location of the mobile device, the terminal location, road network information and obstacle information, etc. The current location of the mobile device is set as the starting point of the path, and the path is searched in the acquired occupancy grid map; a heuristic function (valuation function) h(n) is defined, and the heuristic function is used to estimate the cost from node n to the target node; a heuristic function is defined using Manhattan distance to measure the sum of the distances from the current node to the target node in the horizontal and vertical directions; an open list is created to store the nodes to be explored, and a closed list is created. List) to store the nodes that have been explored; add the starting point to the open list and set its estimated cost (f1 value) to 0; in each iteration, select the node with the smallest f1 value from the open list; move the selected node to the closed list to indicate that it has been explored; calculate the f1 values ​​of the adjacent nodes of the selected node and add them to the open list if they are not in the closed list; update the parent node information to record the best path to these nodes; check whether the target node has been found each time a node is added to the open list; if the target node is found, it means that the path has been found and the search can be ended; after finding the target node, backtrack the parent node information from the target node to the starting point to obtain the complete best running path; optimize the searched best running path, use Bezier curves to smooth the path, reduce sharp turns and acute angles in the path, improve the quality of the path, reduce energy consumption, etc.; remove redundant nodes by merging nodes, reduce the path length, and obtain the best optimized path. Among them, the road network information is obtained by the lane line detection neural network and the traversable area detection neural network, and the obstacle information is obtained by the occupancy grid map to obtain the motion trajectory information of dynamic obstacles.

[0176] In an embodiment of the present application, a path is searched cyclically in an occupancy grid map that is updated in real time. During the movement of the device, it is determined whether the 20 nodes before the current position of the mobile device are passable. If passable, there is no need to search the path again and the previous path can be used, thereby reducing the path planning time.

[0177] Local path planning is the connection part from the global path to the actual chassis control. It obtains the best optimized path released by the global path planning, and outputs the speed and angular velocity information to the chassis after processing. That is, it is responsible for following the global path points to generate a specific, feasible path that avoids obstacles, and for real-time obstacle avoidance when the global path cannot keep up. In the embodiment of the present application, the local path planning algorithm adopted is the TEB (Timed Elastic Band) algorithm. The global path is discretized into a series of discrete points, and then these points are connected to form a preliminary elastic band; the elastic band is adjusted by the TEB algorithm to achieve optimization, and the optimization goals include minimizing the length of the elastic band, minimizing the steering angle of the robot, and maximizing the distance between the elastic band and obstacles; the optimized elastic band is used as the mobile planning data of the mobile device, and the movement of the mobile device is controlled according to this mobile planning data.

[0178] It should be noted that TEB (Timed Elastic Band) is a method for local path planning of robots. The main idea of ​​the TEB algorithm is to regard the robot's path as an elastic band and adjust this elastic band through an optimization algorithm so that it can avoid obstacles and be as close to the global path as possible.

[0179] In the embodiment of the present application, the expression of the time elastic band is: Where t is time, x(t) is the state of the mobile device at time t, is an n-dimensional Euclidean space, l(t) and u(t) are the lower and upper bounds of the time elastic band at time t, respectively. The time elastic band at time t is used to describe the dynamic constraints of the mobile device during movement, ensuring that the movement state of the mobile device is within a certain range. The obstacle cost function expression is:

[0180]

[0181] Where x is the state of the mobile device, m obs is the number of dynamic obstacles, d i (x) is the distance between the mobile device and the i-th dynamic obstacle, and ρ(·) is the obstacle cost function. The obstacle cost function is used to describe the obstacle avoidance constraint between the mobile device and the dynamic obstacle, ensuring that the motion state of the mobile device will not collide with the dynamic obstacle. The expression of the path cost function is:

[0182]

[0183] Where α and β are the weights of the obstacle cost function and the control cost function, respectively, u(t) is the control input of the mobile device at time t 0 and t fare the start time and end time of the path respectively. The steering angle can be achieved by minimizing the steering angle between adjacent path segments. The expression of the steering angle is:

[0184]

[0185] Where angle(a, b) is the angle between vectors a and b.

[0186] In one embodiment of the present application, obtaining the posture information of the mobile device includes:

[0187] Obtain the grayscale images of the previous and next two frames of the environment where the mobile device is located and the initial posture information;

[0188] Each frame of grayscale image is processed using the FAST feature point detection algorithm and the Harris corner point detection algorithm to obtain the feature point data corresponding to each frame of grayscale image;

[0189] Describe each feature point in the feature point data to obtain a feature point descriptor corresponding to each feature point; use the Hamming distance to match all feature point descriptors of the previous and next two frames to obtain a feature point pair set;

[0190] The EPnP algorithm is used to process each pair of feature points in the feature point set to obtain the relative pose information between the previous and next two frames of the mobile device corresponding to the feature point; the initial pose information is superimposed with all relative pose information to obtain the current pose information of the mobile device.

[0191] It should be noted that the initial posture information of the mobile device is obtained through GNSS, gyroscope and other devices. Assuming that the mobile device is a rigid body, the posture of the camera device is the posture of the mobile device. First, the RB image is converted into a grayscale image; then, the pixel points of the grayscale image are preliminarily selected by the FAST feature point detection algorithm to obtain preliminary feature points; the preliminary feature points are further screened by the Harris corner point detection algorithm to obtain feature point data; the detected feature point data is described by the BRIEF feature point description algorithm to obtain a feature point descriptor; the feature points of the front and back frames of the camera device are matched according to the Hamming distance of the feature point descriptor to obtain a feature point pair set (such as matching the two feature points with the smallest calculated Hamming distance into a pair of feature point pairs); the EPnP algorithm is used to obtain the relative posture information of the mobile device between the front and back frames through the matched feature point pairs, and then all relative posture information is superimposed according to the initial posture information of the mobile device to obtain the current posture information of the mobile device. Among them, the technical content of calculating the Han-Chinese distance between two feature points and matching through the Han-Chinese distance is a relatively mature technology in this field and will not be elaborated here.

[0192] In the embodiment of the present application, the content of the FAST feature point detection algorithm includes: traversing each pixel in the grayscale image as the central pixel; calculating the pixel difference between the central pixel and the pixels within a set distance range from the central pixel, if the pixel difference exceeds the set pixel difference threshold, the central pixel is considered to be a feature point. The grayscale image can also be suppressed to a non-large value, so that there is a pixel with the highest score within the set range, and this pixel is used as a feature point.

[0193] It should be noted that the pixel difference threshold and the set distance can be set according to the requirements and are not specifically limited here. The formula for calculating the feature point score is:

[0194]

[0195] Where c is the pixel difference threshold, V is the score, p is the pixel value of the center pixel, and p' is the pixel value of other pixels within the set range.

[0196] In the embodiment of the present application, the content of the FAST feature point detection algorithm also includes: using the main direction calculation formula to calculate the main direction of the feature point, the main direction calculation formula is:

[0197]

[0198]

[0199] θ=arctan(m 01 / m 10 )

[0200] Where B is the setting range of the feature point, m pq is the moment of the grayscale image block, I(x, y) is the pixel value at the coordinate (x, y) position in the grayscale image, C is the centroid of the feature area, and θ is the main direction of the feature point.

[0201] In the embodiment of the present application, the content of the Harris corner point detection algorithm includes: using the corner point response value calculation formula to calculate the corner point response value of each feature point; sorting all the corner point response values ​​from large to small according to the numerical value, and selecting feature points that match the set number from large to small from the sorted corner point response values ​​as feature point data.

[0202] It should be noted that the number of settings can be set according to the needs and is not specifically limited here. The corner point response value calculation formula is:

[0203]

[0204] R 0 =detM-α(traceM) 2

[0205] Where w(x, y) is the window coefficient, I x is the grayscale gradient of the feature point in the X-axis direction, I y is the grayscale gradient of the feature point in the Y-axis direction, det M is the determinant of matrix M, traceM is the trace of matrix M, α is the empirical coefficient, R 0 is the corner point response value.

[0206] In the embodiment of the present application, the BRIEF algorithm is used to describe the features of each feature point in the feature point data, and the feature point descriptor corresponding to each feature point is obtained. Specifically, the feature point is taken as the center, and the pixel points in the set range are used as the domain window; according to the main direction θ of the feature point obtained by the FAST feature point detection algorithm, the selected domain window is rotated by an angle θ; in the rotated domain window, 256 pairs of pixel points are selected in a certain pattern, and the feature points are assigned binary values ​​according to the assignment formula to form a binary code string as the feature descriptor of the feature point, and the assignment formula is:

[0207]

[0208]

[0209] g n (p,θ):=f n (p)|(x i ,y i )∈S θ

[0210] In the formula, x and y are a pair of pixels selected in the domain window, p(x) and p(y) are the pixel values ​​of the corresponding pixels, τ(p; x, y) is a binary code value, and f n (p) is a binary code string, g n (p, θ) is the feature point descriptor of the feature point. All feature points are traversed and the BRIEF algorithm is used to obtain the feature point descriptors of all feature points.

[0211] In one embodiment of the present application, the EPnP algorithm is used to process each pair of feature points in the feature point pair set, and the relative pose information between the previous and next frames of the mobile device corresponding to the feature point is obtained, including:

[0212] Obtain the number of feature point pairs and all feature points corresponding to the grayscale image of the previous frame and the grayscale image of the next frame from the feature point pair set and record them as the first key feature point and the second key feature point, respectively, and construct the coordinate system of the previous frame and the coordinate system of the next frame according to the grayscale images of the previous and next frames;

[0213] Obtain the three-dimensional coordinates of each first key feature point and second key feature point in the front frame coordinate system and the rear frame coordinate system and record them as the first three-dimensional coordinates and the second three-dimensional coordinates respectively; obtain the first control three-dimensional coordinates of the first front frame control point according to all the first three-dimensional coordinates; obtain the second control three-dimensional coordinates of the first rear frame control point according to all the second three-dimensional coordinates;

[0214] According to all the first three-dimensional coordinates and the three-dimensional coordinates of the first front frame control point, a third control three-dimensional coordinate of the front frame control point corresponding to the three main directions of the first front frame control point is calculated; according to all the second three-dimensional coordinates and the three-dimensional coordinates of the first rear frame control point, a fourth control three-dimensional coordinate of the rear frame control point corresponding to the three main directions of the first rear frame control point is calculated;

[0215] Obtain the front frame conversion coefficient and the rear frame conversion coefficient of each front frame control point and the rear frame control point; calculate according to all the front frame conversion coefficients, the first control three-dimensional coordinates and all the third control three-dimensional coordinates to obtain the first three-dimensional coordinate representative data corresponding to all the first key feature points of the front frame grayscale image; calculate according to all the rear frame conversion coefficients, the second control three-dimensional coordinates and all the fourth control three-dimensional coordinates to obtain the second three-dimensional coordinate representative data corresponding to all the second key feature points of the rear frame grayscale image;

[0216] Iterative optimization is performed using a singular value decomposition method according to the number of feature point pairs, the first three-dimensional coordinate representative data, and the second three-dimensional coordinate representative data to obtain a rotation matrix and a translation matrix with the minimum error value;

[0217] The pose information includes a rotation matrix and a translation matrix.

[0218] It should be noted that, according to the key feature points that are successfully matched, the mean center feature point of the grayscale image is obtained by the mean calculation formula, and the mean center feature point is set as the first control point, and then the three control points corresponding to the three main directions of the feature point are obtained by the control point calculation formula. For example, the mean calculation formula for calculating the control points of the previous frame is:

[0219]

[0220] The control point calculation formula for calculating the three control points corresponding to the three main directions of the feature points in the grayscale image of the previous frame is:

[0221]

[0222]

[0223]

[0224]

[0225] In the formula, is the three-dimensional coordinate of the mean center feature point in the previous frame coordinate system, is the three-dimensional coordinate of the feature point in the previous frame camera coordinate system, A is the 3×n matrix formed by subtracting the coordinates of the feature point from the mean center feature point coordinates, is the three-dimensional coordinate of the previous frame control point in the previous frame coordinate system, is the matrix A T The eigenvalues ​​of A, is the eigenvector corresponding to the eigenvalue. The first three-dimensional coordinate representative data or the second three-dimensional coordinate representative data is calculated using the representative feature point coordinate formula. The representative feature point coordinate formula is:

[0226]

[0227]

[0228] In the formula, is the three-dimensional coordinate of the i-th feature point in the previous frame coordinate system, is the three-dimensional coordinate of the jth previous frame control point in the previous frame coordinate system, α ij is the previous frame conversion coefficient between the ith feature point and the jth previous frame control point; is the three-dimensional coordinate of the i-th feature point in the post-frame coordinate system, is the three-dimensional coordinate of the jth post-frame control point in the post-frame coordinate system, α' ij is the post-frame conversion coefficient between the i-th feature point and the j-th post-frame control point. The optimized loss function is iteratively optimized to minimize the error function value. After the optimization, the rotation matrix and translation matrix of the post-frame coordinate system relative to the front-frame coordinate system (i.e., the position information of the camera device in the front-frame coordinate system) are obtained. The optimized loss function is:

[0229]

[0230] Where H is the rotation matrix of the rear frame coordinate system relative to the front frame coordinate system, L is the translation matrix of the rear frame coordinate system relative to the front frame coordinate system, and k is the number of feature point pairs.

[0231] Embodiment 2:

[0232] Figure 2 This is a schematic diagram of the framework of the wheelchair intelligent control device based on neural network optimization described in an embodiment of the present application.

[0233] like Figure 2 As shown, the embodiment of the present application provides a wheelchair intelligent control device based on neural network optimization, including a data acquisition module 10, a data processing module 20, a path planning module 30 and a control execution module 40;

[0234] The data acquisition module 10 is used to acquire the RGB image and depth image of the environment in which the mobile device is located, and to acquire the position information of the mobile device;

[0235] The data processing module 20 is used to process the RGB image and the depth image using a lane detection, a passable area detection and a trajectory prediction neural network to obtain an occupancy grid image;

[0236] A path planning module 30 is used to use a global path planning algorithm and a local path planning algorithm to perform motion planning data of the mobile device on the posture information and the occupancy grid map;

[0237] The control execution module 40 is used to control the operation of the mobile device according to the mobile planning data.

[0238] In the embodiment of the present application, the data processing module includes a lane detection submodule, a traffic area detection submodule, an obstacle detection submodule and a grid map generation submodule;

[0239] The lane detection submodule is used to perform lane detection on the RGB image using a lane line detection neural network to obtain lane information;

[0240] The passable area detection submodule is used to perform area detection on the RGB image and the depth image using a passable area detection neural network to obtain a passable area map;

[0241] The obstacle detection submodule is used to detect dynamic obstacles on the RGB image and depth image using a trajectory prediction neural network to obtain the motion trajectory information of dynamic obstacles;

[0242] The grid map generation submodule is used to construct an initial occupancy grid map according to the RGB map and the depth map, and update the initial occupancy grid map according to the lane information, the passable area map and the motion trajectory information to obtain the occupancy grid map;

[0243] The contents of the lane detection neural network include: setting the lane type and setting a mapping key point on each lane type; introducing a hybrid anchor mechanism and a detection branch in the fully connected layer of the lane detection neural network, anchoring the lane line of the RGB image through the hybrid anchor mechanism, and obtaining the key points corresponding to each lane line; detecting all the key points in the RGB image through the detection branch to determine whether there is a lane line, comparing the key points corresponding to the detected lane line with the mapping key points of each lane type, and obtaining the lane type of the lane line; constructing lane information corresponding to RGB by detecting all lane lines and their corresponding lane types;

[0244] The contents of the passable area detection neural network include: obtaining the normal information of each pixel from the depth map; using the RGB encoder and surface normal encoder of the deep neural network to extract the feature map of each pixel from the RGB map and each normal information respectively, and obtain the RGB feature map and surface normal feature map corresponding to each pixel; using the fusion device of the deep neural network to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel element by element, and obtain the fused feature map corresponding to each pixel; using the Sigmoid layer of the deep neural network to transform the fused feature map to obtain a probability map; obtaining probability data from the probability map, and determining whether the area corresponding to the probability map is marked as a passable area based on the comparison of the probability data with the set probability threshold, and obtaining a passable area map;

[0245] The content of the trajectory prediction neural network includes: constructing a two-dimensional coordinate system and a three-dimensional coordinate system based on the RGB image, using the recognition model of the deep neural network to identify dynamic obstacles in the RGB image, and obtaining the state detection value corresponding to each dynamic obstacle in the environment where the mobile device is located; using the Kalman filter estimation to estimate the state detection value, and obtain the state information prediction value of the corresponding dynamic obstacle; using the Kalman filter algorithm to estimate the position of the corresponding dynamic obstacle based on the state information prediction value and the state detection value, and obtain the optimal estimated position information corresponding to the dynamic obstacle; obtaining the depth data of each dynamic obstacle from the depth map, and calculating the optimal estimated position information of each dynamic obstacle in the three-dimensional coordinate system according to the depth data of each dynamic obstacle. The corresponding obstacle model is constructed in the proposed method, and the optimal three-dimensional information of each dynamic obstacle is obtained in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information; the updated optimal estimated position information of each dynamic obstacle is obtained in real time and recorded as the latest state information; the optimal estimated position information and the latest state information of two adjacent objects are associated and matched by IOU similarity and Re-ID re-identification to obtain the dynamic tracking information of each dynamic obstacle; the dynamic tracking information of each dynamic obstacle is predicted by three-dimensional Kalman filtering to obtain several predicted trajectory points corresponding to each dynamic obstacle, and all the predicted trajectory points are connected to obtain the motion trajectory information corresponding to the dynamic obstacle.

[0246] It should be noted that the content of the module in the device of Example 2 corresponds to the step content of the method in Example 1. The content of the wheelchair intelligent control method based on neural network optimization has been described in Example 1, and the module content of the wheelchair intelligent control device based on neural network optimization will not be described in detail in this embodiment.

[0247] Embodiment three:

[0248] Figure 3 A schematic diagram of a terminal device described in an embodiment of the present application.

[0249] like Figure 3 As shown, an embodiment of the present application provides a terminal device, including a processor and a memory;

[0250] A memory, used for storing program codes and transmitting the program codes to a processor;

[0251] The processor is used to execute the above-mentioned wheelchair intelligent control method based on neural network optimization according to the instructions in the program code.

[0252] It should be noted that the processor is used to execute the steps in the above-mentioned embodiment of the wheelchair intelligent control method based on neural network optimization according to the instructions in the program code. Alternatively, the processor implements the functions of each module / unit in the above-mentioned system / device embodiments when executing the computer program.

[0253] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0254] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that this does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0255] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0256] The memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0257] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0262] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A wheelchair intelligent control method based on neural network optimization, characterized in that: The following steps are involved: Obtaining an RGB image and a depth image of the environment in which the mobile device is located, and obtaining position information of the mobile device; The RGB image and the depth image are processed by lane detection, drivable area detection and trajectory prediction neural network to obtain an occupancy grid image; Using a global path planning algorithm and a local path planning algorithm on the posture information and the occupancy grid map to obtain movement planning data of the mobile device; Controlling the operation of the mobile device according to the mobile planning data; The RGB image and the depth image are processed by lane line detection, passable area detection and trajectory prediction neural network to obtain an occupied grid image including: Using a lane detection neural network to perform lane detection on the RGB image to obtain lane information; Using a traversable area detection neural network to perform area detection on the RGB image and the depth image to obtain a traversable area map; Using a trajectory prediction neural network to perform dynamic obstacle detection on the RGB image and the depth image to obtain motion trajectory information of the dynamic obstacles; Constructing an initial occupancy grid map according to the RGB map and the depth map, and updating the initial occupancy grid map according to the lane information, the traversable area map and the motion trajectory information to obtain an occupancy grid map; The contents of the lane detection neural network include: Setting lane types and setting a mapping keypoint on each of said lane types; Introducing a hybrid anchor point mechanism and a detection branch into the fully connected layer of the lane line detection neural network, anchoring the lane lines of the RGB image through the hybrid anchor point mechanism, and obtaining key points corresponding to each lane line; Detecting all the key points in the RGB image by the detection branch to determine whether there is a lane line, and comparing the key points corresponding to the detected lane line with the mapped key points of each lane type to obtain the lane type of the lane line; Detecting all the lane lines and the corresponding lane types to construct the lane information corresponding to the RGB; The hybrid anchor point mechanism refers to that the lane line with a large horizontal angle in the middle is described by a lateral anchor point, and the lane lines with small horizontal angles on both sides are described by a longitudinal anchor point.

2. The wheelchair intelligent control method based on neural network optimization according to claim 1 is characterized in that: The content of the traversable area detection neural network includes: Obtaining normal information of each pixel from the depth map; Using a deep neural network RGB encoder and a surface normal encoder to extract a feature map of each pixel from the RGB image and each piece of normal information, respectively, to obtain an RGB feature map and a surface normal feature map corresponding to each pixel; Using the fusion device of the deep neural network to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel point element by element, so as to obtain a fused feature map corresponding to each pixel point; The fused feature map is converted using the Sigmoid layer of the deep neural network to obtain a probability map; Probability data is obtained from the probability map, and whether an area corresponding to the probability map is marked as a passable area is determined based on the comparison between the probability data and a set probability threshold, so as to obtain a passable area map.

3. The wheelchair intelligent control method based on neural network optimization according to claim 1 is characterized in that: The content of the trajectory prediction neural network includes: Constructing a two-dimensional coordinate system and a three-dimensional coordinate system according to the RGB image, and using a recognition model of a deep neural network to perform dynamic obstacle recognition on the RGB image to obtain a state detection value corresponding to each dynamic obstacle in the environment where the mobile device is located; The state detection value is estimated by using a Kalman filter to obtain a state information prediction value corresponding to the dynamic obstacle; a position of the dynamic obstacle is estimated by using a Kalman filter algorithm according to the state information prediction value and the state detection value to obtain an optimal estimated position information corresponding to the dynamic obstacle; Acquire depth data of each of the dynamic obstacles from the depth map, construct a corresponding obstacle model in the three-dimensional coordinate system according to the depth data of each of the dynamic obstacles, and acquire optimal three-dimensional information of each of the dynamic obstacles in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information; Acquire updated optimal estimated position information of each of the dynamic obstacles in real time and record it as the latest status information; The optimal estimated position information and the latest state information of two adjacent objects are associated and matched using IOU similarity and Re-ID re-identification to obtain dynamic tracking information of each dynamic obstacle; The dynamic tracking information of each dynamic obstacle is predicted by using a three-dimensional Kalman filter to obtain a plurality of predicted trajectory points corresponding to each dynamic obstacle, and all the predicted trajectory points are connected to obtain the motion trajectory information corresponding to the dynamic obstacle.

4. The wheelchair intelligent control method based on neural network optimization according to claim 1 is characterized in that: The position information and the occupancy grid map are processed using a global path planning algorithm and a local path planning algorithm to obtain movement planning data of the mobile device, including: Acquire the current location and the end location of the mobile device in the occupancy grid map, as well as the road network information and obstacle information between the current location and the end location; Taking the current location as the starting point and the end location as the end point, performing a path search in the occupancy grid map using the A star algorithm as a global path planning algorithm according to the road network information and the obstacle information, and obtaining an optimal running path for the mobile device; The optimal running path is optimized using a Bezier curve to obtain an optimal optimization path; Discretize the optimal optimization path to obtain discrete points; connect all the discrete points to form a preliminary elastic band; The preliminary elastic band is optimized by using the TEB algorithm to obtain an optimized elastic band and the optimized elastic band is used as the movement planning data of the mobile device.

5. The wheelchair intelligent control method based on neural network optimization according to claim 4 is characterized in that: include: Acquire in real time an updated RGB image and an updated depth image of the environment in which the mobile device is located after movement; The occupancy grid map is updated according to the updated RGB map and the updated depth map; the best optimization path is updated according to the updated occupancy grid map, and then the movement planning data of the mobile device is updated.

6. The wheelchair intelligent control method based on neural network optimization according to claim 1 is characterized in that: The content of obtaining the position information of the mobile device includes: Obtaining grayscale images of the previous and next two frames of the environment in which the mobile device is located and initial posture information; The grayscale image of each frame is processed using the FAST feature point detection algorithm and the Harris corner point detection algorithm to obtain feature point data corresponding to the grayscale image of each frame; Performing feature description on each feature point in the feature point data to obtain a feature point descriptor corresponding to each feature point; performing feature point matching on all the feature point descriptors of the previous and next two frames using the Hamming distance to obtain a feature point pair set; The EPnP algorithm is used to process each pair of feature points in the feature point set to obtain relative posture information between the previous and next two frames of the mobile device corresponding to the feature point; the initial posture information is superimposed with all the relative posture information to obtain the current posture information of the mobile device.

7. The wheelchair intelligent control method based on neural network optimization according to claim 6 is characterized in that: The EPnP algorithm is used to process each pair of feature points in the feature point pair set, and the relative posture information between the previous and next frames of the mobile device corresponding to the feature point is obtained, including: Acquire the number of feature point pairs and all feature points corresponding to the grayscale image of the previous frame and the grayscale image of the next frame from the feature point pair set and record them as the first key feature point and the second key feature point respectively, and construct the coordinate system of the previous frame and the coordinate system of the next frame according to the grayscale images of the previous and next frames; Obtaining the three-dimensional coordinates of each of the first key feature point and the second key feature point in the front frame coordinate system and the rear frame coordinate system and recording them as first three-dimensional coordinates and second three-dimensional coordinates respectively; obtaining the first control three-dimensional coordinates of the first front frame control point according to all the first three-dimensional coordinates; obtaining the second control three-dimensional coordinates of the first rear frame control point according to all the second three-dimensional coordinates; According to all the first three-dimensional coordinates and the three-dimensional coordinates of the first front frame control point, a third control three-dimensional coordinate of the front frame control point corresponding to the three main directions of the first front frame control point is calculated; according to all the second three-dimensional coordinates and the three-dimensional coordinates of the first rear frame control point, a fourth control three-dimensional coordinate of the rear frame control point corresponding to the three main directions of the first rear frame control point is calculated; Obtaining a front frame conversion coefficient and a rear frame conversion coefficient of each front frame control point and rear frame control point; calculating based on all the front frame conversion coefficients, the first control three-dimensional coordinates and all the third control three-dimensional coordinates, obtaining first three-dimensional coordinate representative data corresponding to all the first key feature points of the grayscale image of the front frame; calculating based on all the rear frame conversion coefficients, the second control three-dimensional coordinates and all the fourth control three-dimensional coordinates, obtaining second three-dimensional coordinate representative data corresponding to all the second key feature points of the grayscale image of the rear frame; Iterative optimization is performed using a singular value decomposition method according to the number of feature point pairs, the first three-dimensional coordinate representative data, and the second three-dimensional coordinate representative data to obtain a rotation matrix and a translation matrix with a minimum error value; The position information includes a rotation matrix and a translation matrix.

8. A wheelchair intelligent control device based on neural network optimization, characterized in that: It includes a data acquisition module, a data processing module, a path planning module and a control execution module; The data acquisition module is used to acquire an RGB image and a depth image of the environment in which the mobile device is located, and to acquire position information of the mobile device; The data processing module is used to process the RGB image and the depth image using lane line detection, passable area detection and trajectory prediction neural network to obtain an occupancy grid image; The path planning module is used to use a global path planning algorithm and a local path planning algorithm to perform motion planning data of the mobile device on the posture information and the occupancy grid map; The control execution module is used to control the operation of the mobile device according to the mobile planning data; The data processing module includes a lane detection submodule, a traffic area detection submodule, an obstacle detection submodule and a grid map generation submodule; The lane detection submodule is used to perform lane detection on the RGB image using a lane line detection neural network to obtain lane information; The passable area detection submodule is used to perform area detection on the RGB image and the depth image using a passable area detection neural network to obtain a passable area map; The obstacle detection submodule is used to perform dynamic obstacle detection on the RGB image and the depth image using a trajectory prediction neural network to obtain motion trajectory information of the dynamic obstacles; The grid map generation submodule is used to construct an initial occupancy grid map according to the RGB map and the depth map, and update the initial occupancy grid map according to the lane information, the passable area map and the motion trajectory information to obtain an occupancy grid map; Among them, the content of the lane line detection neural network includes: setting the lane type and setting a mapping key point on each lane type; introducing a hybrid anchor point mechanism and a detection branch in the fully connected layer of the lane line detection neural network, anchoring the lane line of the RGB image through the hybrid anchor point mechanism, and obtaining the key point corresponding to each lane line; detecting all the key points in the RGB image through the detection branch to determine whether there is a lane line, comparing the key point corresponding to the detected lane line with the mapping key point of each lane type, and obtaining the lane type of the lane line; constructing the lane information corresponding to the RGB by detecting all the lane lines and their corresponding lane types; the hybrid anchor point mechanism refers to that the lane line with a large horizontal angle in the middle is described by a horizontal anchor point, and the lane lines with small horizontal angles on both sides are described by a longitudinal anchor point.

9. The wheelchair intelligent control device based on neural network optimization according to claim 8, characterized in that: The content of the passable area detection neural network includes: obtaining normal information of each pixel from the depth map; using the RGB encoder and the surface normal encoder of the deep neural network to extract the feature map of each pixel from the RGB map and each normal information respectively, to obtain the RGB feature map and the surface normal feature map corresponding to each pixel; using the fusion device of the deep neural network to add the corresponding channel values ​​of the RGB feature map and the surface normal feature map in each pixel element by element, to obtain the fused feature map corresponding to each pixel; using the Sigmoid layer of the deep neural network to transform the fused feature map to obtain a probability map; obtaining probability data from the probability map, and determining whether the area corresponding to the probability map is marked as a passable area according to the comparison between the probability data and the set probability threshold, to obtain a passable area map; The content of the trajectory prediction neural network includes: constructing a two-dimensional coordinate system and a three-dimensional coordinate system according to the RGB image, using a recognition model of a deep neural network to identify dynamic obstacles on the RGB image, and obtaining state detection values ​​corresponding to each dynamic obstacle in the environment where the mobile device is located; using Kalman filter estimation to estimate the state detection value, and obtaining a state information prediction value corresponding to the dynamic obstacle; using a Kalman filter algorithm to estimate the position of the dynamic obstacle according to the state information prediction value and the state detection value, and obtaining the optimal estimated position information corresponding to the dynamic obstacle; obtaining the depth data of each dynamic obstacle from the depth map, and calculating the optimal estimated position information of each dynamic obstacle in the three-dimensional coordinate system according to the depth data of each dynamic obstacle. The method comprises the following steps: constructing a corresponding obstacle model in the coordinate system, and obtaining the optimal three-dimensional information of each of the dynamic obstacles in the three-dimensional coordinate system according to the obstacle model and the optimal estimated position information; obtaining the updated optimal estimated position information of each of the dynamic obstacles in real time and recording it as the latest state information; performing association matching on all two adjacent optimal estimated position information and the latest state information by using IOU similarity and Re-ID re-identification to obtain the dynamic tracking information of each of the dynamic obstacles; predicting the dynamic tracking information of each of the dynamic obstacles by using a three-dimensional Kalman filter to obtain a plurality of predicted trajectory points corresponding to each of the dynamic obstacles, and connecting all the predicted trajectory points to obtain the motion trajectory information corresponding to the dynamic obstacle.

10. A terminal device, characterized in that: including a processor and a memory; The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the wheelchair intelligent control method based on neural network optimization as described in any one of claims 1 to 7 according to the instructions in the program code.

Images