Blind guiding method and device, electronic equipment and storage medium

Abstract

This invention discloses a method, device, electronic device, and storage medium for guiding blind people. The method includes acquiring three-dimensional point cloud data, image data, and pose data of the current environment in which the guiding robot is located; determining target detection information of a target object in the current environment based on the three-dimensional point cloud data and image data, including the distance and direction of the target object relative to the guiding robot, and the type identification result of the target object; performing point cloud registration and pose optimization processing on the three-dimensional point cloud data and pose data to obtain the optimized target position and target pose at the current moment; thereby enabling path planning for the guiding robot at the current moment based on the target detection information, type identification result, target position, and target pose, so as to control the guiding robot to move according to the planned navigation data, complete the real-time movement of the guiding robot in complex environments, and improve the travel efficiency of blind people.

Description

Technical Field

[0001] This invention relates to the field of positioning and navigation technology, and in particular to a method, device, electronic device and storage medium for guiding the blind. Background Technology

[0002] Blind people face numerous difficulties in getting around due to their visual impairment. Although accessible facilities such as tactile paving have been built in cities, problems such as obstruction, discontinuity, and poor design of these pavings are still widespread. Furthermore, the daily maintenance of these facilities is inadequate; some tactile paving is severely damaged due to years of disrepair and cannot provide effective guidance for blind people. In addition, while blind people can travel with the guidance of guide dogs, the limited number of guide dogs and their low acceptance further restrict their mobility.

[0003] With technological advancements and societal development, an increasing number of navigation aids for the blind have emerged, leading to a more optimistic outlook for their future. However, while blind individuals can utilize navigation and positioning software for travel, these tools often suffer from issues such as insufficient accuracy and operational complexity. Particularly in areas with poor signal, the navigation and positioning software may malfunction due to the inability to update data, severely impacting the travel of blind individuals.

[0004] Therefore, improving the efficiency and convenience of travel for blind people has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method, device, electronic device, and storage medium for guiding the blind, in order to solve the technical problem of low travel efficiency for the blind.

[0006] In a first aspect, embodiments of the present invention provide a method for guiding the blind, applied to a guide robot, the guide robot being equipped with a camera, a lidar, and an inertial navigation device, the method for guiding the blind comprising:

[0007] The system acquires 3D point cloud data and image data of the current environment of the guide robot using the lidar and the camera, and acquires the pose data of the guide robot using the inertial navigation device.

[0008] Based on the 3D point cloud data and the image data, target detection information of the target detection object in the current environment is determined. The target detection information includes the distance and direction of the target detection object relative to the guide robot, as well as the type identification result of the target detection object. The target detection object includes any object existing in the current environment.

[0009] Perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment.

[0010] Based on the target detection information, the type identification result, and the optimized target position and target posture at the current moment, the guide robot is dynamically planned in real time to generate navigation data to guide the guide robot to move.

[0011] The guide robot is controlled to move accordingly based on the navigation data.

[0012] In some embodiments, prior to the step of determining target detection information of a target object in the current environment based on the 3D point cloud data and the image data, the method further includes:

[0013] The external parameters of the lidar and the camera are calibrated to obtain the first calibration parameters;

[0014] The external parameters of the lidar and the inertial navigation device are calibrated to obtain the second calibration parameters;

[0015] Based on the first calibration parameter and the second calibration parameter, the 3D point cloud data, the image data, and the pose data are unified into the same coordinate system.

[0016] In some embodiments, the step of calibrating the extrinsic parameters of the coordinates of the lidar and the camera to obtain the first calibration parameters includes:

[0017] Acquire the first three-dimensional point cloud data and the first image data collected by the guide robot at the same time and location;

[0018] The first 3D point cloud data is converted into pixel coordinates and projected onto the corresponding image using a distortion transformation algorithm to obtain a projected image;

[0019] By aligning the projected image with the image corresponding to the first image data, the first calibration parameters after extrinsic parameter calibration of the lidar and the camera are obtained.

[0020] In some embodiments, the step of calibrating the external parameters of the lidar and the inertial navigation device to obtain the second calibration parameters includes:

[0021] Acquire the second three-dimensional point cloud data and the first pose data of the guide robot at multiple preset fixed positions;

[0022] Perform point cloud registration processing on the second three-dimensional point cloud data to obtain registration matrix data;

[0023] Based on the matrix data corresponding to the first pose data, the second point cloud data, and the registration matrix data, residual optimization processing is performed on the extrinsic parameters of the lidar and the inertial navigation device to obtain the second calibration parameters after the extrinsic parameters of the lidar and the inertial navigation device are calibrated.

[0024] In some embodiments, the step of determining the target detection information of the target object in the current environment based on the 3D point cloud data and the image data includes:

[0025] Filter out ground points from the three-dimensional point cloud data, and perform clustering processing on the remaining non-ground points to generate clustered point cloud clusters;

[0026] The trained image recognition model is invoked to identify the type of the target object in the image data.

[0027] The clusters of point clouds with the same target detection objects and the type identification results are bound together, and the clusters of point clouds that have not undergone the binding process are filtered out to obtain the cluster category data;

[0028] The detection interval of the Kalman filter algorithm is matched with the frame rate of the lidar to obtain the adjusted Kalman filter algorithm.

[0029] Based on the predicted position of the cluster category data after movement according to the adjusted Kalman filter algorithm, and the actual observation position of the cluster category data, the actual position of the cluster category data is determined, and the speed and direction of the cluster category data are calculated based on the actual position of the cluster category data and the movement time interval.

[0030] Based on the speed, the target detection objects corresponding to the clustering category data are distinguished into dynamic detection objects and static detection objects to obtain the distinction result;

[0031] The type identification result, speed, direction, and differentiation result of the target detection object are used as the target detection information of the target detection object.

[0032] In some embodiments, the step of performing point cloud registration and pose optimization processing on the 3D point cloud data and the pose data to obtain the optimized target position and target pose at the current moment includes:

[0033] Remove the dynamic detection objects from the three-dimensional point cloud data, and extract the third three-dimensional point cloud data corresponding to multiple key frames of the guide robot according to the preset moving distance;

[0034] Point cloud registration and pose optimization are performed on the third-dimensional point cloud data and pose data corresponding to the multiple key frames to obtain the optimized actual target position and target pose of the guide robot at the current moment.

[0035] In some embodiments, the method further includes:

[0036] The system broadcasts in real time the target detection information of at least one target detection object closest to the guide robot, and uploads the target location of the guide robot to the application backend that can be logged in and viewed by the blind person's family.

[0037] Secondly, embodiments of the present invention provide a guide device for the blind applied to a guide robot, the guide robot being equipped with a camera, a lidar, and an inertial navigation device, including:

[0038] The acquisition module is used to acquire three-dimensional point cloud data and image data of the current environment in which the guide robot is located through the lidar and the camera, and to acquire the pose data of the guide robot through the inertial navigation device;

[0039] The determination module is used to determine the target detection information of the target detection object in the current environment based on the three-dimensional point cloud data and the image data. The target detection information includes the distance and direction of the target detection object relative to the guide robot, as well as the type identification result of the target detection object. The target detection object includes any object existing in the current environment.

[0040] The processing module is used to perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment.

[0041] The path planning module is used to perform real-time dynamic path planning for the guide robot at the current moment based on the target detection information, the type recognition result, and the optimized target position and target posture at the current moment, and generate navigation data to guide the guide robot to move.

[0042] The control module is used to control the guide robot to move accordingly according to the navigation data.

[0043] Thirdly, embodiments of the present invention provide an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in any of the aforementioned guide methods for the blind.

[0044] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the aforementioned guide methods for the blind.

[0045] This invention provides a method, device, electronic device, and storage medium for guiding blind people. The method acquires 3D point cloud data and image data of the current environment of the guiding robot using lidar and a camera, and obtains the robot's pose data using an inertial navigation device. Based on the 3D point cloud data and image data, it determines the distance and direction of a target object relative to the guiding robot in the current environment, as well as the target object's type identification result. It then performs point cloud registration and pose optimization processing on the 3D point cloud data and pose data to obtain the optimized target position and pose at the current moment. This allows for real-time dynamic path planning for the guiding robot based on target detection information, type identification results, and the optimized target position and pose at the current moment, generating navigation data to guide the robot's movement. This facilitates control of the guiding robot to move according to the navigation data, enabling real-time movement of the guiding robot in complex environments. Blind people only need to follow the robot's guidance to complete their travel, improving their travel efficiency. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a guide method for the blind provided in an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating a normalization process for data collected by different sensors, provided in an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of a process for determining target detection information provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of a point cloud registration and attitude optimization process provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of a guide device for the blind provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0052] Figure 7 This is another structural schematic diagram of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0055] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0056] In related technologies, blind people face numerous difficulties in travel due to their visual impairment. Although accessible facilities such as tactile paving have been built in cities, problems such as obstruction, discontinuity, and poor design of tactile paving are still common. Furthermore, the daily maintenance of these facilities is inadequate; some tactile paving is severely damaged due to years of disrepair and cannot provide effective guidance for blind people. In addition, although blind people can travel with the guidance of guide dogs, the limited number of guide dogs and their low acceptance further restrict their mobility.

[0057] With technological advancements and social development, an increasing number of navigation aids for the blind have emerged, leading to a more optimistic outlook for their future. However, while blind individuals can use navigation software and other assistive tools for travel, these tools often suffer from issues such as insufficient accuracy and operational complexity. Particularly in areas with poor signal, the navigation software may not function properly.

[0058] Therefore, improving the efficiency and convenience of travel for blind people has become an urgent technical problem to be solved.

[0059] To address the technical problems existing in related technologies, this invention provides a method for guiding the blind, applied to a guide robot. The guide robot may be equipped with a camera, lidar, and inertial navigation devices. For details, please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a method for guiding the blind provided in an embodiment of the present invention, which includes steps 101 to 105.

[0060] Step 101: Obtain 3D point cloud data and image data of the current environment of the guide robot through the lidar and the camera, and obtain the pose data of the guide robot through the inertial navigation device.

[0061] In this embodiment, the lidar can be a 3D lidar, the camera can be a high-definition camera, and the inertial navigation device can be a GNSS integrated navigation device. Specifically, the pose data provided in this embodiment can include the positioning data, three-axis angle data, and velocity data of the guide robot, etc.

[0062] Specifically, in this embodiment, the 3D point cloud data collected by the LiDAR can be defined as Cloud, the image data collected by the camera can be defined as Frame, the timestamp can be defined as Time, the attitude data collected by the inertial navigation device can be defined as (Pitch, Roll, Heading), and the navigation position can be defined as (PosX, PosY, PosZ). The LiDAR and the camera are synchronized and timestamped by the PPS+GPRMC clock of the inertial navigation device. The data collected by each sensor can be marked as CloudT = Cloud + Time(i), FrameT = Frame + Time(i), i = (0,1,2,3,4,.......i), where i represents the i-th frame of data, and the attitude data is GD1 = (xspeed,yspeed,zspeed)(Pitch,Roll,Heading) + (PosX,PosY,PosZ) + Time(t), where t represents the t-th frame of data. It should be noted that the frame rate of the 3D point cloud data acquired by the lidar provided in this embodiment can be 10 frames per second, the frame rate of the image data acquired by the camera can be 10 frames per second, and the frame rate of the pose data acquired by the inertial navigation device can be 100 frames per second.

[0063] As an optional embodiment, before performing step 102, this embodiment needs to normalize the data collected by different types of sensors, namely the 3D point cloud data, the image data, and the pose data, to unify the three types of data into the same coordinate system. This facilitates the subsequent fusion and analysis of different types of data, thereby achieving the purpose of real-time path planning and positioning output for the guide robot. For details, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a normalization process for data collected from different sensors, as provided in an embodiment of the present invention. Figure 2 As shown, steps 201 to 203 are included;

[0064] Step 201: Calibrate the external parameters of the lidar and the camera to obtain the first calibration parameters.

[0065] In this embodiment, the camera provided has been pre-calibrated with internal parameters, while the external parameters of the sensors depend on data from other sensors. Therefore, this embodiment only needs to calibrate the external parameters of the lidar, the camera, and the inertial navigation device to complete the normalization process of different sensor data.

[0066] Specifically, the step of calibrating the extrinsic parameters of the coordinates of the lidar and the camera to obtain the first calibration parameters provided in this embodiment may include: acquiring first three-dimensional point cloud data and first image data collected by the guide robot at the same time and location; converting the first three-dimensional point cloud data into pixel coordinates and projecting them into the corresponding image using a distortion conversion algorithm to obtain a projected image; and aligning the projected image with the image corresponding to the first image data to obtain the first calibration parameters after calibrating the extrinsic parameters of the lidar and the camera.

[0067] In this embodiment, for both the LiDAR and the camera, the extrinsic parameter A can be defined as (pitch, roll, heading, x, y, z). Then, the first 3D point cloud data acquired by the LiDAR is converted into (X, Y) pixel coordinates using a distortion transformation method and projected onto the camera interface. The specific transformation algorithm is as follows:

[0068] X = [CloudT.x(i) - p_x] / [f_x * CloudT.z(i)] {i = 0, 1, 2, ..., i = MAX number of point clouds}

[0069] Y = [CloudT.y(i) - p_y] / [f_y * CloudT.z(i)] {i = 0, 1, 2, ..., i = MAX number of point clouds}

[0070] Where p_x, p_y, f_x, and f_y are the intrinsic parameters of the camera.

[0071] X and Y can then be used to represent the projection coordinates of the 3D point cloud data acquired by the LiDAR onto the image FrameT, and the Z value of the point cloud is the projection depth. At this point, the extrinsic parameter A can be transformed into a matrix t. For example, the transformation method for converting the extrinsic parameter A into a matrix is ​​as follows:

[0072] Among them, t=Rtheading*Rtheading*Rtroll*Trs;

[0073] Trs = [x, y, z]

[0074]

[0075] Next, the coordinates in the image data can be multiplied by matrix t, and the determinant of matrix t can be changed by adjusting the value of the extrinsic parameter A, so that the image FrameT and the projection map of the LiDAR are aligned, thereby completing the extrinsic parameter calibration of the LiDAR and the image data and obtaining the first calibration parameters.

[0076] Step 202: Calibrate the external parameters of the lidar and the inertial navigation device to obtain the second calibration parameters.

[0077] In this embodiment, the step of calibrating the extrinsic parameters of the lidar and the inertial navigation device to obtain the second calibration parameters may include: acquiring second three-dimensional point cloud data and first pose data of the guide robot at multiple preset fixed positions; performing point cloud registration processing on the second three-dimensional point cloud data to obtain registration matrix data; and performing residual optimization processing on the extrinsic parameters of the lidar and the inertial navigation device based on the matrix data corresponding to the first pose data, the second point cloud data, and the registration matrix data to obtain the second calibration parameters after calibrating the extrinsic parameters of the lidar and the inertial navigation device.

[0078] In this embodiment, for lidar and inertial navigation devices, the extrinsic parameter can be defined as B(i){i=0~7}, and the matrix of pose data (pitch, roll, heading, x, y, z) can be defined as Bm. The specific transformation method for transforming the extrinsic parameter B(i) into a matrix is ​​the same as the method for transforming the extrinsic parameter A in the above embodiment, and will not be repeated here.

[0079] Then, second-dimensional point cloud data CloudT(i){i=0~n} and first-dimensional pose data GD1(i){i=0~n} are collected from multiple preset fixed positions, and the matrix of GD1 is defined as gdm; the second-dimensional point cloud data corresponding to multiple preset fixed positions are registered using the ICP matching algorithm to calculate the ICP matrix m(i){i=0~n}. The first-dimensional pose data collected by the inertial navigation device after stabilizing at multiple preset fixed positions is GD1(i){i=0~n}, thus the following formula can be obtained:

[0080] gdm(i)*Bm(i)*CloudT(i)=m(i){i=0~n}

[0081] Based on the above formula, residual optimization is performed on Bm(i){i=0~n} to obtain the optimal external parameter Bmm, that is, (pitch,roll,heading,x,y,z) corresponding to Bmm are the second calibration parameters of the lidar and inertial navigation equipment.

[0082] It should be noted that the multiple preset fixed positions provided in this embodiment can be multiple location points traveling back and forth on the same route in the same scenario. Among these multiple location points, the same location can be preferred as both the outbound and return points. After extensive experimental analysis, it was determined that the multiple preset fixed positions provided in this embodiment can be 8 fixed positions on the same straight line, i.e., n=7 in the above formula, where 4 are outbound points and the other 4 are return points, and the corresponding outbound and return points are the same fixed positions. Only the angle of sensor data acquisition is the opposite angle for the outbound and return journeys. That is, the 8 fixed positions are actually 4 fixed points, each corresponding to one outbound point and one return point. Thus, using the 8 fixed positions provided in this embodiment for the external parameter calibration of LiDAR and inertial navigation equipment can enable the sensor to provide stable and reliable spatial position parameters, thereby effectively improving the calibration efficiency of sensor external parameters.

[0083] Step 203: Based on the first calibration parameters and the second calibration parameters, unify the 3D point cloud data, the image data, and the pose data into the same coordinate system.

[0084] After obtaining the first calibration parameters and the second calibration parameters through the methods provided in the above embodiments, the three-dimensional point cloud data, the image data, and the pose data can be fused and unified into the same coordinate system, thereby facilitating subsequent calculation and analysis.

[0085] Step 102: Determine the target detection information of the target object in the current environment based on the three-dimensional point cloud data and the image data.

[0086] In this embodiment, the target detection information includes the distance and direction of the target object relative to the guide robot, as well as the type identification result of the target object. The target object includes any object existing in the current environment.

[0087] In this embodiment, please refer to Figure 3 , Figure 3 This is a flowchart illustrating a method for determining target detection information provided in an embodiment of the present invention, such as... Figure 3 As shown, steps 301 to 307 are included;

[0088] Step 301: Filter out ground points in the three-dimensional point cloud data, and perform clustering processing on the remaining non-ground points to generate clustered point cloud clusters.

[0089] In this embodiment, the three-dimensional point cloud data can be segmented to filter out ground points. Specifically, the process is as follows: Data in the 3D point cloud data where the distance = √(Cloudx*Cloudx*+Cloudy*Cloudy*+Cloudz*Cloudz*) is greater than a preset distance, such as 80 meters, 90 meters, or 100 meters, is filtered out. Then, a 2D coordinate grid is created to generate a grid map (i,j). Based on the mapping relationship between the 3D point cloud data x,y and the grid map (i,j), the 3D point cloud data in each grid is allocated, and the lowest point zmin is determined by the z-value of the 3D point cloud data in each grid. Then, the grid is queried, and the lowest point is found by traversing the eight grid maps surrounding each grid. The data is then weighted and averaged to calculate the filtering parameter Ht. The specific algorithm is: Ht = ∑zmin(i){i = 0~8} / 9. After that, each grid is filtered according to the above Ht+20cm, and grids greater than Ht+20cm are recorded as non-ground filtering points and defined as CloudB. After querying all grid maps, the 3D point cloud data of ground points and non-ground points can be separated.

[0090] In some embodiments, this embodiment can determine the clustering method for finding the grid model based on the LiDAR resolution. For example, the query ratio tanT = tan(0.4 * 3.1415926 / 180) can be set, then the grid size is {CloudB.y * tanT, 0.8m, 0.4m}. Then, each point in CloudB is queried to determine the cluster it falls into based on the model size, generating a clustered point cloud cluster. The length, width, height, center point position, and other relevant information of each point cloud cluster are calculated to facilitate subsequent binding with the target detection object in the image.

[0091] Step 302: Call the trained image recognition model to identify the type of the target object in the image data.

[0092] In this embodiment, an image recognition model can be pre-trained to identify all target detection objects in an image. These target detection objects can include pedestrians, animals, vehicles, traffic light status, zebra crossings, and other objects. Thus, the trained image recognition model can identify the type of target detection objects in the image data and output the length and width information of each target detection object.

[0093] Step 303: Bind the clusters of point clouds and type identification results of the same target detection objects, and filter out the clusters of point clouds that have not undergone the binding process to obtain cluster category data.

[0094] In this embodiment, after obtaining the category recognition results, length information, and width information of each target detection object in the image data, it queries whether the data center point of the clustered point cloud generated in step 301 falls within the length and width of the target detection object in the image data. If the center point of clustered point cloud i falls within the length and width of the image detection of type i, then the point cloud cluster i and the type information i are bound. Point cloud clusters that do not fall within the image detection length and width range are filtered out, and clustering category data is generated, defined as Cluster(i){i=0,1,2...ClusterMax}.

[0095] Step 304: Match the detection interval of the Kalman filter algorithm with the frame rate of the lidar to obtain the adjusted Kalman filter algorithm.

[0096] Step 305: Based on the predicted position of the cluster category data after movement predicted by the adjusted Kalman filter algorithm and the actual observed position of the cluster category data, determine the actual position of the cluster category data, and calculate the speed and direction of the cluster category data based on the actual position and movement time interval of the cluster category data.

[0097] In this embodiment, the clustering category data generated in step 303 is detected using a Kalman filter and data association method. Specifically, in the Kalman filter, the frame frequency of the lidar can be set as the Kalman filter detection interval. The predicted position can then be obtained according to the Kalman data motion equation. Q and R in the equation need to be set manually. In this method, the noise matrix Q in the equation can be set as follows:

[0098] Q matrix: dt*dt / 4,0,0,dt*dt*dt / 2,0,0

[0099] 0,dt*dt / 4,0,0,dt*dt*dt / 2,0

[0100] 0,0,dt*dt / 4,0,0,dt*dt*dt / 2

[0101] dt*dt*dt / 2,0,0,dt*dt,0,0

[0102] 0,dt*dt*dt / 2,0,0,dt*dt,0

[0103] The measurement noise covariance matrix R can be set as follows:

[0104] Matrix R: 1,0,0

[0105] 0,1,0

[0106] 0,0,1

[0107] By combining the predicted (X,Y) output by the prediction equation with the observed X and Y center points in the actual Cluster(i), the actual positions of Xi and Yi of Cluster(i) can be obtained. Then, based on the time interval, the velocity of Cluster(i) (equivalent to a target detection object) in the X and Y directions can be calculated. Based on the velocity in the X and Y directions √(Xspeed*Xspeed+Yspeed*Yspeed), the actual velocity direction and magnitude of Cluster(i) can be calculated. The direction of Cluster(i) is Xspeed / Yspeed. Then, the PDA method is used to perform gate association on multiple targets, calculate the interconnection probability, and generate the ID of stable continuous Cluster(i). The generated Cluster(i) data with velocity, direction, and ID is labeled as Track(i){i=0,1,2...TrackMax}.

[0108] Step 306: Based on the speed, the target detection objects corresponding to the clustering category data are distinguished into dynamic detection objects and static detection objects to obtain the distinction result.

[0109] In this embodiment, the Track(i) data can be queried directly to distinguish between static and dynamic detection objects based on speed, thereby facilitating the classification of dynamic obstacles and marking them as TrackA(i). Then, the distance and type are extracted from Track(i) and played to the blind person using a voice system to remind them of the current environmental conditions around them.

[0110] Step 307: The type identification result, speed, direction and distinction result of the target detection object are used as the target detection information of the target detection object.

[0111] This completes the process of determining the target detection information, enabling the generation of corresponding navigation data based on the target detection information to control the guide robot's movement.

[0112] Step 103: Perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment.

[0113] In this embodiment, to improve the navigation accuracy of the navigation path and output an accurate positioning location, this embodiment can also perform point cloud registration and pose optimization processing on the 3D point cloud data and the pose data to obtain the optimized target position and target pose at the current moment. For details, please refer to... Figure 4 , Figure 4 This is a schematic diagram of a point cloud registration and pose optimization process provided in an embodiment of the present invention, such as... Figure 4 As shown, steps 401 to 402 are included;

[0114] Step 401: Remove the dynamic detection objects from the three-dimensional point cloud data, and extract the third three-dimensional point cloud data corresponding to multiple key frames of the guide robot according to the preset movement distance.

[0115] In this embodiment, the preset moving distance can be 3m, 5m or 7m. That is, in this embodiment, the three-dimensional point cloud data collected when the guide robot moves the preset moving distance is determined as the key frame point cloud data.

[0116] Step 402: Perform point cloud registration and pose optimization processing on the third 3D point cloud data and pose data corresponding to the multiple key frames to obtain the optimized actual target position and target pose of the guide robot at the current moment.

[0117] Specifically, in this embodiment, all clustered point cloud clusters contained in the dynamic Track A(i) in CloudT need to be filtered out, and the CloudT that removes the dynamic Track A(i) is defined as CloudTA. After aligning the Time in CloudTA with the Time in GD1, the initial positions (PosX, PosY, PosZ) are calculated based on the three-axis (xspeed, yspeed, zspeed) linear velocity and frame rate interval obtained in GD1. The calculation process of PosX, PosY, and PosZ is as follows:

[0118] PosX = xspeed * frame rate

[0119] PosY = yspeed * frame rate

[0120] PosZ = zspeed * frame rate.

[0121] Then, a location point storage container P1 and a 3D point cloud data and position pose storage container P2 are established. It is determined whether P1 is empty. If it is empty, the CloudTA corresponding to the key frame and (Pitch, Roll, Heading) in GD1 are stored in container P2. Container P1 stores PosX, PosY, PosZ[0,0,0]. CloudTA is rotated through (Pitch, Roll, Heading) in GD1 and stored in the local map Map. The local map Map is the map Map constructed from the CloudTA corresponding to the key frame before the current time.

[0122] If P1 is not empty, the latest acquired 3D point cloud data CloudTA* is obtained, along with the new pose data GD1(Pitch, Roll, Heading) + (PosX, PosY, PosZ). A rotation and translation matrix RT1 is generated, and then GICP 3D point cloud registration is performed with the local map Map in CloudTAMart. The incremental matrix RT2 is calculated based on the 3D point cloud data of the two keyframes. The actual positional relationship between the two frames is RT3 = RT2 * RT1. Thus, the actual position of the guide robot can be obtained as [RT3(0, 3), RT3(1, 3), RT3(2, 3)].

[0123] Next, the angle calculated based on RT3, as well as the Euclidean distance between the angle and position point of the last frame data in P2, will be used as a condition to set a threshold. The specific threshold is as follows:

[0124] Threshold 1 = P2.Heading - RT3.Heading

[0125] Threshold 2 = P2.Pitch - RT3.Pitch

[0126] Threshold 3 = P2.Roll - RT3.Roll

[0127] Threshold 4=√(P2.PosX-RT3(0,3))*(P2.PosX-RT3(0,3))+(P2.PosY-RT3(1,3))*(P2.PosY-RT3(1,3))

[0128] After setting the threshold, the thresholds 1 to 4 are used to determine whether the latest acquired 3D point cloud data is the 3D point cloud data corresponding to the key frame. If it is the key frame, the position is stored in P1, the 3D point cloud data and the position and pose calculated by RT3 are stored in P2, and then the key frame CloudTA points are added and stored in the local map Map after being transformed by RT3.

[0129] Repeat the method provided in the embodiment of step 401 above until the storage of P2 and P1 data is completed. Then, independently design and optimize the positioning method, assign P1 and P2 to CP1 and CP2 respectively, and determine whether the set conditions are met based on the key frame time and the corresponding threshold set for P1. If the set conditions are met, perform global ICP matching optimization on all positions and poses in the key frame container CP2 using the 3D point cloud data in CP2 to generate a new position and pose of CP2. Then, reassign the new CP2 to P2, and reassign the position in CP1 to P1 after reusing the position in CP2. Then, output the new P1 and P2.

[0130] At this point, point cloud registration and pose optimization processing are completed for the third-dimensional point cloud data and pose data corresponding to the multiple key frames, thereby obtaining the actual target position and target pose of the guide robot at the current moment after optimization.

[0131] Step 104: Based on the target detection information, the type identification result, and the optimized target position and target posture at the current moment, perform real-time dynamic path planning for the guide robot at the current moment to generate navigation data for guiding the guide robot to move.

[0132] In this embodiment, path planning and obstacle avoidance can be performed based on the target detection information, the type identification result, and the optimized target position P1 and target posture P2 at the current moment. Specifically, the position at time P1(i) in P1 can be used to calculate the direction of [PosX, PosY] and the navigation endpoint (X, Y) in P1(i), and then a preset navigation distance, such as 10 meters or 15 meters, can be planned in this direction. Preferably, in this embodiment, a path location point is planned every 15 meters (0.25 meters can be defined as one point, for a total of 60 points). Then, based on the distribution of Track(i) at time i, a fifth-order polynomial method is used on these 60 points to calculate the actual path curve. The path curve is adjusted to avoid obstacles. Furthermore, based on the image data collected by the camera, the presence or absence of traffic lights is detected. This is combined with determining whether to include visually detected zebra crossing information as an optional part of the planned path as a passable area, thereby completing the real-time dynamic path adjustment. Thus, by using the path planning and obstacle avoidance method provided by this embodiment, the real-time movement of the guide robot in complex environments can be achieved, effectively improving the accuracy of path planning and obstacle avoidance.

[0133] Step 105: Control the guide robot to move accordingly according to the navigation data.

[0134] After obtaining the navigation data of the dynamic path output in real time, the guide robot can be controlled to move accordingly according to the navigation data. Blind people only need to follow the guide robot to complete the journey, which effectively improves the travel efficiency of blind people.

[0135] As an optional embodiment, this embodiment can also determine whether the navigation destination (X, Y) has been reached based on [PosX, PosY] in container P1. For example, a threshold condition can be set, such as the distance between the two being within 5m or 3m. That is, if [PosX, PosY] and (X, Y) meet the set threshold condition, the system will automatically report the completion of the trip via voice. Blind people can set whether to stop navigation or go to other locations based on voice prompts.

[0136] In some embodiments, in order to ensure the safety of blind people traveling and to help blind people and their families know the current location of blind people and prevent them from getting lost, the blind guide method provided in this embodiment may further include: broadcasting in real time the target detection information of at least one target detection object closest to the guide robot, and uploading the target location of the guide robot to the application backend that blind family members can log in to view.

[0137] Among them, at least one of the target detection objects closest to the guide robot can be set to be within a preset range of the guide robot, such as 5m, 4m or 1m. In this way, by broadcasting the target detection information of the target detection object closest to the blind person in real time, it can help the blind person perceive the surrounding environment in advance and improve the safety and travel experience of the blind person.

[0138] In summary, this invention provides a method for guiding the blind, applied to a guide robot. The guide robot is equipped with a camera, a lidar, and an inertial navigation device. The method includes: acquiring 3D point cloud data and image data of the current environment of the guide robot using the lidar and the camera; acquiring pose data of the guide robot using the inertial navigation device; determining target detection information of a target object in the current environment based on the 3D point cloud data and the image data; the target detection information including the distance and direction of the target object relative to the guide robot, and the type identification result of the target object; the target object including any object existing in the current environment; performing point cloud registration and pose optimization processing on the 3D point cloud data and the pose data to obtain the optimized target position and target pose at the current moment; and, based on the target detection information, the type identification result, and the optimized target position and target pose at the current moment, performing real-time dynamic path planning for the guide robot at the current moment to generate navigation data for guiding the guide robot to move, and controlling the guide robot to move accordingly according to the navigation data.

[0139] The method provided in this invention has the following beneficial effects: This invention adopts a multi-sensor fusion algorithm, relying on the linear velocity and angle of the IMU part of the lidar and inertial navigation device, and obtains high-precision position and attitude relationship according to the point cloud GICP matching method. It performs joint constraints on the position part of the linear velocity calculation to generate high-precision real-time positioning information, which allows blind family members to know the blind person's precise location anytime and anywhere, solves the problem of GPS signal loss caused by obstruction, and adopts dynamic path planning and dynamic detection methods to plan the optimal path and cross the zebra crossing in real time, thereby improving the efficiency and travel experience of blind people's independent travel.

[0140] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a guide device for the blind. The guide device for the blind can be implemented as an independent entity or integrated into an electronic device, such as a terminal, which may include a mobile phone, a tablet computer, etc.

[0141] To address the same technical problem, this invention also provides a guide device for the blind, which can be applied to a guide robot. The guide robot may be equipped with a camera, lidar, and inertial navigation devices. For details, please refer to... Figure 5 , Figure 5 This is a schematic diagram of a guiding device for the blind provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the blind guide device 500 provided in this embodiment of the invention may include: an acquisition module 501, a determination module 502, a processing module 503, a path planning module 504, and a control module 505;

[0142] The acquisition module 501 is used to acquire three-dimensional point cloud data and image data of the current environment of the guide robot through the lidar and the camera, and to acquire the pose data of the guide robot through the inertial navigation device.

[0143] As an optional embodiment, the blind guide device 500 provided in this embodiment may further include: a first calibration module, a second calibration module, and a unified module;

[0144] The first calibration module is used to calibrate the external parameters of the lidar and the camera to obtain the first calibration parameters.

[0145] In this embodiment, the first calibration module is specifically used to: acquire first three-dimensional point cloud data and first image data collected by the guide robot at the same time and location; convert the first three-dimensional point cloud data into pixel coordinates and project them onto the corresponding image using a distortion conversion algorithm to obtain a projected image; and obtain the first calibration parameters after calibrating the external parameters of the lidar and the camera by aligning the projected image with the image corresponding to the first image data.

[0146] The second calibration module is used to calibrate the external parameters of the lidar and the inertial navigation device to obtain the second calibration parameters.

[0147] In this embodiment, the second calibration module is specifically used to: acquire second three-dimensional point cloud data and first pose data of the guide robot at multiple preset fixed positions; perform point cloud registration processing on the second three-dimensional point cloud data to obtain registration matrix data; and perform residual optimization processing on the extrinsic parameters of the lidar and the inertial navigation device based on the matrix data corresponding to the first pose data, the second point cloud data, and the registration matrix data to obtain second calibration parameters after extrinsic parameter calibration of the lidar and the inertial navigation device.

[0148] The unification module is used to unify the 3D point cloud data, the image data, and the pose data into the same coordinate system according to the first calibration parameters and the second calibration parameters.

[0149] The determination module 502 is used to determine the target detection information of the target detection object in the current environment based on the three-dimensional point cloud data and the image data. The target detection information includes the distance and direction of the target detection object relative to the guide robot, as well as the type identification result of the target detection object. The target detection object includes any object existing in the current environment.

[0150] In this embodiment, the determining module 502 is specifically used for: filtering out ground points in the three-dimensional point cloud data, clustering the remaining non-ground points to generate clustered point cloud clusters; calling the trained image recognition model to identify the type recognition result of the target detection object in the image data; binding the clustered point cloud clusters and type recognition results of the same target detection object, and filtering out the clustered point cloud clusters that have not undergone the binding process to obtain clustered category data; matching the detection interval of the Kalman filter algorithm with the frame rate of the lidar to obtain an adjusted Kalman filter algorithm; predicting the predicted position of the clustered category data after movement based on the adjusted Kalman filter algorithm, and the actual observation position of the clustered category data, determining the actual position of the clustered category data, and calculating the speed and direction of the clustered category data based on the actual position and movement time interval of the clustered category data; classifying the target detection object corresponding to the clustered category data into dynamic detection objects and static detection objects based on the speed to obtain a distinction result; and using the type recognition result, speed, direction, and distinction result of the target detection object as the target detection information of the target detection object.

[0151] The processing module 503 is used to perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment.

[0152] In this embodiment, the processing module 503 is specifically used to: remove the dynamic detection objects in the three-dimensional point cloud data, and extract the third three-dimensional point cloud data corresponding to multiple key frames of the guide robot according to the preset moving distance; perform point cloud registration and posture optimization processing on the third three-dimensional point cloud data and pose data corresponding to the multiple key frames to obtain the optimized actual target position and target posture of the guide robot at the current moment.

[0153] The path planning module 504 is used to perform real-time dynamic path planning for the guide robot at the current moment based on the target detection information, the type recognition result, and the optimized target position and target posture at the current moment, and generate navigation data to guide the guide robot to move.

[0154] The control module 505 is used to control the guide robot to move accordingly according to the navigation data.

[0155] As an optional embodiment, the blind guide device 500 provided in this embodiment may further include: a real-time broadcasting module, which is used to: broadcast in real time the target detection information of at least one target detection object closest to the guide robot, and upload the target position of the guide robot to the application backend that can be logged in and viewed by the blind person's family members.

[0156] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0157] Additionally, please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device can be a mobile terminal such as a smartphone, tablet computer, or other similar device. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602. The processor 601 and the memory 602 are electrically connected.

[0158] The processor 601 is the control center of the electronic device 600. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 602 and calling the data stored in the memory 602, it performs various functions of the electronic device 600 and processes data, thereby monitoring the electronic device 600 as a whole.

[0159] In this embodiment, the processor 601 in the electronic device 600 loads the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 runs the applications stored in the memory 602, thereby realizing any step in the blind guide method provided in the above embodiment.

[0160] The electronic device 600 can implement the steps in any embodiment of the blind guide method provided in the embodiments of the present invention. Therefore, it can achieve the beneficial effects that any blind guide method provided in the embodiments of the present invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0161] Please see Figure 7 , Figure 7 This is another structural schematic diagram of the electronic device provided in the embodiments of the present invention, such as... Figure 7 As shown, Figure 7 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the guide method for the blind provided in the above embodiments. The electronic device 700 can be a mobile terminal such as a smartphone or a laptop computer.

[0162] RF circuit 710 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 710 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 710 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0163] The memory 720 can be used to store software programs and modules, such as the program instructions / modules corresponding to the blind guide method in the above embodiment. The processor 780 executes various functional applications and guides the blind by running the software programs and modules stored in the memory 720.

[0164] Memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 720 may further include memory remotely located relative to processor 780, which can be connected to electronic device 700 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The input unit 730 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 731), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch-sensitive surface 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0166] Display unit 740 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic device 700. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 740 may include display panel 741, optionally configured as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms. Further, touch-sensitive surface 731 may cover display panel 741. When touch-sensitive surface 731 detects a touch operation on or near it, it transmits the information to processor 780 to determine the type of touch event. Subsequently, processor 780 provides corresponding visual output on display panel 741 according to the type of touch event. Although in the figures, touch-sensitive surface 731 and display panel 741 are implemented as two separate components to achieve input and output functions, in some embodiments, touch-sensitive surface 731 and display panel 741 can be integrated to achieve input and output functions.

[0167] The electronic device 700 may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can generate an interruption when the flip is closed or shut down. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the electronic device 700, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0168] Audio circuitry 760, speaker 761, and microphone 762 provide an audio interface between the user and electronic device 700. Audio circuitry 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. Conversely, microphone 762 converts collected sound signals into electrical signals, which are then received by audio circuitry 760, converted back into audio data, and processed by processor 780. The audio data is then transmitted via RF circuitry 710 to, for example, another terminal, or output to memory 720 for further processing. Audio circuitry 760 may also include an earphone jack to facilitate communication between peripheral headphones and electronic device 700.

[0169] Electronic device 700, through transmission module 770 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 770 is shown in the figure, it is understood that it is not an essential component of electronic device 700 and can be omitted as needed without changing the essence of the invention.

[0170] The processor 780 is the control center of the electronic device 700. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 700 by running or executing software programs and / or modules stored in the memory 720, and by calling data stored in the memory 720, thereby providing overall monitoring of the electronic device. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 780.

[0171] The electronic device 700 also includes a power supply 790 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to the processor 780 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 790 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0172] Although not shown, the electronic device 700 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors to implement any step of the blind guidance method provided in the above embodiments.

[0173] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0174] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that, when executed by a processor, can implement any step in the guide method for the blind provided in the above embodiments.

[0175] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0176] Since the instructions stored in the storage medium can execute the steps in any embodiment of the blind guide method provided in the embodiments of the present invention, the beneficial effects that any blind guide method provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0177] The foregoing has provided a detailed description of a method, device, electronic device, and storage medium for guiding the blind, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application. Moreover, those skilled in the art can make several improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for guiding the blind, applied to a guide robot, the guide robot being provided with a camera, a laser radar, and an inertial navigation device, characterized in that, The guide method for the blind includes: The system acquires 3D point cloud data and image data of the current environment of the guide robot using the lidar and the camera, and acquires the pose data of the guide robot using the inertial navigation device. Based on the 3D point cloud data and the image data, target detection information of the target detection object in the current environment is determined. The target detection information includes the distance and direction of the target detection object relative to the guide robot, as well as the type identification result of the target detection object. The target detection object includes any object existing in the current environment. Perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment. Based on the target detection information, the type identification result, and the optimized target position and target posture at the current moment, the guide robot is dynamically planned in real time to generate navigation data to guide the guide robot to move. The guide robot is controlled to move accordingly based on the navigation data.

2. The method of claim 1, wherein, Before the step of determining the target detection information of the target object in the current environment based on the 3D point cloud data and the image data, the method further includes: The external parameters of the lidar and the camera are calibrated to obtain the first calibration parameters; The external parameters of the lidar and the inertial navigation device are calibrated to obtain the second calibration parameters; Based on the first calibration parameter and the second calibration parameter, the 3D point cloud data, the image data, and the pose data are unified into the same coordinate system.

3. The method of claim 2, wherein, The step of calibrating the extrinsic parameters of the coordinates of the lidar and the camera to obtain the first calibration parameters includes: Acquire the first three-dimensional point cloud data and the first image data collected by the guide robot at the same time and location; The first 3D point cloud data is converted into pixel coordinates and projected onto the corresponding image using a distortion transformation algorithm to obtain a projected image; By aligning the projected image with the image corresponding to the first image data, the first calibration parameters after extrinsic parameter calibration of the lidar and the camera are obtained.

4. The method of claim 2, wherein, The step of calibrating the external parameters of the lidar and the inertial navigation device to obtain the second calibration parameters includes: Acquire the second three-dimensional point cloud data and the first pose data of the guide robot at multiple preset fixed positions; Perform point cloud registration processing on the second three-dimensional point cloud data to obtain registration matrix data; Based on the matrix data corresponding to the first pose data, the second point cloud data, and the registration matrix data, residual optimization processing is performed on the extrinsic parameters of the lidar and the inertial navigation device to obtain the second calibration parameters after the extrinsic parameters of the lidar and the inertial navigation device are calibrated.

5. The method of claim 1, wherein, The step of determining the target detection information of the target object in the current environment based on the 3D point cloud data and the image data includes: Filter out ground points from the three-dimensional point cloud data, and perform clustering processing on the remaining non-ground points to generate clustered point cloud clusters; The trained image recognition model is invoked to identify the type of the target object in the image data. The clusters of point clouds with the same target detection objects and the type identification results are bound together, and the clusters of point clouds that have not undergone the binding process are filtered out to obtain the cluster category data; The detection interval of the Kalman filter algorithm is matched with the frame rate of the lidar to obtain the adjusted Kalman filter algorithm. Based on the predicted position of the cluster category data after movement according to the adjusted Kalman filter algorithm, and the actual observation position of the cluster category data, the actual position of the cluster category data is determined, and the speed and direction of the cluster category data are calculated based on the actual position of the cluster category data and the movement time interval. Based on the speed, the target detection objects corresponding to the clustering category data are distinguished into dynamic detection objects and static detection objects to obtain the distinction result; The type identification result, speed, direction, and differentiation result of the target detection object are used as the target detection information of the target detection object.

6. The method of claim 5, wherein, The step of performing point cloud registration and pose optimization processing on the 3D point cloud data and the pose data to obtain the optimized target position and target pose at the current moment includes: Remove the dynamic detection objects from the three-dimensional point cloud data, and extract the third three-dimensional point cloud data corresponding to multiple key frames of the guide robot according to the preset moving distance; Point cloud registration and pose optimization are performed on the third-dimensional point cloud data and pose data corresponding to the multiple key frames to obtain the optimized actual target position and target pose of the guide robot at the current moment.

7. The method of claim 6, wherein, The method further includes: The system broadcasts in real time the target detection information of at least one target detection object closest to the guide robot, and uploads the target location of the guide robot to the application backend that can be logged in and viewed by the blind person's family.

8. A blind guiding device for a blind guiding robot, the blind guiding robot being provided with a camera, a laser radar, and an inertial navigation device, characterized in that, include: The acquisition module is used to acquire three-dimensional point cloud data and image data of the current environment in which the guide robot is located through the lidar and the camera, and to acquire the pose data of the guide robot through the inertial navigation device; The determination module is used to determine the target detection information of the target detection object in the current environment based on the three-dimensional point cloud data and the image data. The target detection information includes the distance and direction of the target detection object relative to the guide robot, as well as the type identification result of the target detection object. The target detection object includes any object existing in the current environment. The processing module is used to perform point cloud registration and pose optimization processing on the three-dimensional point cloud data and the pose data to obtain the optimized target position and target pose at the current moment. The path planning module is used to perform real-time dynamic path planning for the guide robot at the current moment based on the target detection information, the type recognition result, and the optimized target position and target posture at the current moment, and generate navigation data to guide the guide robot to move. The control module is used to control the guide robot to move accordingly according to the navigation data.

9. An electronic device, comprising: The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images