Hospital internal positioning, navigation and obstacle avoidance system for assisting blind person in seeing doctor

Abstract

The invention discloses a hospital internal positioning, navigation and obstacle avoidance system for assisting a blind person in seeing a doctor, which adopts a visual SLAM (Simultaneous Localization and Mapping) algorithm, a multi-view stereo matching algorithm and a global optimization algorithm Bundle AdJustent to reconstruct a three-dimensional map in a hospital, and adopts a PnP algorithm and an RANSAC (Random Sample Consensus) algorithm to obtain three-dimensional coordinates of equipment. A large language model GPT-4o is adopted to analyze the dialogue between the blind person and medical staff to obtain the destination of the blind person, a navigation algorithm A * is combined with a three-dimensional map, built in advance, of a hospital where the device is located in a cloud server, a navigation path is formulated and played to the blind person in real time, action guidance is provided for the blind person, and the blind person can conveniently and rapidly enter the medical staff. And detecting and segmenting a front obstacle by adopting an instance segmentation algorithm YOLOv8-seg, obtaining information of the front obstacle in combination with the depth image and coordinate transformation, and broadcasting the position of the obstacle to the blind.

Description

Technical Field

[0001] The present invention relates to the technical field of positioning, navigation and obstacle avoidance, and particularly to an indoor positioning, navigation and obstacle avoidance system for assisting blind people in seeing a doctor in a hospital. Background Art

[0002] The problem of blind people seeing a doctor is a major social pain point in China, and there are significant bottlenecks for the visually impaired population to overcome visual impairments and enjoy intelligent medical care. Hospitals are characterized by a large flow of people and complex internal buildings. In the complex indoor environment of a hospital, accurate navigation and management require high-precision 3D maps. Currently, most 3D map construction technologies rely on expensive lidar equipment or manual surveying and mapping, which not only have high costs but also have difficulty meeting the requirements of real-time updates in terms of efficiency. In addition, existing technologies have problems of insufficient positioning accuracy and poor real-time performance in indoor position positioning based on images. Although some companies have studied indoor navigation technologies, they have not been applied to hospitals, and simple indoor navigation cannot help blind people avoid obstacles. Although there are obstacle avoidance products that can detect close-range obstacles, the detection range is limited, and they cannot detect the distribution of static objects and dynamic people flow in the entire space. At the same time, the obstacle avoidance products do not have a navigation function and cannot provide assistance to blind people in the planning of the medical treatment route. So far, there is no suitable system that can combine indoor positioning, navigation and obstacle avoidance to help blind people see a doctor. Summary of the Invention

[0003] To solve the problem that existing navigation products cannot assist blind people in seeing a doctor through accurate indoor positioning and navigation and at the same time do not have an obstacle avoidance function with a wide detection range, the present invention provides an indoor positioning, navigation and obstacle avoidance system for assisting blind people in seeing a doctor in a hospital. Through the implementation of positioning, navigation and obstacle avoidance, the blind can independently see a doctor inside the hospital.

[0004] According to one aspect of the specification of the present invention, there is provided an indoor positioning, navigation and obstacle avoidance system for assisting blind people in seeing a doctor in a hospital, including: A client device, configured to collect image, GPS information and IMU information and upload them to the cloud, receive the positioning, navigation and obstacle avoidance information fed back by the cloud and execute it, and perform human-computer interaction at the same time; A cloud server, configured to generate a 3D map based on the collected image, GPS information and IMU information, mark the department name and spatial coordinate information of each room in the 3D map to form an indoor 3D map of the hospital; calculate the 3D coordinates of the client device according to the real-time acquired image and GPS information, in combination with the PnP algorithm and the RANSAC algorithm; parse the destination according to the voice captured by the human-computer interaction, in combination with a large language model; obtain the 3D coordinates of the destination and plan a navigation path according to the parsed destination, in combination with the real-time 3D coordinates of the client device and the indoor 3D map of the hospital; and identify the position of the obstacle based on the real-time image and broadcast it in real time.

[0005] As a further technical solution, the client device includes: An image acquisition module for real-time acquisition of image information; A GPS module for real-time acquisition of GPS information; An IMU module for real-time acquisition of the angular velocity and acceleration information of the client device; An audio interaction module for capturing the voice information in the user's environment and playing back the processed voice information; A communication module for uploading the acquired image information, GPS information, IMU information and voice information to the cloud server, and receiving the processed information sent by the cloud server; A control module for connecting to the image acquisition module, GPS module, IMU module, audio interaction module and communication module respectively to implement the corresponding functions of each module, and connecting to the cloud server to execute the positioning, navigation and obstacle avoidance information feedback by the cloud server.

[0006] As a further technical solution, when the cloud server forms a three-dimensional map of the hospital interior, it further includes: Based on the acquired color image and depth image, extract image features, perform feature point matching, generate a sparse point cloud using the visual SLAM algorithm, and generate a dense point cloud in combination with the multi-view stereo matching algorithm; Improve the accuracy of the point cloud based on the global optimization algorithm Bundle Adjustment, and add textures to generate a realistic three-dimensional map of the hospital interior; Manually label the department name and spatial coordinate information of each room in the three-dimensional map, slice the three-dimensional map and store it in the cloud server, and save the GPS information of the three-dimensional map and the manually labeled department name and spatial coordinate information.

[0007] As a further technical solution, when the cloud server performs positioning, it further includes: Based on the acquired real-time GPS information, retrieve the three-dimensional map of the hospital interior; Extract features from the color image uploaded by the client device and match them with the feature points in the current three-dimensional map of the hospital interior where the client device is located. Combine the PnP algorithm and the RANSAC algorithm to calculate the three-dimensional coordinates of the client device.

[0008] As a further technical solution, when the cloud server performs destination parsing and navigation, it further includes: Call the large language model GPT-4o to parse the conversation between the blind person and the medical staff to obtain the destination, and send the destination to a pre-set navigation program as the navigation end point of the navigation program; The navigation algorithm is used to formulate a navigation path and send it to the user terminal device for real-time playback.

[0009] As a further technical solution, when the cloud server performs destination parsing and navigation, it further includes: Real-time obtain the voice information uploaded by the user terminal device and perform destination parsing; Perform re-navigation based on the real-time parsed destination information and adjust the navigation path in real time.

[0010] As a further technical solution, when the cloud server performs obstacle avoidance, it further includes: Based on the instance segmentation algorithm YOLOv8-seg, detect and segment the obstacles in the acquired color image to obtain the coordinate frames and masks of the obstacles; Correspond the mask of the obstacle to the depth image and calculate the average depth value of each point in the mask of each obstacle; Take the average depth value as the depth value of the obstacle, and convert the pixel coordinates of the obstacle to the normalized image plane coordinates through coordinate transformation to obtain the obstacle position.

[0011] According to one aspect of the specification of the present invention, a method for internal positioning, navigation and obstacle avoidance in a hospital to assist the blind in seeing a doctor is provided, which is implemented by using the described system. The method includes: Real-time collect the image information, GPS information, IMU information and voice information of the user terminal device and upload them to the cloud server; According to the real-time acquired image and GPS information, combined with the PnP algorithm and the RANSAC algorithm, calculate the three-dimensional coordinates of the user terminal device; according to the voice captured by human-computer interaction, combine the large language model to parse the destination; according to the parsed destination, combine the real-time three-dimensional coordinates of the user terminal device and the internal three-dimensional map of the hospital to obtain the three-dimensional coordinates of the destination and plan a navigation path; Perform voice broadcast based on the planned navigation path, and at the same time, identify the position of the obstacle based on the real-time image and perform real-time broadcast of it.

[0012] As a further technical solution, the method further includes: Generate a three-dimensional map based on the image, GPS information and IMU information uploaded by the user terminal device, and mark the department name and spatial coordinate information of each room in the three-dimensional map to form an internal three-dimensional map of the hospital.

[0013] As a further technical solution, the method further includes: Real-time collect the voice information of the environment where the user terminal device is located and upload it to the cloud server; Perform destination parsing according to the real-time acquired voice information, and re-plan the navigation path when the parsed destination changes; Send the re-planned navigation path to the user device and conduct voice broadcast.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention proposes a hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to see a doctor, realizing indoor positioning, navigation and obstacle avoidance functions, improving the indoor positioning accuracy and facilitating the intelligent navigation of blind users by constructing a three-dimensional map of the hospital internal with voice information, and realizing an obstacle avoidance function with a wide detection range through real-time obstacle detection and voice broadcast based on image information, thus realizing indoor positioning, navigation and obstacle avoidance beneficial to the blind to see a doctor.

[0015] 2. The present invention provides intelligent interactive navigation combined with the large language model (GPT-4o): The cloud server calls GPT-4o to parse the conversations between the blind and medical staff, automatically identifies and extracts the destinations within the hospital (such as the Chinese pharmacy, the third surgical consulting room), combines the three-dimensional map with SLAM positioning, intelligently guides the user to move, and adjusts the navigation path in real time, enhancing the blind assistance and the navigation experience of the smart hospital.

[0016] 3. The present invention utilizes cloud computing capabilities: All complex calculations (3D reconstruction, SLAM, conversation parsing) are completed on the cloud server, with a small computational burden on the terminal device, strong device scalability, and improved real-time positioning and barrier-free navigation capabilities in the smart hospital scenario.

[0017] 4. The present invention constructs a three-dimensional hospital map with semantic information. Manually mark the spatial coordinates of hospital departments and endow the three-dimensional map with semantic information, making it available for precise positioning and intelligent navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings used in the description of the embodiments or the prior art. Obviously, the following-described drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is the system hardware architecture diagram provided by the embodiment of the present invention; Figure 2 It is the method flow schematic diagram provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] It should be noted that: In the description and claims of the present invention and the above-mentioned drawings, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the drawings are only exemplary descriptions and do not necessarily include all the content and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. In addition, the technical features in each embodiment or individual embodiment provided by the present invention can be combined with each other arbitrarily to form a new technical solution. This combination is not restricted by the order of steps and / or the structure composition mode, but must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0023] An embodiment of the present invention provides a hospital internal positioning, navigation and obstacle avoidance system for assisting blind people to seek medical treatment, including a client device and a cloud server, and the client device and the cloud server are connected through a network. Among them, the client device is used to collect image, GPS information and IMU information and upload them to the cloud, and receive the positioning, navigation and obstacle avoidance information fed back by the cloud and execute it, and at the same time perform human-computer interaction. The cloud server is used to generate a three-dimensional map based on the collected image, GPS information and IMU information, and mark the department name and spatial coordinate information of each room in the three-dimensional map to form an internal three-dimensional map of the hospital; calculate the three-dimensional coordinates of the client device according to the real-time acquired image and GPS information, combined with the PnP algorithm and the RANSAC algorithm; parse the destination according to the voice captured by human-computer interaction, combined with the large language model; according to the parsed destination, combined with the real-time three-dimensional coordinates of the client device and the internal three-dimensional map of the hospital, obtain the three-dimensional coordinates of the destination and plan a navigation path; and identify the position of the obstacle based on the real-time image and broadcast it in real time.

[0024] Based on the system provided by the embodiment of the present invention, it can provide an auxiliary medical treatment solution with higher positioning accuracy and more intelligent navigation for blind people, can parse the needs of blind users in real time through the large language model and update the navigation route in real time, and at the same time can provide more intelligent navigation services for blind users by constructing an internal map of the hospital with semantic information.

[0025] Please refer to Figure 1 , the hospital internal positioning, navigation and obstacle avoidance system for assisting blind people to seek medical treatment provided by the embodiment of the present invention specifically includes: an RGBD camera, a WiFi module, a GPS module, an audio interaction module, an IMU module and a cloud server. The WiFi module and the RGBD camera are connected to the core board and are used to construct a cloud hospital map. The GPS module and the IMU module are connected to the core board and are used for real-time positioning. The core board is connected to the cloud server to parse the dialogue and formulate a navigation path. The core board is connected to the audio interaction module to broadcast the obstacle information to the blind.

[0026] The RGBD camera is an RGBD camera with a color resolution of 1920x1080, a depth resolution of 1080x720, and a sampling frame rate of 30Hz. Transmit the collected color image and depth image data to the core board.

[0027] The WiFi module selects a 2.4G WiFi module model of the same type with faster transmission to ensure the speed of data transmission to the cloud.

[0028] The GPS module determines the location by receiving signals sent by Global Positioning System (GPS) satellites. The GPS module sends the obtained positioning data to the core board through the serial port. The GPS module provides a certain positioning ability with an error range at the meter level, enabling the cloud server to correctly retrieve the 3D map of the hospital.

[0029] The core board uses the STM32f103 model and realizes corresponding functions by connecting peripherals. The core board is stably powered by a built-in lithium battery and adopts a full-through-hole four-layer board design, which not only significantly reduces the device volume but also ensures that the core board can work stably for a long time. When the core board is connected to WiFi, it transmits the received user voice input, color image, depth image, and GPS data to the cloud server. It receives the behavior guidance returned by the cloud server and transmits it to the voice interaction module.

[0030] The cloud server uses a public cloud server and is interconnected with all devices through the network. When the hospital first purchases the device, the technician connects one device to WiFi and scans the site environment of the hospital. While scanning, the camera data will be transmitted to the core board, and then the core board transmits it to the cloud server. The cloud server saves the picture data of the hospital in a timely manner and uses 3D reconstruction technology to model the 3D scene of the hospital. When the product is in use, the user's conversation, camera image, and GPS will be transmitted to the cloud server in real time. The cloud server will parse the GPS data and then retrieve the 3D map of the hospital. By matching the camera image with the 3D map of the hospital and using SLAM technology, the accurate position of the user can be obtained. By parsing the user's conversation through ChatGPT, the user's movement is guided. By processing the information in the color image and depth image, the information of the obstacles around the user is determined.

[0031] The audio interaction module is a hardware device that can realize voice recognition, processing, and generation. The audio interaction module adopted in the present invention includes voice recognition and speech synthesis. The user's voice is converted into a digital signal, transmitted to the core board, and then transmitted to the cloud server through the core board. The cloud server processes the digital signal, and the processed result is returned to the core board. The core board synthesizes the processed result into voice and plays it.

[0032] The inertial measurement unit (IMU) is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. An IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the object on the three independent axes of the carrier coordinate system, while the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system, measure the angular velocity and acceleration of the object in three-dimensional space, and calculate the attitude of the object based on this. The inertial measurement unit (IMU) is used to record the position information and direction information of the captured images during the 3D reconstruction of the hospital.

[0033] For the construction of the internal three-dimensional map of the cloud hospital, the implementation method is to use an RGBD camera with a color resolution of 1920x1080, a depth resolution of 1080x720, and a sampling frame rate of 30Hz to obtain color images and depth images. The GPS module obtains the GPS information of the current device in real time. The IMU module records the position information and orientation information of the captured images in real time. The core board connected to WiFi uploads the collected color images, GPS information, and IMU information to the cloud server. On the cloud server side, algorithms such as SIFT and ORB are used to extract image features and perform feature point matching. The visual SLAM algorithm (a general term for a class of algorithms) is used to generate a sparse point cloud, and the multi-view stereo matching algorithm (a general term for a class of algorithms) is combined to generate a dense point cloud. The accuracy of the point cloud is improved through the global optimization algorithm Bundle Adjustment, and textures are added to generate a realistic three-dimensional map. Then, the department names and spatial coordinate information of each room in the three-dimensional map are manually labeled. Finally, the three-dimensional map is sliced and stored in the cloud server, and the GPS information of the three-dimensional map and the manually labeled department names and spatial coordinate information are saved.

[0034] For the real-time positioning, the implementation method is to obtain the color images captured by the camera in real time after connecting to WiFi and running normally, and the GPS module obtains the GPS information of the current device in real time. Then, the captured color images and GPS information are transmitted to the cloud server through the core board. In the cloud server, through the GPS information, the pre-built three-dimensional map of the hospital where the device is located is retrieved, the features of the uploaded color images are extracted, and the feature points in the pre-built three-dimensional map of the hospital where the device is located in the cloud server are matched. Combining the PnP algorithm and the RANSAC algorithm, the three-dimensional coordinates of the device capturing the color image are calculated.

[0035] The described human-computer interaction is implemented as follows: Incorporate a large language model, which has the ability to parse conversations between people and can provide destinations. Specifically: The audio interaction module captures the conversation content between medical staff and the blind. The core board transmits this conversation content and the device's GPS information to the cloud server. After the cloud server receives this conversation content and the device's GPS information, the program in the cloud server will call the large language model GPT-4o to parse the conversation content, obtain the destinations inside the hospital mentioned in the conversation content (such as the traditional Chinese medicine pharmacy, the third surgical consultation room, etc.). The cloud server sends the destinations inside the hospital parsed by the large language model to the core board, and the core board broadcasts them to the blind through the audio interaction module and broadcasts to ask the blind if they want to go to this destination. After receiving the specific reply from the blind, the reply content of the blind is transmitted to the cloud server, and the large language model in the cloud server will continue to parse the user's reply content. If the user's destination is parsed, the destination will be sent to the navigation program as the navigation end point of the navigation program. Keep the large language model in the cloud server parsing the user's conversation program at all times to timely obtain or correct the destination of the blind.

[0036] The described real-time navigation is implemented by obtaining the real-time three-dimensional coordinates of the current device through the described real-time positioning, and using these coordinates as the three-dimensional coordinates of the starting point. Obtain the destination of the blind through the described human-computer interaction, and combine with the pre-built three-dimensional map of the hospital where the device is located retrieved through GPS information to obtain the three-dimensional coordinates of the destination. Use the navigation algorithm and combine with the pre-built three-dimensional map of the hospital where the device is located in the cloud server to formulate a navigation path and play it to the blind in real time to provide action guidance for the blind.

[0037] The described real-time obstacle avoidance is implemented by using the instance segmentation algorithm YOLOv8-seg to detect and segment obstacles (people, trash cans, etc.) in the acquired color image, obtaining the coordinate frames and masks of the obstacles, and then corresponding these masks of the obstacles to the depth image, calculating the average depth value of each point in each obstacle mask, and using the average depth value as the depth value of the obstacle, which is used as the Z value of the obstacle in the camera coordinate system. In addition, take the center point of the coordinate frame of the obstacle as the coordinate of the obstacle in the image coordinate system . Under the condition of obtaining the internal parameter matrix K of the camera, K is specifically:

[0038] Among them, : The focal length of the camera (in pixels); : The principal point (the intersection of the optical axis and the image plane).

[0039] Then, convert the pixel coordinates of the obstacle to the normalized image plane coordinates :

[0040] At this time, are the two-dimensional coordinates on the normalized image plane, representing the point positions in the image plane.

[0041] Finally, the normalized image plane coordinates are converted to the three-dimensional coordinates in the camera coordinate system :

[0042] where Z is the Z value of the obstacle in the camera coordinate system calculated by the depth camera.

[0043] After obtaining the three-dimensional coordinates of each obstacle in the camera coordinate system the positions of the obstacles are reported to the blind in real time to prevent the blind from hitting the obstacles.

[0044] Based on the same inventive concept as the foregoing system embodiment, an embodiment of the present invention further provides a method for internal positioning, navigation and obstacle avoidance in a hospital to assist the blind in seeing a doctor. Please refer to Figure 2 The method includes: Step 1, collect the image information, GPS information, IMU information and voice information of the user terminal device in real time and upload them to the cloud server.

[0045] In this step, the image information includes depth images and color images, which are used for the construction, real-time positioning, navigation and obstacle avoidance of the internal three-dimensional map of the hospital. The GPS information is used to combine the GPS information with the real-time three-dimensional space coordinates to determine the destination three-dimensional coordinates during real-time positioning. The IMU information is used to record the position information and orientation information of the captured images during the construction of the three-dimensional map. The voice information is the content of the voice conversation between the blind patient and the medical staff. Through this conversation content, the destination of the blind patient is parsed by combining with a large language model to facilitate the provision of precise navigation.

[0046] Step 2, calculate the three-dimensional coordinates of the user terminal device according to the images and GPS information obtained in real time, combined with the PnP algorithm and the RANSAC algorithm; parse the destination according to the voice captured by human-computer interaction, combined with a large language model; according to the parsed destination, combined with the real-time three-dimensional coordinates of the user terminal device and the internal three-dimensional map of the hospital, obtain the destination three-dimensional coordinates and plan the navigation route.

[0047] s It should be noted that the voice information of the user terminal device is captured and uploaded to the cloud server in real time, and is parsed in real time by a large language model to facilitate timely obtaining of the destination change of the blind patient and re-planning the navigation route.

[0048] Since a three-dimensional map with voice information, specific to each room name and spatial coordinates, is pre-constructed in this embodiment, blind patients can accurately reach their destinations. Compared with existing indoor positioning and navigation methods, this embodiment can provide more accurate positioning and more intelligent navigation, solving the problems that existing positioning systems are difficult to accurately locate the consultation room and blind patients are difficult to accurately find the consultation room.

[0049] Step 3, perform voice broadcast based on the planned navigation path, and at the same time, identify the position of obstacles based on real-time images and perform real-time broadcast of them.

[0050] Preferably, the method further includes: generating a three-dimensional map based on the images, GPS information, and IMU information uploaded by the user terminal device, and marking the department name and spatial coordinate information of each room in the three-dimensional map to form a three-dimensional map of the hospital interior.

[0051] Preferably, the method further includes: Real-time collect the voice information of the environment where the user terminal device is located and upload it to the cloud server; Perform destination parsing according to the real-time obtained voice information, and re-plan the navigation path when the parsed destination changes; Send the re-planned navigation path to the user terminal device and perform voice broadcast.

[0052] In summary of the above embodiments, in order to solve the problem that blind people cannot seek medical treatment independently inside the hospital, the present invention discloses an indoor positioning, navigation and obstacle avoidance system for assisting blind people to seek medical treatment in the hospital. It uses the Visual SLAM algorithm, multi-view stereo matching algorithm and global optimization algorithm Bundle Adjustment to reconstruct the three-dimensional map of the hospital interior, uses the PnP algorithm and RANSAC algorithm to obtain the three-dimensional coordinates of the device, uses the large language model GPT-4o to parse the conversation between the blind person and the medical staff to obtain the destination of the blind person, uses the navigation algorithm A* combined with the pre-built three-dimensional map of the hospital where the device is located in the cloud server to formulate a navigation path and play it to the blind person in real time to provide action guidance for the blind person, uses the instance segmentation algorithm YOLOv8-seg to detect and segment the obstacles in front, combines the depth image and coordinate transformation to obtain the information of the obstacles in front, and broadcasts the position of the obstacles to the blind person.

[0053] Furthermore, the present invention builds a complete set of hardware design architectures. This hardware architecture includes a cloud server, an RGBD camera, a core board, an audio interaction module, a WiFi module, a GPS module, and an IMU module, providing a hardware environment for the application of high-tech technologies.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to seek medical treatment, characterized in that, Including: A client device, which is used to collect image, GPS information and IMU information and upload them to the cloud, receive the positioning, navigation and obstacle avoidance information fed back by the cloud and execute it, and conduct human-computer interaction at the same time; A cloud server, which is used to generate a three-dimensional map based on the collected image, GPS information and IMU information, label the department name and spatial coordinate information of each room in the three-dimensional map to form a three-dimensional map of the hospital interior; calculate the three-dimensional coordinates of the client device according to the real-time acquired image and GPS information, combined with the PnP algorithm and the RANSAC algorithm; parse the destination according to the voice captured by human-computer interaction, combined with a large language model; obtain the three-dimensional coordinates of the destination and plan a navigation path according to the parsed destination, combined with the real-time three-dimensional coordinates of the client device and the three-dimensional map of the hospital interior; and identify the position of obstacles based on real-time images and broadcast them in real time.

2. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to see a doctor according to claim 1, characterized in that, The client device includes: An image acquisition module, which is used to collect image information in real time; A GPS module, which is used to collect GPS information in real time; An IMU module, which is used to collect the angular velocity and acceleration information of the client device in real time; An audio interaction module, which is used to capture the voice information of the user's environment and broadcast the processed voice information; A communication module, which is used to upload the collected image information, GPS information, IMU information and voice information to the cloud server, and receive the processed information sent by the cloud server; A control module, which is used to connect the image acquisition module, GPS module, IMU module, audio interaction module and communication module respectively to realize the corresponding functions of each module, and connect to the cloud server to execute the positioning, navigation and obstacle avoidance information fed back by the cloud server.

3. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to see a doctor according to claim 1, characterized in that, When forming the three-dimensional map of the hospital interior, the cloud server further includes: Based on the acquired color image and depth image, extract image features, conduct feature point matching, generate a sparse point cloud using the visual SLAM algorithm, and generate a dense point cloud in combination with the multi-view stereo matching algorithm; Improve the accuracy of the point cloud based on the global optimization algorithm Bundle Adjustment, and add textures to generate a realistic three-dimensional map of the hospital interior; Manually label the department name and spatial coordinate information of each room in the three-dimensional map, slice the three-dimensional map and store it in the cloud server, and save the GPS information of the three-dimensional map and the manually labeled department name and spatial coordinate information.

4. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to seek medical treatment according to claim 1, characterized in that, When conducting positioning, the cloud server further includes: Based on the acquired real-time GPS information, retrieve the three-dimensional map of the hospital interior; Extract features from the color image uploaded by the client device, match them with the feature points in the current three-dimensional map of the hospital interior where the client device is located, and calculate the three-dimensional coordinates of the client device in combination with the PnP algorithm and the RANSAC algorithm.

5. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to seek medical treatment according to claim 1, wherein When conducting destination parsing and navigation, the cloud server further includes: Call the large language model GPT-4o to parse the conversation between the blind person and the medical staff to obtain the destination, and send the destination to a pre-set navigation program as the navigation end point of the navigation program; Adopt a navigation algorithm to formulate a navigation path and send it to the user device for real-time playback.

6. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to see a doctor according to claim 5, characterized in that, When performing destination parsing and navigation, the cloud server further includes: Real-time acquisition of voice information uploaded by the client device and performing destination parsing; Based on the destination information parsed in real time, re-navigating and adjusting the navigation path in real time.

7. The hospital internal positioning, navigation and obstacle avoidance system for assisting the blind to seek medical treatment according to claim 1, wherein When performing obstacle avoidance, the cloud server further includes: Detecting and segmenting obstacles in the acquired color image based on the instance segmentation algorithm YOLOv8-seg to obtain the coordinate frames and masks of the obstacles; Mapping the mask of the obstacle to the depth image and calculating the average depth value of each point in the obstacle mask; Taking the average depth value as the depth value of the obstacle, and converting the pixel coordinates of the obstacle to the normalized image plane coordinates through coordinate transformation to obtain the obstacle position.

8. A method for internal positioning, navigation and obstacle avoidance in a hospital to assist blind people in seeing a doctor, implemented using the system described in any one of claims 1-7, characterized in that, The method includes: Real-time acquisition of image information, GPS information, IMU information and voice information of the client device and uploading them to the cloud server; Based on the real-time acquired images and GPS information, combined with the PnP algorithm and the RANSAC algorithm, calculating the three-dimensional coordinates of the client device; based on the voice captured by human-computer interaction, parsing the destination in combination with the large language model; based on the parsed destination, combined with the real-time three-dimensional coordinates of the client device and the internal three-dimensional map of the hospital, obtaining the three-dimensional coordinates of the destination and planning a navigation path; Performing voice broadcast based on the planned navigation path, and at the same time identifying the position of the obstacle based on the real-time image and performing real-time broadcast.

9. The method for internal positioning, navigation and obstacle avoidance in a hospital to assist the blind in seeing a doctor according to claim 8, wherein The method further includes: Generating a three-dimensional map based on the images, GPS information and IMU information uploaded by the client device, and marking the department name and spatial coordinate information of each room in the three-dimensional map to form an internal three-dimensional map of the hospital.

10. The method for internal positioning, navigation and obstacle avoidance in a hospital to assist the blind in seeing a doctor according to claim 8, characterized in that, The method further includes: Real-time acquisition of the voice information of the environment where the client device is located and uploading it to the cloud server; Performing destination parsing based on the real-time acquired voice information, and re-planning the navigation path when the parsed destination changes; Sending the re-planned navigation path to the client device and performing voice broadcast.

Images