Intelligent hospital guide system and method based on AR vision and voice navigation

Abstract

The invention provides an intelligent hospital guide system and method based on AR vision and voice navigation. The intelligent hospital guide system comprises a user interaction layer, a technical support layer, a data and background layer, a third-party service layer and a hardware dependence layer. According to the intelligent hospital guide system based on AR vision and voice navigation, accurate navigation is achieved, a barrier-free road is guided, through the advanced AR vision navigation technology, the APP provides an accurate and errorless navigation path for a patient, the possibility of navigation errors is greatly reduced, virtuality and reality are seamlessly fused, and the intelligent hospital guide system based on AR vision and voice navigation has the advantages of being high in practicability and the like. The patient can walk on a familiar map in a hospital, and each step is full of confidence; the voice navigation function brings unprecedented convenience to the user, and especially for the visual impairment, the old or the users with inconvenience in hands, the service is as an intimate guide to lead the visual impairment, the old or the users with inconvenience in hands to easily shuttle to all corners of a hospital and enjoy barrier-free medical experience.

Description

TECHNICAL FIELD

[0001] The application relates to the medical field, in particular to an intelligent guiding system and method based on AR vision and voice navigation. BACKGROUND

[0002] With the application of technological innovation in the medical field, at present, a batch of medical auxiliary APPs based on mobile devices have emerged in the market, such as health monitoring, drug reminding, online consultation and various functions, which greatly improve the efficiency, convenience and overall experience of medical services.

[0003] However, most of these medical auxiliary APPs lack intuitive visual assistance and professional guiding functions, especially in the complex scene of a hospital, patients often face confusion and anxiety when looking for departments and doctors.

[0004] Therefore, it is necessary to provide an intelligent guiding system based on AR vision and voice navigation to solve the above technical problems. SUMMARY

[0005] The application provides an intelligent guiding system based on AR vision and voice navigation, which solves the problem that most medical auxiliary APPs lack intuitive visual assistance and professional guiding functions, especially in the complex scene of a hospital, patients often face confusion and anxiety when looking for departments and doctors.

[0006] To solve the above technical problems, the application provides an intelligent guiding system based on AR vision and voice navigation, which comprises:

[0007] A user interaction layer, a technical support layer, a data and background layer, a third-party service layer and a hardware dependent layer;

[0008] The input end of the technical support layer is connected to the output end of the user interaction layer, and the input end of the data and background layer is connected to the output end of the technical support layer;

[0009] The input end of the third-party service layer is connected to the output end of the data and background layer, and the input end of the hardware dependent layer is connected to the output end of the third-party service layer;

[0010] The user interaction layer comprises a guiding service module, an AR vision navigation module, a voice interaction module and a real-time map interface;

[0011] The technical support layer comprises a SLAM positioning and modeling module, a voice synthesis module, a voice recognition module and a path planning algorithm module;

[0012] The data and background layer comprises a hospital department database and a user behavior log;

[0013] The third-party service layer includes a map APP and a voice APP.

[0014] The hardware-dependent layer includes an AR scene resource library, a mobile phone camera, a gyroscope / accelerometer, a microphone, and a speaker.

[0015] Preferably, the AR visual navigation module uses SLAM positioning and modeling technology for map construction and real-time positioning, and realizes positioning and tracking based on adjacent image feature matching estimation algorithm, and the voice interaction module receives user voice instructions through a voice recognition module and feeds back the guidance information to the user in the form of voice through a voice synthesis module.

[0016] Preferably, the path planning algorithm module is used to plan an optimal path from the current position to the target department according to the hospital department database information and the user position, and cooperates with the AR visual navigation module and the voice interaction module, the hospital department database is used to store the information of the distribution and position of the hospital departments, and the user behavior log is used to record the operation behavior of the user in the guidance process, thereby providing data support for optimizing the guidance service.

[0017] Preferably, the map APP provides basic map data, the voice APP assists the voice interaction module in voice processing, the mobile phone camera is used for image data acquisition, the gyroscope / accelerometer is used for assisting positioning, and the microphone and the speaker are used for realizing the input and output of voice.

[0018] Preferably, when the SLAM positioning and modeling is used to construct a map, the steps of initial pose estimation, map construction, loop closure detection, and global optimization are used to improve the accuracy and consistency of the map, the AR scene resource library stores scene data for augmented reality navigation, cooperates with the SLAM positioning and modeling and the AR visual navigation module, and presents an intuitive navigation interface for the user.

[0019] Preferably, the SLAM positioning and modeling includes a data acquisition module, a front-end processing module, a back-end optimization module, a positioning and tracking module, a multi-sensor data fusion module, and an error processing and recovery module.

[0020] Preferably, the data acquisition module includes image data acquisition and sensor data acquisition, the image data acquisition is realized by continuously shooting each area of the hospital by the mobile phone camera, the sensor data acquisition includes magnetic sensor assisted positioning by sensing a magnetic field, and the like, and the front-end processing module is used for image feature extraction and preprocessing of image data and preprocessing of sensor data.

[0021] Preferably, the back-end optimization module is used to improve the accuracy and consistency of the map through initial pose estimation, map construction, loop closure detection and global optimization, the positioning and tracking module estimates the position and attitude of the device in real time based on feature matching or visual odometry, tracks the position and attitude, updates the motion, in the case of large errors, re-performs the initial pose estimation and map construction.

[0022] Preferably, the multi-sensor data fusion module fuses image and sensor data using Kalman filtering algorithm to provide more accurate navigation and guidance services, and the error processing and recovery module includes error detection and error recovery.

[0023] A method of an intelligent guidance system based on AR vision and voice navigation, comprising the following steps:

[0024] S1, user interaction process: the user starts the intelligent guidance system and enters the real-time map interface, can issue voice instructions through the voice interaction module, such as querying the location of a department, the voice recognition module receives the instructions and converts them into text information to pass to other modules of the system;

[0025] S2, positioning and navigation process:

[0026] S21, positioning: the system uses SLAM positioning and modeling technology, combines image data collected by the phone camera and sensor data of the gyroscope / accelerometer for positioning, first acquires image and sensor data through the data acquisition module, the front-end processing module pre-processes the data, the back-end optimization module performs initial pose estimation and map construction, the positioning and tracking module estimates the position and attitude of the device in real time based on feature matching or visual odometry, and realizes accurate positioning;

[0027] S22, navigation: the path planning algorithm module plans the optimal path according to the hospital department database information and the current position of the user, the AR visual navigation module presents the path in the form of augmented reality on the real-time map interface, and the voice synthesis module informs the user of the navigation information in the form of voice, guiding the user to the target department;

[0028] S3, data processing and interaction process: the data and the hospital department database of the back-end layer provide basic data for path planning, the user behavior log records the user's operation behavior, the map APP and voice APP auxiliary system of the third-party service layer assist the system operation, and the modules interact with each other, such as the multi-sensor data fusion module fuses image and sensor data, and shares data with the path planning, AR navigation and other modules, to ensure the accuracy and smoothness of the navigation and guidance services.

[0029] Compared with the related art, the intelligent guidance system based on AR vision and voice navigation provided by the present application has the following beneficial effects:

[0030] The application provides an intelligent guide system based on AR vision and voice navigation, accurate navigation, and leading the way without obstacles: through advanced AR vision navigation technology, the APP provides an accurate navigation path for patients, greatly reducing the possibility of navigation errors, seamlessly integrating virtual and reality, and making patients feel like walking on a familiar map in the hospital, with every step full of confidence;

[0031] Optimizing user experience and bringing touchable care: voice navigation function brings unprecedented convenience to users, especially for those with visual impairment, the elderly or users with hand inconvenience, this service is like a thoughtful guide, leading them to easily move around the hospital, enjoying barrier-free medical experience;

[0032] Realize information synchronization, accurate and accurate guidance: the deep integration of this system with the hospital information system ensures the real-time updating and accuracy of navigation information, patients do not need to worry about information lag, every guidance is based on the latest hospital layout and operation status, ensuring that patients can accurately reach the destination;

[0033] Personalized service, efficiency and warmth: the APP can provide personalized guide services according to the specific needs of users, which not only improves service efficiency, but also adds humanistic care, whether it is appointment time, department selection or examination process, intelligent optimization suggestions can be obtained, so that every patient can enjoy customized medical experience. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The first embodiment of the structure schematic diagram of the intelligent guide system based on AR vision and voice navigation provided by the application is shown in the figure.

[0035] Figure 2 For Figure 1 The flow structure schematic diagram of the intelligent guide system is shown in the figure.

[0036] Figure 3 The structure schematic diagram of the second embodiment of the intelligent guide system based on AR vision and voice navigation provided by the application is shown in the figure. DETAILED DESCRIPTION

[0037] The application will be further described below in combination with the drawings and embodiments.

[0038] First embodiment

[0039] Please refer to Figure 1 and Figure 2 , among them, Figure 1 The first embodiment of the structure schematic diagram of the intelligent guide system based on AR vision and voice navigation provided by the application is shown in the figure.Figure 2 For Figure 1 The flow structure diagram of the intelligent guidance system is shown. An intelligent guidance system based on AR vision and voice navigation includes:

[0040] A user interaction layer, a technical support layer, a data and background layer, a third-party service layer, and a hardware dependent layer;

[0041] The input end of the technical support layer is connected to the output end of the user interaction layer, and the input end of the data and background layer is connected to the output end of the technical support layer;

[0042] The input end of the third-party service layer is connected to the output end of the data and background layer, and the input end of the hardware dependent layer is connected to the output end of the third-party service layer;

[0043] The user interaction layer includes a guidance service module, an AR visual navigation module, a voice interaction module, and a real-time map interface.

[0044] The technical support layer includes SLAM positioning and modeling, a voice synthesis module, a voice recognition module, and a path planning algorithm module.

[0045] The data and background layer includes a hospital department database and a user behavior log.

[0046] The third-party service layer includes a map APP and a voice APP.

[0047] The hardware dependent layer includes an AR scene resource library, a mobile phone camera, a gyroscope / accelerometer, a microphone, and a speaker.

[0048] The AR visual navigation module uses SLAM positioning and modeling technology for map construction and real-time positioning, and realizes positioning and tracking based on adjacent image feature matching estimation algorithm. The voice interaction module receives user voice instructions through the voice recognition module and feeds back guidance information to the user in the form of voice through the voice synthesis module.

[0049] The path planning algorithm module is used to plan the optimal path from the current location to the target department according to the hospital department database information and the user location, and works cooperatively with the AR visual navigation module and the voice interaction module. The hospital department database is used to store the information of hospital department distribution and location, and the user behavior log is used to record the operation behavior of the user in the guidance process, providing data support for optimizing the guidance service.

[0050] The map APP provides basic map data, the voice APP assists the voice interaction module in voice processing, the mobile phone camera is used for image data acquisition, the gyroscope / accelerometer is used for auxiliary positioning, and the microphone and speaker are used for realizing the input and output of voice.

[0051] The SLAM positioning and modeling improves the accuracy and consistency of the map through the steps of initial pose estimation, map construction, loop closure detection, and global optimization when constructing the map. The AR scene repository stores scene data for augmented reality navigation, which cooperates with the SLAM positioning and modeling and the AR visual navigation module to present an intuitive navigation interface to the user.

[0052] In exploring alternatives to the APP with AR visual, voice navigation, and guidance:

[0053] (1) Alternative to visual navigation technology:

[0054] Using a classic 2D map navigation interface supplemented by clear voice prompts, the user is provided with an intuitive guidance service. The augmented reality technology is used, but it does not rely on visual recognition. Position information is obtained through text or voice input, and a more flexible navigation experience is achieved.

[0055] (2) Alternative to voice navigation technology:

[0056] Using advanced speech recognition technology, users can directly navigate and guide operations through voice commands. Natural language processing technology is integrated to enable the APP to analyze more complex and natural user commands and provide more intelligent navigation services.

[0057] (3) Alternative to the guidance system:

[0058] Users interact with online doctors by filling out questionnaires or exchanging text, and obtain professional guidance services. Artificial intelligence chat robots are used for preliminary illness inquiry and diagnosis, and appropriate doctors or experts are recommended based on user needs.

[0059] (4) Alternative to the user interface:

[0060] Through touch screen or mouse operation, users navigate and guide in an intuitive graphical interface. Mixed reality technology is combined to integrate the real world with virtual information, providing a more immersive and interactive guidance experience.

[0061] (5) Alternative to data processing technology:

[0062] Traditional database technology is used to store and manage user data, ensuring data security and reliability.

[0063] Cloud computing services are used to process user data through remote servers, enabling efficient data management and in-depth analysis.

[0064] (6) Alternative to hardware integration:

[0065] It does not rely on specific AR glasses or devices, but uses the AR application on the user's existing smartphone or tablet to achieve navigation functions, combining GPS and indoor positioning technologies such as Wi-Fi, Bluetooth, ultrasonic waves, etc., to provide a navigation solution without AR technology.

[0066] (7) Algorithm and model alternatives:

[0067] Using a rule-based system for triage, simplifying the operation process and reducing technical complexity, using deep learning technology, through different network architectures or training data sets, to achieve more accurate triage services.

[0068] Hospital panoramic map shooting:

[0069] In the AR navigation function of the hospital APP, the 3D hospital panoramic model and VR panoramic preview technology are ingeniously integrated to create a vivid and three-dimensional medical environment picture for patients. This APP with AR vision and voice navigation and triage is characterized by hospital panoramic map shooting, which obtains the positions of the name signs of each department in the hospital through the camera, uses multi-lens shooting technology to shoot all facilities inside and outside each department, i.e. uses multiple cameras to shoot from multiple angles for each facility, and accurately fits the images of these angles to generate a complete panoramic map of the department. Using the same method, the facilities panoramic Figure 1 is presented to provide a comprehensive and detailed hospital space map for patients.

[0070] (2) Current position prompt

[0071] According to the requirements of the "APP with AR vision and voice navigation and triage", a two-dimensional code is set up on the hospital hall, corridor, and department signs, and through Bluetooth beacon, Wi-Fi, UWB (Ultra Wide Band) or visual SLAM (Simultaneous Localization and Mapping) technology, a sub-meter level positioning accuracy is achieved to prompt the current position. For example, after the patient opens the APP, the camera scans the surrounding environment (such as the two-dimensional code on the corridor sign or department sign), combined with the Bluetooth beacon signal, the APP displays "Current position: Outpatient hall 2nd floor A area" in real time.

[0072] (3) Destination setting

[0073] According to the real-time position and destination, combined with the patient's registration information, the optimal path is automatically planned to avoid congested areas (such as dynamically adjusting through hospital flow monitoring data).

[0074] (4) AR vision and voice navigation

[0075] According to the requirements of the "APP with AR visual navigation and guidance", the establishment process of the hospital VR electronic map includes: the hospital information acquisition unit splices the panoramic map of each department of the hospital and the panoramic map of the facilities of the hospital building, draws a comprehensive three-dimensional hospital panoramic map, uses 3D scanning technology to accurately capture the contours and details of each facility in the hospital through a 3D scanner, and draws a 3D model of the hospital panoramic map and the facilities, so as to establish a hospital VR electronic map according to the hospital electronic map, the hospital panoramic map and the 3D model of the facilities.

[0076] According to the requirements of the "APP with voice navigation and guidance", the establishment process of the semantic relationship library includes: converting the positions of each department of the hospital in the hospital VR electronic map into coordinate point clouds in three-dimensional space, labeling each coordinate point in these coordinate point clouds, and describing the corresponding building names and hospital department information in detail, establishing a coordinate point cloud map of the hospital, and establishing a path topology map of the hospital according to the positional relationship of each building and each department in the hospital VR electronic map of the hospital. The coordinate point cloud map and the path topology map of the hospital together constitute the semantic relationship library.

[0077] Through the sensor data of the mobile phone gyroscope, accelerometer and the like, the positioning deviation is dynamically calibrated and corrected in real time to ensure the continuity of navigation. For example, the patient is prompted to turn 10 meters ahead, go up and down the stairs, take the elevator, etc. Virtual arrows, landmarks (such as "turn left to the radiology department") and floor signs are superimposed on the mobile phone screen, which are integrated with the real environment. For example, when the patient looks at the corridor, a floating arrow showing "go straight for 50 meters and turn right" is displayed on the screen. At the same time, voice broadcast navigation can be used. Through TTS (text-to-speech) technology, key nodes (such as "arrive at the elevator and go to the 3rd floor") are broadcast in real time. Support for multiple language switching, combined with the hospital department database, automatically match department, doctor or facility name.

[0078] (5) Data integration and processing

[0079] Integrate the existing database information of the hospital, including department layout, doctor information, appointment process, etc.; optimize the navigation path through algorithm to ensure that the patient reaches the destination in the shortest time and the most convenient way, realize AI personalized recommendation: such as you have an appointment for blood drawing at 9:00, please go to the 2nd floor laboratory first, the escalator is currently congested, you can choose to take the elevator, through the integration of hospital big data and medical system, combined with electronic medical record and queuing calling system, real-time prompt "next examination room is empty, please go as soon as possible".

[0080] User interaction layer (front-end interface)

[0081] (1) AR visual navigation module

[0082] Function: superimpose virtual navigation arrows and department identifiers onto the real scene in real time through the camera.

[0083] Example: User opens camera, AR interface displays "Turn left ahead → Gastroenterology Department".

[0084] (2) Voice Interaction Module

[0085] Function: Supports voice input (e.g., "I want to hang in the ophthalmology department") and voice broadcast navigation guidance.

[0086] Example: User says "Find the pediatric department", system replies "The pediatric department is on the 3rd floor east, and has planned a route for you".

[0087] (3) Guidance Service Module

[0088] Function: Recommend departments based on symptoms, display doctor schedules, and appointment booking entry.

[0089] Example: Input "headache", recommend "neurology department" and display the list of on-duty doctors.

[0090] (4) Real-time Map Interface

[0091] Function: 2D / 3D floor map display, mark key points such as toilets, elevators, etc.

[0092] 2. Technical Support Layer (Core Algorithm)

[0093] SLAM positioning and modeling

[0094] Technology: Real-time positioning and scene modeling (such as corridor and stair identification) through mobile phone sensors and cameras.

[0095] Speech Recognition (ASR)

[0096] Technology: Convert user speech into text, support dialects and medical terms (such as "cardiovascular department").

[0097] Path Planning Algorithm

[0098] Technology: Combine hospital map data to dynamically calculate the optimal path (avoid congestion areas).

[0099] 3. Data and Background Layer (Server)

[0100] Hospital Department Database

[0101] Data: Department location, doctor information, schedule, emergency status.

[0102] AR Scene Resource Library

[0103] Data: Preloaded 3D navigation markers (arrows, department icons), AR special effect materials.

[0104] User Behavior Log

[0105] Purpose: Record user search keywords, navigation path, optimize subsequent recommendations.

[0106] 4. Third-party service layer

[0107] Map AP: Call Gaode / Baidu Map to realize floor map rendering.

[0108] AR SDK: ARKit (iOS) or ARCore (Android) are used to realize AR rendering.

[0109] Voice API: Integrate voice services from Xunfei and Ali Cloud to support voice input and synthesis.

[0110] 5. Hardware-dependent layer

[0111] Mobile phone camera: used for AR scene capture and environment recognition.

[0112] Sensor: gyroscope and accelerometer assist in positioning direction and moving speed.

[0113] Microphone and speaker: realize voice interaction function.

[0114] Data flow diagram

[0115] User voice input→voice recognition (ASR)→guidance logic processing→path planning→AR navigation rendering.

[0116] Compared with the related art, the intelligent guidance system based on AR vision and voice navigation has the following beneficial effects:

[0117] The intelligent guidance system based on AR vision and voice navigation provided by the application has the following beneficial effects:

[0118] Optimize user experience and bring touchable care: the voice navigation function brings unprecedented convenience to users, especially for those with visual impairment, the elderly or users with hand inconvenience. This service is like a thoughtful guide, leading them to easily move around in every corner of the hospital and enjoy barrier-free medical experience.

[0119] Realize information synchronization and accurate guidance: the deep integration of the system with the hospital information system ensures real-time updating and accuracy of navigation information. Patients do not need to worry about information lag. Each guidance is based on the latest hospital layout and operating conditions to ensure that patients can accurately reach the destination.

[0120] Personalized service, efficiency and warmth: the APP can provide personalized guide service according to the specific needs of the user, which not only improves the service efficiency, but also adds humanistic care. Whether it is appointment time, department selection or examination process, intelligent optimization suggestions can be obtained, so that every patient can enjoy the customized medical experience.

[0121] Second embodiment

[0122] Please refer to Figure 3 Based on the first embodiment of the present application, the second embodiment of the present application provides another intelligent guide system based on AR vision and voice navigation. The second embodiment is only a preferred way of the first embodiment, and the implementation of the second embodiment will not affect the separate implementation of the first embodiment.

[0123] Specifically, the second embodiment of the present application provides an intelligent guide system based on AR vision and voice navigation, which is different from the first embodiment. The intelligent guide system based on AR vision and voice navigation, the SLAM positioning and modeling includes data acquisition module, front-end processing module, back-end optimization module, positioning and tracking module, multi-sensor data fusion module, error processing and recovery module.

[0124] The data acquisition module includes image data acquisition and sensor data acquisition. The image data acquisition is continuously photographed by the camera of the mobile phone in each area of the hospital, and the sensor data acquisition includes magnetometer sensing magnetic field auxiliary positioning, etc. The front-end processing module is used for image feature extraction and preprocessing of image data, and preprocessing of sensor data.

[0125] The back-end optimization module is used to improve the accuracy and consistency of the map through initial pose estimation, map construction, loop detection and global optimization. The positioning and tracking module estimates the position and attitude of the device in real time based on feature matching or visual odometry, tracks the position and attitude, updates the motion, and re-performs the initial pose estimation and map construction in the case of large error.

[0126] The multi-sensor data fusion module fuses image and sensor data with Kalman filtering algorithm, provides more accurate navigation and guide service, and the error processing and recovery module includes error detection and error recovery.

[0127] Intelligent prediction and personalized service: combined with user historical guidance data and behavior patterns, use machine learning algorithms to intelligently predict user needs, for example, if a user frequently queries a certain department, the system can actively push relevant department information and fast navigation when the user next enters the hospital, and according to user age, physical condition and other personalized information, provide targeted guidance services, such as planning barrier-free routes for users with limited mobility.

[0128] Deep integration with hospital information system: further strengthen the docking with internal HIS (hospital information system), LIS (laboratory information system), PACS (medical image archive and communication system) and other systems, in the guidance process, if the user needs examination and testing, the system can directly obtain the examination department location and queuing situation, plan the route with the shortest waiting time for the user, and update the waiting progress in real time.

[0129] Cross-floor and complex environment optimization in large hospitals: for large multi-story hospitals, introduce floor recognition technology, such as recognizing floor signs through image recognition, combining barometers and other sensors to sense floor changes, achieve more accurate cross-floor navigation, for complex building structures and dense crowd environments in hospitals, optimize SLAM algorithm and sensor fusion strategy, improve positioning and navigation performance in complex scenarios.

[0130] Remote guidance and extension of out-of-hospital services: expand system functions to outside the hospital, support remote guidance, users can upload symptom descriptions, image data, etc. through the system, the system uses artificial intelligence diagnosis model for preliminary analysis, and provides remote diagnosis suggestions and medical guidance in combination with hospital expert resources, and provides out-of-hospital rehabilitation guidance path planning for patients in the rehabilitation period, such as surrounding rehabilitation agency location navigation and other services.

[0131] The working principle of the intelligent guidance system based on AR vision and voice navigation provided by the application is as follows:

[0132] Data acquisition: the data acquisition module as the starting link, through the continuous shooting of the camera of the mobile phone to each area of the hospital to carry out image data acquisition, and using magnetometer and other sensors to sense magnetic field and other information to realize sensor data acquisition, all-around obtain environment and equipment related data, provide basis for subsequent processing;

[0133] Front-end processing: the front-end processing module extracts features from the collected image data, and carries out pretreatment operations such as grayscale, filtering, etc. to enhance image features and remove noise interference; pretreatments such as filtering and denoising, calibration error and deviation are carried out on sensor data to improve data quality, so that the data is more suitable for subsequent processing requirements;

[0134] Backend optimization: The backend optimization module uses the PnP algorithm to perform initial pose estimation based on preprocessed sensor data and image features; a map is constructed based on the pose and image features; loop closure is detected through image feature matching to avoid cumulative errors during map construction; global optimization is performed to improve the accuracy and consistency of the map, and an accurate hospital map model is constructed;

[0135] Positioning and tracking: The positioning and tracking module estimates the position and attitude of the device in real time based on feature matching or visual odometry, continuously tracks the position and attitude during user movement and updates in time, and re-performs initial pose estimation and map construction once there is a large error to ensure the accuracy of positioning;

[0136] Multi-sensor data fusion: The multi-sensor data fusion module uses Kalman filtering and other algorithms to fuse image data and sensor data, integrates the advantages of different sensors to improve data reliability, and at the same time, data fusion is performed with other modules such as path planning and AR navigation to provide more accurate and comprehensive data support for each module, ensuring the accuracy of navigation and guidance services;

[0137] Error handling and recovery: The error handling and recovery module monitors in real time whether errors occur during positioning and map construction, and once an error is detected, error recovery is performed through re-estimating the pose, adjusting the map parameters, or enabling a backup positioning method, to ensure that the system can still operate stably under abnormal conditions;

[0138] Module coordination and service output: Optimized data acquisition and processing modules lay the foundation for efficient data interaction and collaborative work between system modules, and each module processes data, such as the path planning module planning the optimal guidance path, the AR navigation module presenting augmented reality navigation guidance combined with positioning and map information, and the voice module providing voice guidance prompts, to jointly output high-quality intelligent guidance services for users.

[0139] Compared with related technologies, the intelligent guidance system based on AR vision and voice navigation provided by the present application has the following beneficial effects:

[0140] The intelligent guidance system based on AR vision and voice navigation provided by the present application improves the accuracy of positioning and navigation: through optimization of the data acquisition and processing module, the multi-sensor data fusion module fuses image and sensor data using Kalman filtering and other algorithms, and combined with the initial pose estimation, map construction, loop detection and global optimization of the backend optimization module, the accuracy of system positioning and navigation is effectively improved, providing more accurate department navigation services for users;

[0141] Stability enhancement: The positioning and tracking module tracks the device position and pose in real time, updates in motion in time, and can re-perform initial pose estimation and map construction when the error is large; the error processing and recovery module can monitor and process positioning and map construction errors in real time, these mechanisms enhance the stability of the system in complex environments and sudden situations, and ensure the continuous normal operation of the guide service;

[0142] Data processing efficiency improvement: The data acquisition module comprehensively acquires image and sensor data, the front-end processing module performs targeted preprocessing, and the back-end optimization module efficiently processes data to construct an accurate map, the entire process optimizes the data processing process, significantly improves the data processing efficiency compared with the first embodiment system, and makes the system response more rapid;

[0143] Optimization of system cooperation and service quality: The optimized data acquisition and processing module promotes more efficient data interaction and cooperative work between the modules of the system, and improves the guide service quality as a whole, which can better meet the needs of users to quickly and accurately find the target department in the hospital.

[0144] A method of an intelligent guide system based on AR vision and voice navigation, comprising the following steps:

[0145] S1, user interaction process: the user starts the intelligent guide system, enters the real-time map interface, and can issue voice instructions through the voice interaction module, such as querying the location of a department, the voice recognition module receives the instructions and converts them into text information to pass to other modules of the system;

[0146] S2, positioning and navigation process:

[0147] S21, positioning: the system uses SLAM positioning and modeling technology, combines image data collected by the phone camera and sensor data of the gyroscope / accelerometer for positioning, first acquires image and sensor data through the data acquisition module, pre-processes the data through the front-end processing module, and performs initial pose estimation and map construction through the back-end optimization module, the positioning and tracking module estimates the device position and pose in real time based on feature matching or visual odometry, and realizes accurate positioning;

[0148] S22, navigation: the path planning algorithm module plans the optimal path according to the hospital department database information and the current position of the user, the AR visual navigation module presents the path in the form of augmented reality on the real-time map interface, and the voice synthesis module informs the user of the navigation information in the form of voice, guiding the user to the target department;

[0149] S3, data processing and interaction process: data and hospital department database of the background layer provide basic data for path planning, user behavior log records user operation behavior, third party service layer map APP and voice APP auxiliary system operation, data interaction between modules, such as multi-sensor data fusion module fuses image and sensor data, and shares data with path planning, AR navigation and other modules, to ensure the accuracy and fluency of navigation and guidance service.

[0150] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An intelligent medical guidance system and method based on AR vision and voice navigation, characterized in that: include: User interaction layer, technical support layer, data and background layer, third-party service layer and hardware dependency layer; The input end of the technical support layer is connected to the output end of the user interaction layer, and the input end of the data and background layer is connected to the output end of the technical support layer; The input end of the third-party service layer is connected to the output end of the data and background layer, and the input end of the hardware dependency layer is connected to the output end of the third-party service layer; The user interaction layer includes a medical guidance service module, an AR visual navigation module, a voice interaction module, and a real-time map interface; The technical support layer includes SLAM positioning and modeling, speech synthesis module, speech recognition module, and path planning algorithm module; The data and backend layer include hospital department databases and user behavior logs; The third-party service layer includes map APP and voice APP; The hardware dependency layer includes an AR scene resource library, a mobile phone camera, a gyroscope / accelerometer, a microphone, and a speaker.

2. The intelligent medical guidance system based on AR vision and voice navigation according to claim 1 is characterized in that: The AR visual navigation module uses SLAM positioning and modeling technology to build maps and perform real-time positioning, and realizes positioning and tracking based on the adjacent image feature matching estimation algorithm. The voice interaction module receives user voice commands through the voice recognition module, and feeds back the guidance information to the user in the form of voice through the voice synthesis module.

3. The intelligent medical guidance system based on AR vision and voice navigation according to claim 1 is characterized in that: The path planning algorithm module is used to plan the optimal path from the current location to the target department based on the hospital department database information and the user's location, and work in conjunction with the AR visual navigation module and the voice interaction module. The hospital department database is used to store information on the distribution and location of hospital departments. The user behavior log is used to record the user's operational behavior during the guidance process, providing data support for optimizing the guidance service.

4. The intelligent medical guidance system based on AR vision and voice navigation according to claim 1 is characterized in that: The map APP provides basic map data, the voice APP assists the voice interaction module to perform voice processing, the mobile phone camera is used for image data acquisition, the gyroscope / accelerometer is used for auxiliary positioning, and the microphone and speaker are used to realize voice input and output.

5. The intelligent medical guidance system based on AR vision and voice navigation according to claim 1 is characterized in that: When the SLAM positioning and modeling are used to construct a map, the accuracy and consistency of the map are improved through the steps of initial pose estimation, map construction, closed-loop detection and global optimization. The AR scene resource library stores scene data for augmented reality navigation, and cooperates with the SLAM positioning and modeling and the AR visual navigation module to present an intuitive navigation interface to the user.

6. The intelligent medical guidance system based on AR vision and voice navigation according to claim 1 is characterized in that: The SLAM positioning and modeling includes a data acquisition module, a front-end processing module, a back-end optimization module, a positioning and tracking module, a multi-sensor data fusion module, and an error processing and recovery module.

7. The intelligent medical guidance system based on AR vision and voice navigation according to claim 6 is characterized in that: The data acquisition module includes image data acquisition and sensor data acquisition. Image data acquisition is performed by continuously shooting various areas of the hospital with a mobile phone camera. Sensor data acquisition includes magnetometer sensing of magnetic field assisted positioning, etc. The front-end processing module is used to extract image features and preprocess image data, and preprocess sensor data.

8. The intelligent medical guidance system based on AR vision and voice navigation according to claim 6 is characterized in that: The back-end optimization module is used to improve the accuracy and consistency of the map through initial pose estimation, map construction, closed-loop detection and global optimization. The positioning and tracking module estimates the position and posture of the device in real time based on feature matching or visual odometry, tracks the position and posture, and updates them during movement. In case of large errors, initial pose estimation and map construction need to be repeated.

9. The intelligent medical guidance system based on AR vision and voice navigation according to claim 6, characterized in that: The multi-sensor data fusion module uses a Kalman filter algorithm to fuse image and sensor data to provide more accurate navigation and guidance services. The error handling and recovery module includes error detection and error recovery.

10. A method of an intelligent medical guidance system based on AR vision and voice navigation, as described in claims 1 to 5, characterized in that: The following steps are involved: S1. User interaction process: The user opens the intelligent medical guidance system and enters the real-time map interface. The user can issue voice commands through the voice interaction module, such as querying the location of a department. The voice recognition module receives the command and converts it into text information to be transmitted to other modules of the system; S2. Positioning and navigation process: S21. Positioning: The system uses SLAM positioning and modeling technology, combined with image data collected by the mobile phone camera and sensor data from the gyroscope / accelerometer for positioning. First, the data acquisition module obtains image and sensor data. The front-end processing module pre-processes the data. The back-end optimization module performs initial pose estimation and map construction. The positioning and tracking module estimates the device position and posture in real time based on feature matching or visual odometry to achieve precise positioning. S22. Navigation: The path planning algorithm module plans the optimal path based on the hospital department database information and the user's current location. The AR visual navigation module presents the path on the real-time map interface in the form of augmented reality. At the same time, the speech synthesis module informs the user of the navigation information in the form of speech, guiding the user to the target department. S3. Data processing and interaction process: The hospital department database at the data and background layer provides basic data for route planning. The user behavior log records user operation behavior. The map APP and voice APP of the third-party service layer assist the system operation. Data interaction is carried out between modules. For example, the multi-sensor data fusion module fuses image and sensor data, and shares data with modules such as route planning and AR navigation to ensure the accuracy and smoothness of navigation and guidance services.

Images