Obstacle sensing and visual impairment guiding method, head-mounted device and storage medium

Abstract

The invention provides an obstacle sensing and visual impairment guiding method, a head-mounted device and a storage medium, and the method comprises the steps: obtaining an RGB image collected by an RGB camera and a depth image collected by a depth camera in a current scene; predicting the RGB image by adopting a preset YOLOX model and predicting the depth image by adopting a preset SAM segmentation model to obtain the category, distance and position of each obstacle in the current scene; determining the speed of each obstacle by using a preset Kalman filtering estimation algorithm; performing obstacle danger coefficient evaluation according to the category, distance, position and speed of each obstacle to obtain an obstacle danger coefficient of each obstacle; and performing visual impairment guiding operation according to the obstacle danger coefficient. By applying the obstacle sensing and visual impairment guiding method, the danger coefficient of each obstacle can be evaluated, and the visual impairment guiding safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of blind guide technology, and more specifically, to an obstacle perception and visually impaired guidance method, a head-mounted device applying the obstacle perception and visually impaired guidance method, and a computer-readable storage medium applying the obstacle perception and visually impaired guidance method. Background Art

[0002] At present, there are some head-mounted visually impaired assistive devices on the market, such as guide glasses and guide caps. These devices use sensors such as RGB cameras to assist in detecting obstacles, and report obstacle information to the visually impaired through sound. The sensor data of these devices generally include IMU (inertial measurement unit) data, RGB camera data, and depth data (based on 3D ToF, structured light or binocular cameras). However, in today's obstacle recognition process, the image captured by the camera is usually calculated and processed based on defined objects, and it is impossible to identify and judge unspecified non-standard obstacles. In addition, the distance measurement performed by visual recognition is not a real measurement in physical space, and the accuracy is low. At the same time, it can only simply provide the distance measurement information of the obstacle, but cannot judge the threat of the obstacle, and cannot better provide guidance operations for the visually impaired.

[0003] Therefore, a more optimized obstacle perception method needs to be considered. Summary of the invention

[0004] The first object of the present invention is to provide an obstacle perception and visually impaired guidance method that can evaluate the risk factor of each obstacle and improve the safety of visually impaired guidance.

[0005] The second object of the present invention is to provide a head-mounted device that can evaluate the risk factor of each obstacle and improve the safety of guiding the visually impaired.

[0006] The third object of the present invention is to provide a computer-readable storage medium that can evaluate the risk factor of each obstacle and improve the safety of visually impaired guidance.

[0007] In order to achieve the above-mentioned first purpose, the obstacle perception and visually impaired guidance method provided by the present invention includes: in the current scene, obtaining the RGB image collected by the RGB camera and the depth image collected by the depth camera; using the preset YOLOX model to predict the RGB image and using the preset SAM segmentation model to predict the depth image, to obtain the category, distance and position of each obstacle in the current scene; using the preset Kalman filter estimation algorithm to determine the speed of each obstacle; evaluating the obstacle hazard coefficient according to the category, distance, position and speed of each obstacle, to obtain the obstacle hazard coefficient of each obstacle; and performing visually impaired guidance operations according to the obstacle hazard coefficient.

[0008] It can be seen from the above scheme that the obstacle perception and visually impaired guidance method of the present invention combines the YOLOX detection model and the SAM segmentation model to obtain the RGB image collected by the RGB camera and the depth image collected by the depth camera, and can obtain scene information from different angles, accurately obtain detailed information of the obstacle, including distance, category, position and other information, and improve the reliability and accuracy of obstacle information acquisition. Combined with the preset Kalman filter estimation algorithm, the speed of the obstacle is obtained. Combining this information, not only can the current state (category, distance, position) of the obstacle be perceived, but also its movement trend can be understood, and the risk factor of each obstacle can be comprehensively evaluated, further improving the system's dynamic perception of obstacles. At the same time, based on the accurate obstacle risk factor assessment results, the visually impaired guidance operation can provide personalized and effective guidance for the visually impaired according to the degree of danger of different obstacles, thereby improving the safety and convenience of their travel.

[0009] In a further scheme, a preset YOLOX model is used to predict the RGB image and a preset SAM segmentation model is used to predict the depth image, and the step of obtaining the category, distance and position of each obstacle in the current scene includes: using the preset YOLOX model to predict the category and bounding box of the obstacle on the RGB image, and obtaining the first obstacle information of each obstacle in the current scene; using the preset SAM segmentation model to predict the distance and segmentation mask of the obstacle on the depth image, and obtaining the second obstacle information of each obstacle in the current scene; the first obstacle information and the second obstacle information are merged to obtain the category, distance and position of each obstacle in the current scene.

[0010] It can be seen that the YOLOX model is good at target detection and can efficiently predict the category and bounding box of obstacles in RGB images, making full use of the visual information such as color and texture of RGB images to accurately determine the category of obstacles. The SAM segmentation model focuses on the processing of depth images and performs well in predicting obstacle distances and segmentation masks. Through depth image data, it can accurately determine the spatial distance and shape contour of obstacles. This division of labor allows each model to play its best role in its field of expertise, improving the accuracy and efficiency of obstacle information acquisition. By processing RGB images and depth images separately, multi-dimensional obstacle information can be obtained. The first obstacle information and the second obstacle information are fused, which can make up for the limitations of single model processing. Through information fusion, the advantages of the two complement each other, and the category, distance and position of each obstacle can be obtained more accurately, thereby providing more reliable basic data for subsequent risk factor assessment and visual impairment guidance.

[0011] In a further solution, the first obstacle information and the second obstacle information are fused to obtain the category, distance and position of each obstacle in the current scene, including: matching the bounding box with the segmentation mask to determine the same obstacle in the RGB image and the depth image; and fusing the first obstacle information and the second obstacle information of the same obstacle.

[0012] It can be seen that by matching the bounding box with the segmentation mask for obstacles, the same obstacle in the RGB image and the depth image can be accurately determined. This is because the bounding box in the RGB image marks the approximate range of the obstacle in the two-dimensional plane, while the segmentation mask of the depth image more carefully depicts the outline of the obstacle and its position in space. Combining and matching the two can effectively avoid misjudgments caused by factors such as viewing angle and occlusion, ensure that the subsequent fused information comes from the same actual obstacle, improve the corresponding accuracy of the information, and provide a more reliable basis for visually impaired guidance. After determining the same obstacle, the corresponding first obstacle information (category and bounding box related information) and the second obstacle information (distance and segmentation mask related information) are fused, which helps to ensure that the category, distance and location information of each obstacle obtained has high consistency and reliability.

[0013] In a further solution, the obstacle hazard factor is obtained by the following formula: ;in, , , , is the weight coefficient, is the influence function of relative velocity, is the influence function of distance, is the influence function of the category, is the influence function of the position.

[0014] In a further solution, the influence function of relative velocity is The calculation formula is as follows: ,in, is the relative speed, which is the difference between the speed of the obstacle and the speed of the head-mounted device. is the preset maximum possible relative speed, λ is the adjustment parameter; the influence function of distance The calculation formula is as follows: ,in, is the minimum safe distance, is the distance of the obstacle; the influence function of the category The calculation formula is as follows: ,in, is the weight of category c; the influence function of position The calculation formula is as follows: ,in, is the weight of position p.

[0015] It can be seen that calculating the risk factor of obstacles by comprehensively considering multiple key factors such as relative speed, distance, category and location can more realistically reflect the actual threat of obstacles to visually impaired people and provide a more reliable basis for visually impaired guidance. Using influence functions to quantify the impact of each factor on the risk factor can achieve a more accurate assessment. These functions can be designed and optimized based on actual data and experience to accurately reflect the relationship between different factors and the degree of danger, and can adapt to various types of obstacles and complex scenes to obtain reasonable risk factor assessment results.

[0016] In a further solution, the minimum safe distance is determined based on the speed of the head mounted device, and the speed of the head mounted device is positively correlated with the minimum safe distance.

[0017] It can be seen that the minimum safe distance is determined according to the speed of the head-mounted device, and the speed is positively correlated with the minimum safe distance, which can dynamically adapt to the movement speed of the visually impaired. When the visually impaired moves faster, the corresponding minimum safe distance will increase to ensure that there is enough time and space to react and brake when encountering obstacles; when the movement speed is slower, the minimum safe distance will also decrease, allowing the visually impaired to move more flexibly in the environment under the premise of relative safety. This dynamic adjustment method meets actual safety needs and improves the safety of the visually impaired at different movement speeds.

[0018] In a further solution, the minimum safe distance Obtained by the following formula: ,in, and is the preset correlation coefficient, is the speed of the head mounted device, Indicates the preset braking distance.

[0019] It can be seen from this that by considering the impact of speed on the distance required for visually impaired people to brake in different situations, personalized adjustments can be made based on factors such as the performance of different head-mounted devices, the reaction ability of visually impaired people, and specific usage scenarios to improve user experience.

[0020] In a further solution, the step of performing visually impaired guidance operations according to the obstacle hazard factor includes: planning a forward route and providing prompts according to the obstacle hazard factor.

[0021] It can be seen that planning the route and prompts according to the obstacle danger factor can accurately identify the danger level of different obstacles, and then plan the safest route and provide prompts for the visually impaired, effectively avoiding obstacles that may cause serious harm to them, providing convenient travel assistance for the visually impaired.

[0022] In order to achieve the second purpose of the present invention, the present invention provides a head-mounted device including a processor and a memory, the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned obstacle perception and visually impaired guidance method are implemented.

[0023] In order to achieve the third purpose of the present invention, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a controller, the steps of the above-mentioned obstacle perception and visually impaired guidance method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of an embodiment of the obstacle perception and visually impaired guidance method of the present invention.

[0025] Figure 2 It is a flowchart of the steps of obtaining the category, distance and position of each obstacle in the current scene in the embodiment of the obstacle perception and visually impaired guidance method of the present invention.

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0027] The obstacle perception and visually impaired guidance method of the present invention is a computer program applied to a head-mounted device, and is used to guide the visually impaired on the head-mounted device. Preferably, the head-mounted device includes an RGB camera (not shown), a depth camera (not shown) and an IMU module (not shown), the RGB camera is used to collect RGB images in the current scene environment, the depth camera is used to collect depth images in the current scene environment, and the IMU module is used to collect the moving speed of the head-mounted device.

[0028] Obstacle perception and visually impaired guidance method implementation example: like Figure 1As shown, in this embodiment, when the obstacle perception and visually impaired guidance method is working, step S1 is first executed to obtain the RGB image collected by the RGB camera and the depth image collected by the depth camera in the current scene. When the head-mounted device detects that it is currently in a scene where obstacle perception and visually impaired guidance are required, for example, the visually impaired person turns on the guidance function or the device enters a specific area, the RGB camera and the depth camera are triggered to start collecting images. The frequency of image acquisition can be controlled by setting a timer or based on event triggering, for example, a frame of image is collected every certain time (such as 0.1 seconds). Or when a scene change is detected, such as when the moving direction or speed of the visually impaired person changes, image acquisition is triggered. By simultaneously acquiring the RGB image collected by the RGB camera and the depth image collected by the depth camera, color and depth information are integrated. The RGB image provides rich texture and color features, which helps to identify the category of obstacles. The depth image directly provides the distance information between the object and the device. The two types of data complement each other and can improve the accuracy and comprehensiveness of obstacle perception.

[0029] After obtaining the RGB image and the depth image, step S2 is executed to predict the RGB image using the preset YOLOX model and the depth image using the preset SAM segmentation model to obtain the category, distance and position of each obstacle in the current scene. The YOLOX model is an efficient target detection model that can quickly and accurately detect objects in the image and determine their categories. The SAM (Segment Anything Model) segmentation model can process the depth image to further accurately determine the location and range of the obstacle. The preset YOLOX model is used to predict the RGB image, and the preset SAM model is used to predict the depth image, and the processing of the RGB image and the depth image is handed over to different preset models respectively. The YOLOX model is good at target detection and can efficiently predict the category and bounding box of obstacles in the RGB image, making full use of the color, texture and other visual information of the RGB image to accurately determine the category of the obstacle. The SAM segmentation model focuses on the processing of the depth image and performs well in predicting the obstacle distance and segmentation mask. The spatial distance and shape contour of the obstacle can be accurately determined through the depth image data. This division of labor allows each model to play its best role in the field it excels in, improving the accuracy and efficiency of obstacle information acquisition. At the same time, it can make up for the limitations of single model processing and more accurately obtain the type, distance and location of each obstacle.

[0030] In this embodiment, see Figure 2, when the preset YOLOX model is used to predict the RGB image and the preset SAM segmentation model is used to predict the depth image to obtain the category, distance and position of each obstacle in the current scene, step S11 is first performed to predict the category and bounding box of the obstacle in the RGB image using the preset YOLOX model to obtain the first obstacle information of each obstacle in the current scene. The RGB image can directly obtain the obstacle category and bounding box through the preset YOLOX model prediction. The YOLOX model is a well-known technology for those skilled in the art and will not be repeated here. The output of the preset YOLOX model can be expressed as: ,in, represents the category of the i-th obstacle, represents the bounding box of the ith obstacle, where are the coordinates of the center point of the bounding box, is the width of the bounding box, is the height of the bounding box.

[0031] Next, step S2 is executed to use the preset SAM segmentation model to predict the distance and segmentation mask of the obstacle in the depth image, and obtain the second obstacle information of each obstacle in the current scene. For the depth map, firstly, the depth map is transformed into Mapping to RGB space , The color space mapping function f can be a simple linear mapping or a more complex nonlinear mapping, depending on the range of the depth map and the required RGB representation. The rendered depth map is input into the SAM model, and the depth image is segmented by geometric relationships to obtain the distance of each obstacle. The output of the SAM model can be expressed as: ,in: represents the distance to the jth obstacle, represents the segmentation mask of the j-th obstacle.

[0032] After obtaining the first obstacle information and the second obstacle information, step S13 is executed to fuse the first obstacle information and the second obstacle information to obtain the category, distance and position of each obstacle in the current scene. In order to obtain detailed information of the obstacle, the first obstacle information and the second obstacle information need to be fused to improve the accuracy and efficiency of obtaining the obstacle information.

[0033] In this embodiment, the step of fusing the first obstacle information and the second obstacle information to obtain the category, distance and position of each obstacle in the current scene includes: matching the bounding box with the segmentation mask to determine the same obstacle in the RGB image and the depth image; fusing the first obstacle information and the second obstacle information of the same obstacle. By matching the bounding box with the segmentation mask, the same obstacle in the RGB image and the depth image can be accurately determined. Because the bounding box in the RGB image marks the approximate range of the obstacle in the two-dimensional plane, and the segmentation mask of the depth image more carefully depicts the outline of the obstacle and its position in space. Combining and matching the two can effectively avoid misjudgment caused by factors such as viewing angle and occlusion, ensure that the subsequent fused information comes from the same actual obstacle, improve the corresponding accuracy of the information, and provide a more reliable basis for visually impaired guidance. After determining the same obstacle, the corresponding first obstacle information (category and bounding box related information) and the second obstacle information (distance and segmentation mask related information) are fused, which helps to ensure that the category, distance and position information of each obstacle obtained have high consistency and reliability.

[0034] When matching the bounding box with the segmentation mask to determine the same obstacle in the RGB image and the depth image, the bounding box output by the preset YOLOX model is used. Segmentation mask output by the SAM model For matching, the intersection over union (IoU) is used to measure the matching degree. The specific formula is as follows: , when the IoU value of the bounding box and the segmentation mask exceeds the preset threshold, they are considered to correspond to the same obstacle, thus forming a matching pair. For each successfully matched obstacle, the category, distance and location information are fused and sorted to form the complete obstacle information of the obstacle: .

[0035] After obtaining the category, distance and position of each obstacle in the current scene, execute step S3 to determine the speed of each obstacle using the preset Kalman filter estimation algorithm. In order to prevent repeated detection of the same obstacle, each obstacle, including position and speed, is continuously tracked by the preset Kalman filter estimation algorithm. Assigning a unique identifier to each obstacle and recording its historical trajectory can effectively distinguish different obstacles and avoid repeated detection. During the tracking process, if multiple similar obstacle features are detected in the same area, duplicate detection is excluded by comparing their position, size, shape and other features, retaining only the most reliable detection result, and obtaining the speed of the tracked target.

[0036] Specifically, in the process of presetting the Kalman filter estimation algorithm, the state vector of the obstacle is assumed to be: ,in, and are the horizontal and vertical coordinates of the obstacle in the kth frame, is the horizontal velocity of the obstacle at the kth frame, are the vertical velocities of the obstacle in the kth frame. The state transfer equation is: , where: F is the state transfer matrix, is the process noise. The observation equation is: ,in, is the observation vector, which usually includes the location information of the obstacle; H is the observation matrix; is the observation noise.

[0037] After determining the speed of each obstacle, step S4 is executed to evaluate the obstacle hazard coefficient according to the category, distance, position and speed of each obstacle to obtain the obstacle hazard coefficient of each obstacle. By comprehensively considering multiple key factors such as relative speed, distance, category and position to evaluate the obstacle hazard coefficient, it can more truly reflect the actual threat of obstacles to visually impaired people and provide a more reliable basis for visually impaired guidance.

[0038] In this embodiment, the obstacle risk factor is obtained by the following formula: ;in, , , , is the weight coefficient, is the influence function of relative velocity, is the influence function of distance, is the influence function of the category, It is the influence function of the position. Using the influence function to quantify the impact of each factor on the risk factor can achieve a more accurate assessment. These functions can be designed and optimized based on actual data and experience to accurately reflect the relationship between different factors and the degree of risk, and can adapt to various types of obstacles and complex scenarios to obtain reasonable risk factor assessment results.

[0039] Specifically, the influence function of relative velocity is The calculation formula is as follows: ,in, is the relative speed. The relative speed is the difference between the speed of the obstacle and the speed of the head-mounted device. The speed of the head-mounted device can be detected by the IMU module. is to preset the maximum possible relative speed, and λ is an adjustment parameter used to control the growth rate of the function. Usually λ is a positive value, and a larger λ value will make the function grow faster. The greater the relative speed, the higher the risk factor.

[0040] Distance influence function The calculation formula is as follows: ,in, is the minimum safe distance, is the distance to the obstacle. The closer the distance, the higher the risk factor. In this embodiment, the minimum safety distance is determined according to the speed of the head-mounted device, and the speed of the head-mounted device is positively correlated with the minimum safety distance. Preferably, the minimum safety distance Obtained by the following formula: ,in, and is the preset correlation coefficient, is the speed of the head mounted device, Indicates the preset braking distance. The minimum safe distance is determined based on the speed of the head-mounted device, and the speed is positively correlated with the minimum safe distance, which can dynamically adapt to the moving speed of the visually impaired. When the visually impaired moves faster, the corresponding minimum safe distance will increase to ensure that there is enough time and space to react and brake when encountering obstacles; when the moving speed is slower, the minimum safe distance will also decrease, allowing the visually impaired to move more flexibly in the environment under the premise of relative safety. This dynamic adjustment method meets actual safety needs and improves the safety of the visually impaired at different moving speeds.

[0041] Category Influence Function The calculation formula is as follows: ,in, is the weight of category c. Different categories of obstacles have different potential dangers to visually impaired people. For example, vehicles are usually more threatening than ordinary obstacles such as tables and chairs. A category mapping table can be pre-defined to assign a basic danger coefficient weight to each category, for example:

[0042] Position influence function The calculation formula is as follows: ,in, It's location The position of obstacles has an important impact on the danger level of the visually impaired. For example, an obstacle directly in front of the visually impaired is more dangerous than one located to the side. The direction of the visually impaired person's advance can be divided into different areas, and a danger coefficient weight is set for each area. For example, the weight of the area directly in front is , the weight of the left area is , the weight of the right area is According to the position of the obstacle relative to the visually impaired person, determine the corresponding .

[0043] After obtaining the obstacle hazard coefficient of each obstacle, step S5 is executed to perform visually impaired guidance operations according to the obstacle hazard coefficient. By performing visually impaired guidance operations according to the obstacle hazard coefficient, personalized and effective guidance can be provided to visually impaired people according to the hazard levels of different obstacles, thereby improving their travel safety and convenience.

[0044] The steps of guiding the visually impaired according to the obstacle hazard coefficient include: planning the route and prompting according to the obstacle hazard coefficient. Planning the route and prompting according to the obstacle hazard coefficient can accurately identify the degree of danger of different obstacles, and then plan the safest route for the visually impaired and give prompts, effectively avoiding obstacles that may cause serious harm to them, and providing convenient travel assistance for the visually impaired. When giving prompts, sound, vibration or Braille can be used.

[0045] In an optional embodiment, when performing visual obstacle guidance operations according to the obstacle risk factor, the calculated risk factor of each obstacle is analyzed and divided into different risk levels. For example, the risk factor can be set to 0-3 as a low risk level, 3-7 as a medium risk level, and 7-10 as a high risk level. In this way, the risk level of each obstacle can be more intuitively understood so that corresponding guidance measures can be taken.

[0046] For obstacles at a low risk level, the visually impaired guidance system can issue relatively gentle prompts. For example, the head-mounted device can use a low-volume voice prompt to inform the visually impaired person "There is a low-risk obstacle ahead, please pay attention", and at the same time, it can be accompanied by slight vibration feedback to attract the attention of the visually impaired, but it will not cause excessive interference. The visually impaired person is guided to maintain the current direction and speed, and only needs to pay a little attention to the obstacle.

[0047] When the obstacle is at a medium-risk level, the guidance system needs to remind the visually impaired more clearly. The volume of the voice prompt can be appropriately increased, and the content can be more specific, such as "There is a medium-risk obstacle on the right ahead, it is close, please be careful." At the same time, the intensity of the vibration feedback can also be increased. Guide the visually impaired to adjust the direction of travel appropriately to avoid obstacles. The visually impaired can be instructed by voice to deviate to the left or right by a certain angle, or slow down the speed of travel, so as to pass the area safely.

[0048] For obstacles with high risk levels, the guidance system should immediately issue a strong alarm. The voice prompt is issued at a high volume and in an urgent tone, such as "There is a high-risk obstacle ahead, stop moving immediately", and the head-mounted device continuously vibrates with high intensity to give feedback, which attracts the attention of the visually impaired. At this time, the system should automatically plan a safe obstacle avoidance path and tell the visually impaired how to move in detail through voice, for example, "Please turn left 90 degrees, then walk forward 2 meters, then turn right 45 degrees and continue moving forward."

[0049] When the visually impaired person follows the guidance, the system continuously monitors the status of obstacles and the position of the visually impaired person to ensure that he or she can avoid danger safely. When the visually impaired person is moving, the visually impaired guidance system continuously monitors the changes in the danger factor of obstacles in the surrounding environment in real time. If the danger factor of an obstacle that was originally at a low danger level rises to a medium or high danger level due to a change in position or an increase in speed, the system should immediately adjust the guidance strategy and issue prompts and guidance of the corresponding level. Conversely, if the danger factor of the obstacle decreases, the intensity of the guidance should also be reduced accordingly.

[0050] In addition to voice and vibration feedback, the visually impaired guidance system can also be combined with other multimodal guidance methods. For example, using Braille display devices to present obstacle danger information and guidance instructions to the visually impaired in Braille, providing more intuitive tactile information. Or by linking with auxiliary devices such as smart crutches, controlling the vibration mode or direction indication of the crutches, the guidance effect can be further enhanced.

[0051] When multiple obstacles of the same category are detected clustered together, the device will summarize the number and category of these obstacles instead of reporting them one by one. For example, "There are three bicycles ahead", which can reduce unnecessary information and improve reporting efficiency.

[0052] In complex scenarios (such as shopping malls with large traffic), the device will simplify the reporting content to avoid too much information disturbing the user. For example, the device will uniformly report: "The current environment is complex, please walk carefully", which not only reminds users to pay attention to safety, but also avoids confusion caused by too much detailed information.

[0053] As can be seen from the above, the obstacle perception and visually impaired guidance method of the present invention combines the YOLOX detection model and the SAM segmentation model to obtain the RGB image collected by the RGB camera and the depth image collected by the depth camera, so as to obtain scene information from different angles, accurately obtain detailed information of the obstacle, including distance, category, position and other information, and improve the reliability and accuracy of obstacle information acquisition. Combined with the preset Kalman filter estimation algorithm, the speed of the obstacle is obtained. Combining this information, not only the current state (category, distance, position) of the obstacle can be perceived, but also its movement trend can be understood, and the risk factor of each obstacle can be comprehensively evaluated, further improving the system's dynamic perception of obstacles. At the same time, based on the accurate obstacle risk factor assessment results, the visually impaired guidance operation can provide personalized and effective guidance for the visually impaired according to the degree of danger of different obstacles, thereby improving the safety and convenience of their travel.

[0054] Head mounted device embodiment: The head mounted device of this embodiment includes a controller, and when the controller executes a computer program, the steps in the above obstacle perception and visually impaired guidance method embodiment are implemented.

[0055] For example, the computer program can be divided into one or more modules, one or more modules are stored in the memory and executed by the controller to complete the present invention. One or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the head mounted device.

[0056] The head mounted device may include, but is not limited to, a controller and a memory. Those skilled in the art will appreciate that the head mounted device may include more or fewer components, or a combination of certain components, or different components, for example, the head mounted device may also include an input and output device, a network access device, a bus, etc.

[0057] For example, the controller may be a central processing unit (CPU), other general controllers, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general controller may be a microcontroller or any conventional controller. The controller is the control center of the head-mounted device, and uses various interfaces and lines to connect the various parts of the entire head-mounted device.

[0058] The memory can be used to store computer programs and / or modules. The controller realizes various functions of the head-mounted device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. For example, the memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound receiving function, a sound conversion to text function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text data, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0059] Computer readable storage medium embodiment: If the module integrated in the head mounted device of the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the implementation of all or part of the process in the above obstacle perception and visually impaired guidance method embodiment can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer readable storage medium. When the computer program is executed by the controller, the steps of the above obstacle perception and visually impaired guidance method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The storage medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunications signals.

[0060] It should be noted that the above are only preferred embodiments of the present invention, but the design concept of the invention is not limited thereto, and any non-substantial modifications made to the present invention using this concept also fall within the protection scope of the present invention.

Claims

1. An obstacle perception and visually impaired guidance method, applied to a head-mounted device, characterized in that: The method comprises: In the current scene, obtain the RGB image collected by the RGB camera and the depth image collected by the depth camera; The RGB image is predicted using a preset YOLOX model and the depth image is predicted using a preset SAM segmentation model to obtain the category, distance and position of each obstacle in the current scene; Determine the speed of each obstacle using a preset Kalman filter estimation algorithm; Performing an obstacle hazard coefficient evaluation according to the type, distance, position and speed of each obstacle to obtain an obstacle hazard coefficient of each obstacle; Perform visually impaired guidance operations according to the obstacle hazard factor.

2. The obstacle perception and visually impaired guidance method according to claim 1, characterized in that: The steps of using a preset YOLOX model to predict the RGB image and using a preset SAM segmentation model to predict the depth image, and obtaining the category, distance and position of each obstacle in the current scene include: Using the preset YOLOX model to predict the category and bounding box of the obstacle in the RGB image, and obtaining the first obstacle information of each obstacle in the current scene; Using the preset SAM segmentation model to predict the distance and segmentation mask of the obstacle in the depth image, and obtaining the second obstacle information of each obstacle in the current scene; The first obstacle information and the second obstacle information are merged to obtain the category, the distance, and the position of each obstacle in the current scene.

3. The obstacle perception and visually impaired guidance method according to claim 2, characterized in that: The step of fusing the first obstacle information and the second obstacle information to obtain the category, the distance, and the position of each obstacle in the current scene includes: Performing obstacle matching between the bounding box and the segmentation mask to determine the same obstacle in the RGB image and the depth image; The first obstacle information and the second obstacle information of the same obstacle are merged.

4. The obstacle perception and visually impaired guidance method according to claim 1, characterized in that: The obstacle hazard coefficient is obtained by the following formula: ； in, , , , is the weight coefficient, is the influence function of relative velocity, is the influence function of distance, is the influence function of the category, is the influence function of the position.

5. The obstacle perception and visually impaired guidance method according to claim 4, characterized in that: The influence function of the relative velocity The calculation formula is as follows: ,in, is a relative speed, the relative speed being the difference between the speed of the obstacle and the speed of the head mounted device, is the preset maximum possible relative speed, λ is the adjustment parameter; The influence function of the distance The calculation formula is as follows: ,in, is the minimum safe distance, is the distance of the obstacle; The influence function of the class The calculation formula is as follows: ,in, is the weight of category c; The influence function of the position The calculation formula is as follows: ,in, is the weight of position p.

6. The obstacle perception and visually impaired guidance method according to claim 5, characterized in that: The minimum safety distance is determined according to the speed of the head mounted device, and the speed of the head mounted device is positively correlated with the minimum safety distance.

7. The obstacle perception and visually impaired guidance method according to claim 6, characterized in that: The minimum safety distance Obtained by the following formula: ,in, and is the preset correlation coefficient, is the speed of the head mounted device, Indicates the preset braking distance.

8. The obstacle perception and visually impaired guidance method according to any one of claims 1 to 7, characterized in that: The steps of performing the visually impaired guidance operation according to the obstacle risk factor include: Plan the route and provide prompts based on the risk factor of the obstacles.

9. A head mounted device, comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the steps of the obstacle perception and visually impaired guidance method as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the controller, the steps of the obstacle perception and visually impaired guidance method as described in any one of claims 1 to 8 are implemented.

Images