Head mounted device and method for path planning

Abstract

The invention provides a head-mounted device and a method for path planning. The method comprises the following steps: obtaining a layout map, a point cloud map and historical path points of a field domain; obtaining a current point cloud through the sensor, and comparing the current point cloud with the point cloud map to generate a current path point; and generating a virtual path between the current path point and the historical path point according to the layout map and the point cloud map, and outputting the virtual path.

Description

Technical Field

[0001] This invention relates to extended reality (XR) technology, and more particularly to a head-mounted device and method for path planning. Background Technology

[0002] XR systems have been widely applied in various fields, such as indoor navigation. The head-mounted display (HMD) of an XR system can display relevant information about the user's location. For example, when aircraft maintenance personnel walk to a specific location in the cabin, the HMD can display information related to that location. However, XR systems require a period of positioning time to activate. If the user cannot complete all tasks in the area within a short time, the XR system may need to reposition itself when the user returns to the area and activates the system again to complete the remaining tasks. This results in a waste of the user's time. Summary of the Invention

[0003] The present invention provides a head-mounted device and method for path planning that can automatically guide a user to the location where the user finished their previous task.

[0004] This invention discloses a head-mounted device for path planning, comprising sensors, a storage medium, and a processor. The storage medium stores a layout map of a field, a point cloud map, and historical path points. The processor is coupled to the sensors and the storage medium, wherein the processor acquires the current point cloud through the sensors and compares the current point cloud with the point cloud map to generate a current path point, wherein the processor generates a virtual path between the current path point and historical path points based on the layout map and the point cloud map, and outputs the virtual path.

[0005] In one embodiment of the present invention, the processor generates a virtual path based on a first obstacle in the layout diagram and a second obstacle in the point cloud map.

[0006] In one embodiment of the present invention, the processor described above generates a virtual path based on a third obstacle in the current point cloud.

[0007] In one embodiment of the present invention, the processor obtains a first reference point corresponding to the layout map and a second reference point corresponding to the point cloud map, wherein the processor aligns the layout map and the point cloud map according to the first reference point and the second reference point to generate an overlay map, wherein the processor generates a virtual path according to the overlay map.

[0008] In one embodiment of the present invention, the overlay map indicated above corresponds to a first obstacle in the layout map and a second obstacle in the point cloud map.

[0009] In one embodiment of the present invention, the processor described above performs a path planning algorithm based on an overlay map to generate a virtual path.

[0010] In one embodiment of the present invention, the above-described path planning algorithm includes the dynamic window method.

[0011] In one embodiment of the present invention, the head-mounted device further includes an inertial measurement unit. The inertial measurement unit is coupled to a processor, wherein the processor stores the work progress and corresponding waypoints in a storage medium based on the measurement results of the inertial measurement unit.

[0012] In one embodiment of the present invention, the storage medium further stores historical work progress corresponding to historical path points, wherein in response to the head-mounted device reaching a historical path point, the processor configures the virtual scene output by the head-mounted device according to the historical work progress.

[0013] In one embodiment of the present invention, the sensor described above includes at least one of the following: radar, light source, and image extraction device.

[0014] In one embodiment of the present invention, the layout diagram described above includes a computer-aided design diagram.

[0015] The present invention provides a method for path planning, applicable to a head-mounted device, comprising: acquiring a layout map of a field, a point cloud map, and historical path points; acquiring a current point cloud through a sensor, and comparing the current point cloud with the point cloud map to generate a current path point; and generating a virtual path between the current path point and historical path points based on the layout map and the point cloud map, and outputting the virtual path.

[0016] In one embodiment of the present invention, the step of generating a virtual path between the current path point and the historical path point based on the layout diagram and the point cloud map includes: generating a virtual path based on a first obstacle in the layout diagram and a second obstacle in the point cloud map.

[0017] In one embodiment of the present invention, the step of generating a virtual path based on a first obstacle in the layout diagram and a second obstacle in the point cloud map includes: generating a virtual path based on a third obstacle in the current point cloud.

[0018] In one embodiment of the present invention, the step of generating a virtual path between the current path point and the historical path point based on the layout diagram and the point cloud map includes: obtaining a first reference point corresponding to the layout diagram and a second reference point corresponding to the point cloud map; aligning the layout diagram and the point cloud map based on the first reference point and the second reference point to generate an overlay map; and generating a virtual path based on the overlay map.

[0019] In one embodiment of the present invention, the overlay map indicated above corresponds to a first obstacle in the layout map and a second obstacle in the point cloud map.

[0020] In one embodiment of the present invention, the step of generating a virtual path based on the overlay map includes: performing a path planning algorithm based on the overlay map to generate a virtual path.

[0021] In one embodiment of the present invention, the above-described path planning algorithm includes the dynamic window method.

[0022] In one embodiment of the present invention, the above method further includes: storing the work progress and corresponding path points based on the measurement results of the inertial measurement unit.

[0023] In one embodiment of the present invention, the method further includes: storing historical work progress corresponding to historical path points; and configuring the virtual scene output by the head-mounted device according to the historical work progress in response to the head-mounted device reaching a historical path point.

[0024] Based on the above, the head-mounted device of the present invention can store the user's location when completing the previous task as a historical waypoint. When the user returns to the field, the head-mounted device can overlay the layout map of the field with the point cloud map to generate an overlay map, and generate a virtual path on the overlay map based on the sensing results obtained by the sensors, so as to provide guidance services to the user through the virtual path. Attached Figure Description

[0025] Figure 1 A schematic diagram of a head-mounted device for path planning is shown according to an embodiment of the present invention.

[0026] Figure 2 A flowchart of execution path planning is illustrated according to an embodiment of the present invention.

[0027] Figure 3 A schematic diagram illustrating the generation of an overlay map is shown according to an embodiment of the present invention.

[0028] Figure 4 A schematic diagram illustrating the generation of a virtual path is shown according to an embodiment of the present invention.

[0029] Figure 5 A flowchart of a method for path planning is illustrated according to an embodiment of the present invention. Detailed Implementation

[0030] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element references are used in the drawings and description to denote the same or similar parts.

[0031] Figure 1A schematic diagram of a head-mounted device 100 for path planning is illustrated according to an embodiment of the present invention. The head-mounted device 100 may include a processor 110, a storage medium 120, a transceiver 130, and one or more sensors 140. In one embodiment, the head-mounted device 100 may further include a display 150 or an inertial measurement unit (IMU) 160. The head-mounted device 100 may be worn on a user's head and may provide the user with an XR environment (or XR scene), such as a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment.

[0032] Processor 110 may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microcontroller (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), or other similar elements or combinations thereof. Processor 110 may be coupled to storage medium 120, transceiver 130, sensor 140, display 150, and inertial measurement unit 160, and access and execute multiple modules and various applications stored in storage medium 120.

[0033] Storage medium 120 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar components or combinations thereof, used to store multiple modules or various applications executable by processor 110. In this embodiment, storage medium 120 may store information including a layout diagram of the field, a point cloud map of the field, or historical path points corresponding to the field.

[0034] Transceiver 130 transmits or receives signals wirelessly or via a wired connection. Transceiver 130 may also perform operations such as low-noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and similar functions.

[0035] Sensor 140 can be used to sense the environment surrounding head-mounted device 100 to generate point clouds. Sensor 140 is, for example, a radar, lidar, or image extraction device (e.g., a camera).

[0036] Display 150 can be used to display image data, such as providing an XR environment or XR scene to a user wearing head-mounted device 100. Display 150 may include a liquid-crystal display (LCD) or an organic light-emitting diode (OLED) display. In one embodiment, display 150 can provide an image beam to the user's eyes to form an image on the user's retina, allowing the user to see the XR scene created by head-mounted device 100.

[0037] The inertial measurement unit 160 can be used to measure acceleration, thereby obtaining the posture of the user wearing the head-mounted device 100. For example, the processor 110 can determine whether the user is looking down or up based on the measurement results of the inertial measurement unit 160.

[0038] Figure 2 A flowchart illustrating execution path planning is shown according to an embodiment of the present invention, wherein the flowchart may be derived from, for example... Figure 1 The head-mounted device 100 shown is implemented. In step S201, the head-mounted device 100 can be activated.

[0039] In step S202, the head-mounted device 100 may perform relocalization. Specifically, the processor 110 may sense the environment surrounding the head-mounted device 100 via the sensor 140 to obtain a current point cloud. The current point cloud contains environmental information of a portion of the user's location. For example, if an obstacle appears in the sensor 140 or the user's line of sight (LoS), the current point cloud may contain one or more points corresponding to that obstacle.

[0040] In step S203, the processor 110 of the head-mounted device 100 can load historical path points from the storage medium 120, where the historical path points correspond to the area where the head-mounted device 100 or the user is located. Historical path points can be used to indicate the user's location when completing a previous task on a layout diagram or point cloud map of the area. For example, if an aircraft maintenance worker finishes their work and turns off the head-mounted device 100 after inspecting equipment near the wing, the storage medium 120 can store historical path points corresponding to the location near the wing.

[0041] In step S204, processor 110 can generate a virtual path between the current path point and historical path points, and output the virtual path to the user through an output device (e.g., display 150 or a speaker coupled to processor 110). The user can move in the field according to the virtual path output by head-mounted device 100 to reach the historical path point.

[0042] Specifically, the processor 110 can compare the features of the current point cloud obtained by the sensor 140 with the features of the point cloud map of the field to generate a current path point, which can be used to indicate the current location of the head-mounted device 100 or the user on the point cloud map.

[0043] On the other hand, the processor 110 can obtain reference points in the layout diagram of the field and corresponding reference points in the point cloud map, and align the layout diagram and the point cloud map according to the two reference points to generate an overlay map, wherein the layout diagram includes, for example, a computer-aided design (CAD) drawing. Figure 3A schematic diagram illustrating the generation of an overlay map 330 is shown according to an embodiment of the present invention. The processor 110 may, for example, obtain reference point 311 on a layout map 310 and reference point 321 on a point cloud map 320 from user input information, wherein reference point 311 and reference point 321 correspond to the same location. For example, the user may use the entrance to a site as a reference point and, through instruction, mark reference point 311 on the layout map 310 and reference point 321 on the point cloud map 320. In one embodiment, an edge detection algorithm may be performed based on feature points in the point cloud map 320 to detect road / wall boundaries. The processor 110 may overlay the layout map 310 and the point cloud map 320 based on reference point 311 and reference point 321, along with edges in the point cloud map 320, to generate the overlay map 330. Any obstacles appearing in the layout map 310 or the point cloud map 320 may be mapped onto the overlay map 330.

[0044] After obtaining the overlay map 330, the processor 110 can generate a virtual path based on the overlay map 330. Specifically, the processor 110 can mark the current path point and historical path points on the overlay map 330, and generate a virtual path between the current path points and historical path points according to a path planning algorithm. Figure 4 For example, processor 110 can mark the current path point 331 and the historical path point 332 on the overlay map 330, and generate a virtual path 333 between the current path point 331 and the historical path point 332 according to a path planning algorithm. The path planning algorithm may include a dynamic window approach.

[0045] The virtual path 333 is generated taking into account obstacles appearing in the layout diagram 310 and obstacles appearing in the point cloud map 320. In one embodiment, when generating the virtual path 333, the processor 110 may further consider obstacles in the current point cloud. That is, the generation of the virtual path 333 takes into account obstacles detected in the field at different times. Thus, when the head-mounted device 100 guides the user based on the virtual path 333, the probability of the user encountering obstacles can be significantly reduced.

[0046] Upon reaching a historical waypoint, the processor 110 can load the historical work progress corresponding to the historical waypoint from the storage medium 120 and configure the virtual scene output by the head-mounted device 100 (e.g., displayed on the display 150) according to the historical work progress. For example, after a maintenance worker arrives at a historical waypoint representing a location near the wing according to the guidance of the head-mounted device 100, the display 150 can display a virtual scene to indicate to the maintenance worker the equipment that the maintenance worker has already repaired or the equipment that the maintenance worker has not yet repaired.

[0047] Back Figure 2 After the user completes their work, in step S205, the user can input a command to the head-mounted device 100 to pause or turn off the head-mounted device 100. In step S206, the processor 110 can respond to the user's input command and store the user's work progress and the corresponding path points (i.e., path points representing the head-mounted device 100 or the user's current location) as historical work records and historical path points in the storage medium 120.

[0048] In one embodiment, the processor 110 can store the work record and corresponding waypoints in the storage medium 120 based on the measurement results of the inertial measurement unit 160. For example, when the measurement results of the inertial measurement unit 160 indicate that the user should adopt a specific posture (e.g., maintaining a head-down posture for a period of time exceeding a threshold), the processor 110 can store the user's current work record and the waypoints corresponding to the user's current position in the storage medium 120 based on the measurement results. That is, the user can adopt a specific posture to record work progress and waypoints.

[0049] Figure 5 A flowchart of a method for path planning is illustrated according to an embodiment of the present invention, wherein the method may be derived from, for example... Figure 1 The head-mounted device 100 shown is implemented. In step S501, a layout map, a point cloud map, and historical path points of the field are obtained. In step S502, the current point cloud is obtained through sensors, and the current point cloud is compared with the point cloud map to generate the current path point. In step S503, a virtual path between the current path point and the historical path points is generated based on the layout map and the point cloud map, and the virtual path is output.

[0050] In summary, the head-mounted device of the present invention can store the user's location when completing a previous task as a historical path point. When the user returns to the field, the head-mounted device can overlay the layout map of the field with a point cloud map to generate an overlay map, and generate a virtual path on the overlay map based on the sensing results obtained by the sensors. The head-mounted device can use the virtual path to guide the user back to the historical path point. Since the overlay map contains information such as the layout map and the point cloud map of the field, the head-mounted device can ensure that the virtual path avoids obstacles located in the layout map or the point cloud map when generating the virtual path. In addition, the head-mounted device can automatically record the user's work progress and location when the user performs a specific action. When the user returns to that location to complete the remaining work, the head-mounted device can display relevant information about the work progress to the user, making it easier for the user to complete the work.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A head-mounted device for path planning, characterized in that, include: sensor; Storage media, layout diagram of the storage site, point cloud map, and historical path points; as well as The processor is coupled to the sensor and the storage medium, wherein The processor acquires the current point cloud through the sensor and compares the current point cloud with the point cloud map to generate the current path point, wherein The processor generates a virtual path between the current path point and the historical path point based on the layout diagram and the point cloud map, and outputs the virtual path.

2. The head-mounted device according to claim 1, wherein... The processor generates the virtual path based on the first obstacle in the layout diagram and the second obstacle in the point cloud map.

3. The head-mounted device according to claim 2, wherein... The processor generates the virtual path based on a third obstacle in the current point cloud.

4. The head-mounted device according to claim 1, wherein... The processor obtains a first reference point corresponding to the layout map and a second reference point corresponding to the point cloud map, wherein... The processor aligns the layout map and the point cloud map based on the first reference point and the second reference point to generate an overlay map, wherein The processor generates the virtual path based on the overlay map.

5. The head-mounted device of claim 4, wherein the overlay map indicates a first obstacle corresponding to the layout map and a second obstacle corresponding to the point cloud map.

6. The head-mounted device according to claim 4, wherein... The processor executes a path planning algorithm based on the overlay map to generate the virtual path.

7. The head-mounted device according to claim 6, wherein the path planning algorithm includes the dynamic window method.

8. The head-mounted device according to claim 1, wherein the head-mounted device further comprises: An inertial measurement unit, coupled to the processor, wherein The processor stores the work progress and corresponding path points in the storage medium based on the measurement results of the inertial measurement unit.

9. The head-mounted device according to claim 1, wherein the storage medium further stores historical work progress corresponding to the historical path points, wherein In response to the head-mounted device reaching the historical path point, the processor configures the virtual scene output by the head-mounted device according to the historical work progress.

10. A method for path planning, applicable to head-mounted devices, characterized in that, include: Obtain the layout map, point cloud map, and historical path points of the site; The current point cloud is obtained through sensors, and the current point cloud is compared with the point cloud map to generate the current path point; as well as Based on the layout diagram and the point cloud map, a virtual path is generated between the current path point and the historical path point, and the virtual path is output.

Images