Head-mounted device and plane detection method in head-mounted device

Abstract

The application provides a head-mounted device and a plane detection method in the head-mounted device. The head-mounted device is worn on the head of a user, and continuously obtains scene images of a real world according to the direction in which the head of the user rotates to detect a plane in the scene, and then completes rendering of a virtual object in the head-mounted device and the like. In the process of plane detection, the head-mounted device can perform point cloud-plane updating on a 3D point cloud observed by each frame of image shot by a camera. Then, a plane obtained this time is detected according to the current 3D point cloud, and the plane obtained this time is merged with a plane detected before. After the merging and updating operation of plane detection for multiple times, the plane detected in the head-mounted device of the application can be more accurate, so as to ensure the accuracy of plane detection, reduce the deviation between the detected plane and the plane in the real world, and improve the experience of the user wearing the head-mounted device.

Description

Technical Field

[0001] This application relates to the fields of virtual reality and augmented reality technologies, and more particularly to a head-mounted device and a planar detection method in the head-mounted device. Background Technology

[0002] Virtual Reality (VR) technology uses computers to simulate virtual environments, creating a sense of immersion. Augmented Reality (AR) technology uses computers and other scientific technologies to simulate physical information such as sight, sound, smell, and touch that are difficult to experience in the real world within a certain time and space, and then applies it to the real world for display. VR devices are devices that use virtual reality technology to present virtual scenes to users. AR devices are devices that use augmented reality technology to present a combination of virtual and real scenes to users.

[0003] Plane detection is crucial for the use cases of head-mounted devices. For example, in AR applications, virtual objects can be placed on a plane in the real world detected by the AR device. The virtual objects are placed based on anchor points, which are achieved by clicking on the real-world plane rendered by the AR device. Therefore, the accuracy of plane detection in AR devices directly affects the rendering effect of virtual objects.

[0004] Currently, most planar detection methods in head-mounted displays are based on point clouds, which are determined using SLAM (Simultaneous Localization and Mapping) technology. The point cloud map generated by SLAM consists of sparse 3D points. Because the number of 3D points is limited, the accuracy of the point cloud is difficult to guarantee, and consequently, the accuracy of the planes detected from these point clouds is also difficult to guarantee. This can easily lead to discrepancies between the detected planes and the real-world planes, or the detection of multiple planes. Summary of the Invention

[0005] This application provides a head-mounted device and a planar detection method in the head-mounted device to solve the problem of low accuracy of current planar detection methods based on sparse 3D points.

[0006] In a first aspect, this application provides a head-mounted device, including: a camera configured to capture images of the environment in which the head-mounted device is located; and a controller configured to: acquire a current 3D point cloud observed in a current frame image; update an existing point cloud list using the current 3D point cloud to obtain a current point cloud list; update planes in a plane list and remove abnormal planes according to a point cloud-plane mapping relationship using the current point cloud list to obtain a first plane list; the plane list stores planes obtained in each plane detection operation; perform plane detection using the current 3D point cloud to obtain a plurality of current planes in the current frame image; and update the first plane list using all current planes in the current frame image.

[0007] The head-mounted device of this application is worn on the user's head and continuously acquires real-world scene images based on the direction of the user's head rotation to detect planes in the scene, thereby completing the rendering of virtual objects in the head-mounted device. During the plane detection process, the head-mounted device can update the plane-point cloud list based on the 3D point cloud observed in each frame of the image captured by the camera. Then, it detects the plane obtained this time based on the currently observed 3D point cloud and merges the obtained plane with the previously detected planes. Through multiple plane detection merging and updating operations, the planes detected in the head-mounted device of this application can be more accurate, thereby ensuring the accuracy of plane detection, reducing the deviation between the detected plane and the real-world plane, and improving the user's experience of wearing the head-mounted device.

[0008] In some implementations, the controller is further configured to: obtain the ID of each 3D point in the current 3D point cloud; determine whether the ID exists in the point cloud list; if the ID exists in the point cloud list, update the position information of the 3D point corresponding to the ID in the point cloud list; if the ID does not exist in the point cloud, add the 3D point corresponding to the ID to the point cloud list.

[0009] In some implementations, the controller is further configured to: calculate current plane attributes using 3D points in the current point cloud list based on the point cloud-plane mapping relationship; and calculate historical plane attributes using 3D points in the point cloud list before the update; the plane attributes include the plane equation, plane normal vector, and plane center point of the target plane; calculate the change value of the plane attribute corresponding to the point cloud list using the current plane attribute and the historical plane attribute; when the change value of the plane attribute does not meet a preset condition, remove the plane corresponding to the plane attribute from the plane list, thereby obtaining a first plane list.

[0010] In some implementations, the controller is further configured to: project the currently observed 3D point cloud onto a preset plane normal vector using a dot product, and count the number of projection points on each projection plane; if the number of projection points on the projection plane exceeds a preset number, then determine all projection points on the projection plane as a candidate plane point group; remove outliers from the candidate plane point group to obtain a core point set; and use the core point set to calculate the current plane.

[0011] In some implementations, the controller is further configured to: determine the current normal vector of the current plane and the historical normal vector of the historical plane in the first plane list; calculate the normal vector angle between the historical normal vector and the current normal vector; and if the normal vector angle is less than a preset angle threshold, merge the current plane with the historical plane.

[0012] In some implementations, the controller is further configured to: determine the historical center point of the historical plane in the first plane list; calculate the distance from the historical center point to the current plane; and if the distance is less than a preset distance threshold, merge the current plane with the historical plane.

[0013] In some implementations, the controller is further configured to: determine the proportion of the number of overlapping 3D points between the current plane and the historical planes in the first plane list to the total number of overlapping 3D points on the historical plane; if the proportion of overlapping points is greater than a preset proportion threshold, then merge the current plane with the historical plane.

[0014] In some implementations, the controller is further configured to: add the current plane to the first plane list if all the angles between all historical normal vectors and the current normal vector are greater than or equal to a preset angle threshold; or if the distances from all historical center points to the current plane are greater than or equal to a preset distance threshold; or if the proportion of the number of overlapping points between all historical planes and the current plane is less than or equal to a preset proportion threshold.

[0015] Secondly, this application also provides a plane detection method in a head-mounted device, comprising: acquiring a current 3D point cloud observed in a current frame image; updating an existing point cloud list using the current 3D point cloud to obtain a current point cloud list; updating planes in a plane list and removing abnormal planes according to a point cloud-plane mapping relationship using the current point cloud list to obtain a first plane list; the plane list stores planes obtained in each plane detection operation; performing plane detection using the current 3D point cloud to obtain a plurality of current planes in the current frame image; and updating the first plane list using all current planes in the current frame image.

[0016] In some implementations, the method further includes: obtaining the ID of each 3D point in the current 3D point cloud;

[0017] Determine if the ID exists in the point cloud list; if the ID exists in the point cloud list, update the position information of the 3D point corresponding to the ID in the point cloud list; if the ID does not exist in the point cloud, add the 3D point corresponding to the ID to the point cloud list.

[0018] The planar detection method in the head-mounted device of the second aspect of this application can be applied to the head-mounted device of the first aspect and is specifically implemented by the controller in the head-mounted device. Therefore, the beneficial effects of the planar detection method in the head-mounted device of the second aspect are the same as the beneficial effects of the head-mounted device of the first aspect, and will not be repeated here. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of a display system structure including a virtual reality device is shown according to some embodiments;

[0021] Figure 2 A schematic diagram of a VR scene global interface according to some embodiments is shown;

[0022] Figure 3 A schematic diagram of the recommended content area of ​​the global interface according to some embodiments is shown;

[0023] Figure 4 A schematic diagram of the application shortcut operation entry area of ​​the global interface according to some embodiments is shown;

[0024] Figure 5 A schematic diagram of a floating object in a global interface according to some embodiments is shown;

[0025] Figure 6 A schematic flowchart illustrating a planar detection operation in a head-mounted device according to some embodiments is shown;

[0026] Figure 7 This diagram illustrates a flowchart of updating a point cloud list in a head-mounted device according to some embodiments;

[0027] Figure 8 A schematic flowchart illustrating one method for obtaining a first list of planes in a head-mounted device according to some embodiments is shown;

[0028] Figure 9 A schematic diagram of a process for planar detection using 3D point clouds in a head-mounted device according to some embodiments is shown;

[0029] Figure 10 A schematic diagram of a real-world object with a vertical plane is shown according to some embodiments;

[0030] Figure 11 A diagram showing the correspondence between the number of projection planes and projection points according to some embodiments is provided.

[0031] Figure 12 A schematic diagram of a first process for updating a first plane list in a head-mounted device according to some embodiments is shown;

[0032] Figure 13 A schematic diagram of a second process for updating a first plane list in a head-mounted device according to some embodiments is shown;

[0033] Figure 14 A third schematic diagram of a process for updating a first plane list in a head-mounted device according to some embodiments is shown. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0035] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0036] It should be understood that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate, for example, to allow implementation in orders other than those given in the embodiments illustrated or described in this application.

[0037] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0038] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0039] Throughout this specification, references to "multiple embodiments," "some embodiments," "one embodiment," or simply "embodiment" indicate that a specific feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Therefore, phrases such as "in multiple embodiments," "in some embodiments," "in at least another embodiment," or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner. Therefore, without limitation, a specific feature, structure, or characteristic shown or described in connection with one embodiment may be combined, in whole or in part, with features, structures, or characteristics of one or more other embodiments. Such modifications and variations are intended to be included within the scope of this application.

[0040] In this application embodiment, the head-mounted device 500 generally refers to a display device that can be worn on a user's face to provide the user with an immersive experience, including but not limited to virtual reality devices such as VR glasses, augmented reality (AR) devices, VR gaming devices, mobile computing devices, and other wearable computers. Some embodiments of this application use VR glasses as an example to illustrate the technical solution; it should be understood that the provided technical solution can also be applied to other types of head-mounted devices. The head-mounted device 500 can operate independently or be connected as an external device to other smart display devices, wherein the display device can be a smart TV, computer, tablet computer, server, etc.

[0041] The head-mounted device 500, when worn on a user's face, displays media assets, providing close-up images to the user's eyes for an immersive experience. To display the media assets, the head-mounted device 500 can include multiple components for displaying the images and for wearing on the face. Taking VR glasses as an example, the head-mounted device 500 can include a shell, positioning fasteners, an optical system, a display component, a posture detection circuit, and an interface circuit. In practical applications, the optical system, display component, posture detection circuit, and interface circuit can be housed within the shell to display the specific images; positioning fasteners are connected to both sides of the shell for wearing on the user's face.

[0042] When in use, the attitude detection circuit has built-in attitude detection components such as gravity acceleration sensor and gyroscope. When the user's head moves or turns, the user's attitude can be detected and the detected attitude data can be transmitted to the controller and other processing components, so that the processing components can adjust the specific screen content in the display component according to the detected attitude data.

[0043] like Figure 1 As shown, in some embodiments, the head-mounted device 500 can be connected to the display device 200 and a network-based display system can be built between the head-mounted device 500, the display device 200 and the server 400. Data interaction can be performed in real time between the head-mounted device 500, the display device 200 and the server 400. For example, the display device 200 can obtain media data from the server 400 and play it, and transmit specific screen content to the head-mounted device 500 for display.

[0044] The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. The specific type, size, and resolution of the display device are not limited. Those skilled in the art will understand that the display device 200 can be modified in terms of performance and configuration as needed. The display device 200 can provide broadcast television reception functionality and may also include, but is not limited to, intelligent network television functionality with computer support, including but not limited to, internet television, smart television, and Internet Protocol television (IPTV).

[0045] Display device 200 and head-mounted device 500 also communicate with server 400 via various communication methods. Display device 200 and head-mounted device 500 can communicate via local area network (LAN), wireless local area network (WLAN), and other networks. Server 400 can provide display device 200 with various content and interactive features. For example, display device 200 can interact by sending and receiving information, as well as electronic program guides (EPGs), receiving software updates, or accessing remotely stored digital media libraries. Server 400 can be a cluster or multiple clusters, and may include one or more types of servers. Other network services such as video-on-demand and advertising services can be provided through server 400.

[0046] During data interaction, users can operate the display device 200 via the mobile terminal 300 and the remote control 100. The mobile terminal 300 and the remote control 100 can communicate with the display device 200 via a direct wireless connection or a non-direct connection. Specifically, in some embodiments, the mobile terminal 300 and the remote control 100 can communicate with the display device 200 via direct connection methods such as Bluetooth or infrared. When sending control commands, the mobile terminal 300 and the remote control 100 can directly transmit the control command data to the display device 200 via Bluetooth or infrared.

[0047] In other embodiments, the mobile terminal 300 and the remote controller 100 can also access the same wireless network as the display device 200 via a wireless router to establish a non-direct connection communication with the display device 200 through the wireless network. When sending control commands, the mobile terminal 300 and the remote controller 100 can first send the control command data to the wireless router, and then the wireless router forwards the control command data to the display device 200.

[0048] In some embodiments, users can also use the mobile terminal 300 and the remote control 100 to interact directly with the head-mounted device 500. For example, the mobile terminal 300 and the remote control 100 can be used as controllers in a virtual reality scene to achieve functions such as motion-sensing interaction.

[0049] In some embodiments, the display component of the head-mounted device 500 includes a display screen and driving circuitry associated with the display screen. To present a concrete image and provide a stereoscopic effect, the display component may include two display screens, corresponding to the user's left and right eyes respectively. When presenting a 3D effect, the content displayed on the left and right screens will be slightly different, and may respectively display the left and right cameras used during the filming of the 3D source material. Because the user observes the image content with their left and right eyes, a more stereoscopic display image can be observed when the device is worn.

[0050] The optical system in the head-mounted device 500 is an optical module composed of multiple lenses. Positioned between the user's eyes and the display screen, the optical system increases the optical path through the refraction of light signals by the lenses and the polarization effect of polarizers on the lenses, ensuring that the content displayed is clearly presented within the user's field of vision. Furthermore, to accommodate different users' visual conditions, the optical system also supports focusing. This involves adjusting the position of one or more lenses using a focusing component, changing the distance between the lenses, and thus altering the optical path and adjusting the image clarity.

[0051] The interface circuit of the head-mounted device 500 can be used to transmit interactive data. Besides transmitting posture data and display content data, in practical applications, the head-mounted device 500 can also connect to other display devices or peripherals through the interface circuit to achieve more complex functions through data interaction with the connected devices. For example, the head-mounted device 500 can connect to a display device through the interface circuit to output the displayed image to the display device in real time. As another example, the head-mounted device 500 can also connect to a controller through the interface circuit, which can be held and operated by the user to perform related operations in the VR user interface.

[0052] The VR user interface can be presented in various different UI layouts based on user operations. For example, the user interface may include a global interface, such as the global UI after the AR / VR terminal is started. Figure 2 As shown, the global UI can be displayed on the display screen of the AR / VR terminal or on the display device's monitor. The global UI may include a recommended content area 1, a business category extension area 2, an application quick operation entry area 3, and a floating element area 4.

[0053] Recommended content area 1 is used to configure different category tabs; within these tabs, media assets, special features, etc., can be configured; the media assets may include 2D films, educational courses, tourism, 3D, 360-degree panoramic, live streaming, 4K films, applications, games, and other businesses with media asset content, and these tabs can select different template styles and support simultaneous recommendation and arrangement of media assets and special features, such as... Figure 3 As shown.

[0054] In some embodiments, the content recommendation area 1 may further include a main interface and a sub-interface. For example... Figure 3 As shown, the part located in the center of the UI layout is the main interface, while the parts located on either side of the main interface are the secondary interfaces. The main interface and the secondary interfaces can be used to display different recommended content. For example, based on the type of recommended content, the main interface can display 3D content; the secondary interface on the left can display 2D content; and the secondary interface on the right can display panoramic content.

[0055] Clearly, the main interface and sub-interfaces can display different content layouts while showing different business content. Furthermore, users can control the switching between the main and sub-interfaces through specific interactive actions. For example, by controlling the focus icon to move left and right, moving it to the right after it's at the far right of the main interface will center the sub-interface on the right side of the UI layout. At this point, the main interface switches to displaying panoramic content, the left sub-interface switches to displaying 3D content, and the right sub-interface switches to displaying 2D content.

[0056] Additionally, to facilitate user viewing, different display effects can be used to show the main interface and the secondary interface separately. For example, the transparency of the secondary interface can be increased to create a blurred effect, thus highlighting the main interface. Alternatively, the secondary interface can be set to grayscale while the main interface remains in color, further emphasizing the main interface.

[0057] In some embodiments, a status bar may also be provided at the top of the recommended content area 1. The status bar may contain multiple display controls, including commonly used options such as time, network connection status, and battery level. The content included in the status bar can be customized by the user; for example, weather or user avatars can be added. The content included in the status bar can be selected by the user to perform corresponding functions. For example, when the user clicks the time option, the head-mounted device 500 can display a time device window on the current interface or jump to a calendar interface. When the user clicks the network connection status option, the head-mounted device 500 can display a WiFi list on the current interface or jump to a network settings interface.

[0058] The content displayed in the status bar can vary depending on the specific settings of the item. For example, the time control can directly display the specific time text information, and display different text at different times; the battery control can display different pattern styles according to the current remaining battery level of the headset 500.

[0059] The status bar enables users to perform frequently used control operations, allowing for quick setup of the head-mounted device 500. Since the setup process for the head-mounted device 500 involves numerous steps, not all commonly used settings options can typically be displayed in the status bar. Therefore, in some embodiments, the status bar may also include extended options. When an extended option is selected, an extended window is displayed on the current interface, where multiple settings options can be further configured to implement other functions of the head-mounted device 500.

[0060] For example, in some embodiments, when the extended options are selected, a "Quick Center" option can be set in the extended window. After the user clicks the Quick Center option, the head-mounted device 500 can display the Quick Center window. The Quick Center window may include "Screenshot," "Screen Recording," and "Screen Projection" options, used to activate the corresponding functions respectively.

[0061] The Business Category Extension Area 2 supports configuring extended categories for different business types. When a new business type is added, a separate tab can be configured to display the corresponding page content. The extended categories in Business Category Extension Area 2 can also be sorted, and businesses can be removed from the platform. In some embodiments, Business Category Extension Area 2 may include the following content: Movies & TV, Education, Travel, Applications, and My Account. In some embodiments, Business Category Extension Area 2 is configured to display large business category tabs and supports configuring more categories; its icons are configurable, such as... Figure 3 As shown.

[0062] The application quick access area 3 can prioritize pre-installed applications for promotional purposes, and supports configuring special icon styles to replace default icons. Multiple pre-installed applications can be specified. In some embodiments, the application quick access area 3 also includes left-hand and right-hand movement controls for selecting different icons, such as... Figure 4 As shown.

[0063] The floating area 4 can be configured to be located above the left or right diagonal side of a fixed area, and can be configured as a replaceable image or a jump link. For example, after receiving a confirmation operation, the floating area can jump to an application or display a specified function page, such as... Figure 5 As shown. In some embodiments, the suspended object may not be configured with a jump link and may be used simply for visual display.

[0064] In some embodiments, the global UI also includes a status bar at the top for displaying the time, network connection status, battery status, and more quick access points. When an icon is selected using the AR / VR terminal's controller, the icon will display a text prompt that expands to the left and right, and the selected icon will stretch and expand to the left and right according to its position.

[0065] For example, after selecting the search icon, the search icon will display the text "Search" and the original icon. Clicking the icon or text will take you to the search page. Another example is that clicking the favorites icon will take you to the favorites tab, clicking the history icon will display the history page by default, clicking the search icon will take you to the global search page, and clicking the message icon will take you to the message page.

[0066] In some embodiments, interaction can be performed through peripheral devices, such as the controller of an AR / VR terminal, which can operate the user interface of the AR / VR terminal, including a back button; a home button, which can be long-pressed to achieve a reset function; volume up and down buttons; and a touch area that can realize the functions of clicking, sliding, holding and dragging the focus.

[0067] In the foregoing embodiments, a VR device is a device that uses virtual reality technology to present virtual images to users. An AR device, on the other hand, is a device that uses augmented reality technology to present images that combine virtual and real elements to users. Furthermore, Virtual Reality (VR) technology simulates a virtual environment using computers, thereby providing a sense of immersion. Augmented Reality (AR) technology simulates physical information such as sight, sound, smell, and touch—essentially difficult to experience in the real world within a certain time and space—using computers and other scientific technologies, and then applies this simulation to the real world.

[0068] Plane detection is crucial for the use cases of head-mounted displays. For example, in AR applications, virtual objects can be placed on a plane in the real world detected by the AR device. The virtual objects are placed based on anchor points, which are achieved by clicking on the real-world plane rendered by the AR device. Therefore, the accuracy of plane detection in AR devices directly affects the rendering effect of virtual objects.

[0069] Currently, most plane detection methods in head-mounted displays are based on point clouds, which are determined using SLAM (Simultaneous Localization and Mapping) technology. The point cloud map generated by SLAM consists of sparse 3D points. Because the number of 3D points is limited, the accuracy of the point cloud is difficult to guarantee, and consequently, the accuracy of the planes detected from these point clouds is also difficult to guarantee. This can easily lead to discrepancies between the detected planes and the real-world planes, or the detection of multiple planes.

[0070] To address the aforementioned issues, this application provides a head-mounted device 500, which may include a camera and a controller.

[0071] The planes and placed virtual objects detected by the head-mounted device 500 are displayed in the rendered scene, which is a virtual scene constructed by the head-mounted device 500's rendering engine through a rendering program. For example, a head-mounted device 500 based on the Unity 3D rendering engine can construct a Unity 3D scene when displaying images. Various virtual objects and functional controls can be added to the Unity 3D scene to render specific usage scenarios. For instance, when playing multimedia resources, a display panel can be added to the Unity 3D scene to display the multimedia resource images. Simultaneously, virtual object models such as seats, speakers, and characters can be added to the Unity 3D scene to create a cinematic effect.

[0072] To output the rendered images, the head-mounted device 500 can also set up virtual cameras within the Unity 3D scene. For example, the head-mounted device 500 can set up a left-eye camera and a right-eye camera in the Unity 3D scene according to the position of the user's eyes. The two virtual cameras can simultaneously capture images of objects in the Unity 3D scene, thus outputting rendered images to the left and right displays respectively. To achieve a better immersive experience, the angles of the two virtual cameras in the Unity 3D scene can be adjusted in real time using the head-mounted device 500's posture sensors, allowing for real-time output of rendered images of the Unity 3D scene from different viewing angles as the user moves while wearing the head-mounted device 500.

[0073] In this embodiment of the application, when the environment in which the head-mounted device 500 is located is being observed, the virtual camera mentioned above needs to capture images of the real world.

[0074] Figure 6 A schematic flowchart illustrating a planar detection operation in a head-mounted device according to some embodiments is shown. Figure 6 As shown, the controller of the head-mounted device 500 can be configured to perform the following steps:

[0075] Step S101: Obtain the current 3D point cloud observed in the current frame image.

[0076] In step S101, the obtained 3D point cloud is generated by the SLAM algorithm based on real-world images captured by a camera. Each 3D point in the 3D point cloud has attributes, which may include the position of the 3D point, i.e., its three-dimensional spatial coordinates, and may also include its own identifier ID. The ID of the same 3D point obtained from two consecutive frames is the same, but its position may change, i.e., the three-dimensional spatial coordinates of the 3D point may change.

[0077] Step S102: Update the existing point cloud list using the current 3D point cloud to obtain the current point cloud list.

[0078] The attributes of 3D points can also be used to indicate which plane the 3D point belongs to. In the head-mounted device 500, the 3D points generated by the SLAM algorithm each belong to their own plane, thus establishing a point cloud-plane mapping relationship. Multiple planes can exist in the environmental image, and a plane list can be created for each of these planes.

[0079] In two environmental images acquired by the camera, the position of the same 3D point may change, or the current frame image may have more 3D points than the previous frame image. In such cases, the point cloud list containing the 3D points needs to be updated.

[0080] Figure 7 A schematic diagram of a process for updating a point cloud list in a head-mounted device according to some embodiments is shown. Figure 7 As shown, when the head-mounted device 500 updates the point cloud list, the controller of the head-mounted device 500 can also be configured to perform the following steps:

[0081] Step S201: Obtain the ID of each 3D point in the current 3D point cloud.

[0082] In two consecutive frames captured by the camera, the position of the same 3D point may change, but the ID of the 3D point and the plane to which it belongs remain unchanged. 3D points generated from the current frame all possess the attributes of position, ID, and plane to which they belong. Therefore, obtaining the ID of a 3D point in the current frame facilitates comparison with 3D points in the previous frame, thereby determining how to update the point cloud list containing the 3D point.

[0083] Step S202: Determine whether the ID of the current 3D point exists in the point cloud list.

[0084] Step S203: If the ID exists in the point cloud list, update the position information of the 3D point corresponding to the ID in the point cloud list.

[0085] If the ID of the current 3D point exists in the point cloud list, it means that the current 3D point also exists on the plane of the previous frame image. This may mean that the position of the current 3D point has changed. Therefore, it is necessary to update the position information of the 3D point in the point cloud list corresponding to the plane of the previous frame image, thereby completing the point cloud list update operation and obtaining the current point cloud list.

[0086] Step S204: If the ID does not exist in the point cloud, then add the 3D point corresponding to the ID to the point cloud list.

[0087] If a 3D point that did not exist on a plane in the previous frame appears in the current frame, it means that a new 3D point has been added to that plane in the current frame. Therefore, the new 3D point needs to be added to the point cloud list corresponding to that plane to complete the point cloud list update operation and obtain the current point cloud list.

[0088] It is worth noting that, since the environmental image obtained in the aforementioned embodiment is a 3D point cloud, that is, a collection of several 3D points, when updating the point cloud list, it is necessary to perform the above steps S201-S204 on each 3D point in the current 3D point cloud to obtain all the updated point cloud lists corresponding to the current frame image.

[0089] Step S103: Based on the point cloud-plane mapping relationship, update the planes in the plane list using the current point cloud list and remove abnormal planes to obtain the first plane list.

[0090] In this embodiment, the operation of removing abnormal planes can be achieved by updating the plane's attributes using the updated 3D points in the current point cloud list, and then determining whether the updated plane attributes have changed significantly compared to the original plane attributes. If the change is significant, the plane corresponding to the plane attributes is determined to be an abnormal plane and needs to be removed from the plane list.

[0091] Because of the point cloud-plane mapping relationship, the plane properties of a plane can be determined using the point cloud corresponding to that plane. If the point cloud corresponding to that plane is updated, the updated plane properties can also be determined. These plane properties can include the plane's equation, its normal vector, its center point, and so on.

[0092] Figure 8 A schematic flowchart illustrating one method for obtaining a first list of planes in a head-mounted device according to some embodiments is shown. Figure 8 As shown, during the process of obtaining the first list of planes based on the aforementioned plane attributes, the controller of the head-mounted device 500 can also be configured to perform the following steps:

[0093] Step S301: Based on the point cloud-plane mapping relationship, calculate the current plane attributes using the 3D points in the current point cloud list, and calculate the historical plane attributes using the 3D points in the point cloud list before the update.

[0094] After step S301, a plane may correspond to a current plane equation and an updated plane equation, a current normal vector and an updated normal vector, and a current center point and an updated center point.

[0095] Step S302: Calculate the change value of the plane attribute corresponding to the point cloud list using the current plane attribute and the historical plane attribute.

[0096] Among them, based on the above-mentioned plane properties, the changes in plane properties may be due to changes in the current plane equation compared to the plane equation before the update, or an angle being formed between the current normal vector angle and the normal vector before the update, or a deviation in the position of the current center point relative to the center point before the update, etc.

[0097] Various changes are quantified into change values, thereby further determining whether the change values ​​of each plane attribute meet the preset conditions.

[0098] Step S303: When the change value of the plane attribute does not meet the preset conditions, the plane corresponding to the plane attribute is removed from the plane list to obtain the first plane list.

[0099] Because the 3D point cloud generated by SLAM is a sparse collection of points, the plane determined each time based on the diluted 3D points is not very accurate. Consequently, the determined plane attributes will also have some slight deviations. If the deviation is within the error range, then the change in the plane attributes is considered to be within an acceptable range, that is, the change in the plane attributes meets the preset conditions. For example, the angle between the normal vectors is less than a threshold, the distance between the center points is less than a threshold, etc.

[0100] If the deviation exceeds the error range, then the plane corresponding to the current point cloud list is considered to be an abnormal plane, meaning that the change in the plane's attributes does not meet the preset conditions. For example, the angle between the normal vectors is greater than or equal to a threshold, the distance between the center points is greater than or equal to a threshold, etc.

[0101] Step S104: Perform plane detection using the current 3D point cloud to obtain several current planes in the current frame image.

[0102] In step S103 above, an update operation was performed on the plane list, that is, abnormal planes were removed from the plane list. Afterwards, plane detection can be further performed using the current 3D point cloud to obtain all current planes in the current frame image.

[0103] Step S105: Update the first plane list using all current planes in the current frame image.

[0104] In this embodiment, the plane list stores the planes obtained in each plane detection operation, that is, it stores all the planes detected for the same environment. The plane list can be updated. In the first frame image, an initial plane list can be established, including the results of plane detection for the first frame image; then, in the second frame image, planes are detected again and the plane list corresponding to the first frame image is updated according to the current detection results, and so on, using each subsequent frame image to continuously update the plane list.

[0105] For example, after acquiring the previous frame image of the real-world environment, some 3D points in the observed 3D point cloud may belong to plane 1, some to plane 2, and some to plane 3. Planes 1, 2, and 3 can then form a plane list. When the camera acquires the current frame image of the real-world environment, planes 1, 2, and 3 can be detected again from the 3D point cloud in the current frame image. Therefore, the current planes 1, 2, and 3 can be used to update the plane list corresponding to the previous frame image.

[0106] In this embodiment of the application, updating the first plane list can be done by merging the currently detected plane with the same or similar planes in the first plane list, or by adding the currently detected plane that does not exist in the first plane list to the first plane list.

[0107] Through the above steps of abnormal plane removal, plane detection, and plane list update, the head-mounted device 500 can detect all planes in the current real-world scene with high accuracy and correctness, which can greatly reduce the deviation between the detected planes and the real-world planes.

[0108] In the foregoing embodiments, when performing plane detection, a preset plane normal vector for plane detection can be pre-configured, and then operations such as projection can be performed based on the preset plane normal vector. Figure 9 A schematic diagram of a process for planar detection using 3D point clouds in a head-mounted device according to some embodiments is shown. Figure 9 As shown, in some embodiments, the controller of the head-mounted device 500 may also be configured to perform the following steps:

[0109] Step S401: Perform a dot product projection of the currently observed 3D point cloud with the preset plane normal vector, and count the number of projection points on each projection plane.

[0110] Taking the detection of a vertical plane as an example, we first define the normal vector of the vertical plane, and then project the 3D points in the currently observed 3D point cloud onto the normal vector by multiplying them by the normal vector. Figure 10 A schematic diagram of a real-world object with a vertical plane is shown according to some embodiments. Figure 10 In the object image shown, projecting each 3D point onto the normal vector by multiplying each point onto the projection curve yields a projection curve. See details below. Figure 11 The graph shown. Figure 11 In the diagram, the horizontal axis of the curve represents... Figure 10 The curve represents the position of the object on each longitudinal section in the horizontal direction, with the vertical axis indicating the number of projection points on each longitudinal section. Each longitudinal section is the projection plane.

[0111] Step S402: If the number of projection points on the projection plane exceeds the preset number, then all projection points on the projection plane are determined as candidate plane point groups.

[0112] Specifically, in step S402, a histogram can be calculated based on the statistical results in step S401. The group interval of the histogram is defined as the planar resolution. For example, in this embodiment, the group interval can be set to 10cm. Then, the number of projection points on the projection plane at every 10cm interval is sorted from largest to smallest. If the number of projection points exceeds a preset number, the projection points on the projection plane are determined as a candidate planar point group.

[0113] Step S403: Remove outlier points from the candidate plane point group to obtain the core point set.

[0114] In the embodiments of this application, plane detection is performed based on 3D points generated by the SLAM algorithm. However, the 3D points generated by the SLAM algorithm are relatively sparse, which is not accurate enough and thus affects the accuracy of plane detection. Therefore, in step S403, the 3D points in the candidate plane point group can be further refined, that is, the core point set can be extracted.

[0115] Considering the potential for misclassification during histogram analysis, adjacent groups are also included in the extraction of the core point set to remove outliers. The specific process may include the following steps:

[0116] Step (a): Calculate the center point of the candidate plane point group, count the distance of each 3D point in the candidate plane to this center point, and remove 3D points whose distance exceeds the threshold from the candidate plane point group.

[0117] Step (b): Calculate the normal vector of each 3D point in the remaining candidate plane point group in step (a), and calculate the angle between each normal vector and the pre-set plane normal vector. 3D points with angles exceeding the threshold are further removed from the remaining candidate plane point group in step (a).

[0118] The calculation of the normal vector of a single 3D point requires first constructing a kd-tree for all points, searching for the neighboring points of the point through the kd-tree, calculating the covariance between the point and its neighboring points, and then using the PCA (Principal Components Analysis) algorithm to calculate the normal vector.

[0119] Step (c) continues to use the SMD algorithm to remove outliers from the remaining candidate planar points from step (b). The SMD algorithm uses Mahalanobis distance (instead of Euclidean distance) to calculate the distance between the 3D point and the reference center point.

[0120] In the SMD algorithm, a subset of points from the remaining candidate planar point group in step (b) is randomly selected first, and the covariance of this subset is calculated. Then, using this covariance, the Mahalanobis distance from all 3D points in the remaining candidate planar point group to the reference center point is calculated. After sorting the Mahalanobis distances, the subset of 3D points with the smallest distance is selected as the point set for the next iteration. This iteration continues until the difference in the determinant of the covariances calculated in two consecutive iterations is less than a threshold, thus selecting the core point set.

[0121] Step S404: Calculate the current plane using the core point set.

[0122] When calculating the current plane, PCA can be used to calculate the normal vector of the current plane and the distance from the origin to the current plane, thereby determining the plane equation of the current plane. It can also calculate the plane properties such as the pose and boundary of the current plane.

[0123] After the above steps S401-S404, the 3D points can be refined, thereby avoiding the problem that the low accuracy of sparse 3D generated by the SLAM algorithm affects the accuracy of plane detection.

[0124] In this embodiment, since a plane has attributes such as equation, normal vector, 3D point cloud, and center point, these attributes can be used to determine whether a current plane in the current frame image can be merged with a plane in the first plane list when updating the first plane list.

[0125] Figure 12 A schematic diagram of a first process for updating a first plane list in a head-mounted device according to some embodiments is shown. Figure 12 As shown, the controller of the head-mounted device 500 can also be configured to perform the following steps:

[0126] Step S501: Determine the current normal vector of the current plane and the historical normal vectors of the historical planes in the first plane list.

[0127] Since the first plane list may include multiple historical planes, it is necessary to determine the historical normal vector for each historical plane separately.

[0128] Step S502: Calculate the angle between the historical normal vector and the current normal vector.

[0129] For example, historical plane 1 corresponds to historical normal vector a, historical plane 2 corresponds to historical normal vector b, historical plane 3 corresponds to historical normal vector c, and the current normal vector of the current plane is N. Then the calculated angles between the normal vectors are the angle A between historical normal vector a and current normal vector N, the angle B between historical normal vector b and current normal vector N, and the angle C between historical normal vector c and current normal vector N.

[0130] Step S503: If the angle between the normal vectors is less than a preset angle threshold, then merge the current plane with the historical plane.

[0131] In this embodiment of the application, the preset included angle threshold can be selected between 10° and 20°.

[0132] Taking the case where there is only one plane in the first plane list that can be merged with the current plane as an example, the included angle A is less than the preset included angle threshold, while included angles B and C are both greater than or equal to the preset included angle threshold. In this case, the current plane can be merged with historical plane 1, thereby completing the update of the first plane list.

[0133] In some cases, the first plane list may contain multiple historical planes that can be merged with the current plane. For example, angles A and B may both be less than a preset angle threshold, while angle C may be greater than or equal to the preset angle threshold. In this case, the changes in other attributes of historical plane 1 and historical plane 2 can be compared with the preset thresholds to determine which historical planes can be merged.

[0134] Among them, the changes in other attributes can be the distance from the center point of the historical plane to the current plane, or the number of points on the historical plane that coincide with the current plane, etc.

[0135] Step S504: If all the angles between all historical normal vectors and the current normal vector are greater than or equal to a preset angle threshold, then add the current plane to the first plane list.

[0136] If there are no historical planes in the first plane list that can be merged with the current plane, for example, if the included angles A, B, and C are all greater than or equal to the preset included angle threshold, then the current plane can be considered a newly detected plane and can be added to the first plane list.

[0137] Figure 13 A second flowchart illustrating the updating of a first plane list in a head-mounted device according to some embodiments is shown. Figure 13 As shown, the controller of the head-mounted device 500 can also be configured to perform the following steps:

[0138] Step S601: Determine the historical center point of the historical plane in the first plane list.

[0139] Since the first plane list may include multiple historical planes, it is necessary to determine the historical center point for each historical plane separately.

[0140] Step S602: Calculate the distance from the historical center point to the current plane.

[0141] For example, historical plane 1 corresponds to historical center point D, historical plane 2 corresponds to historical center point E, and historical plane 3 corresponds to historical center point F. Then the calculated distances are the distance L1 from historical center point D to the current plane, the distance L2 from historical center point E to the current plane, and the distance L3 from historical center point F to the current plane.

[0142] Step S603: If the distance is less than a preset distance threshold, the current plane is merged with the historical plane.

[0143] In this embodiment of the application, the preset distance threshold can be selected between 5cm and 10cm.

[0144] Taking the case where there is only one plane in the first plane list that can be merged with the current plane as an example, the distance L1 is less than the preset distance threshold, while the distances L2 and L3 are both greater than or equal to the preset distance threshold. In this case, the current plane can be merged with historical plane 1, thereby completing the update of the first plane list.

[0145] In some cases, the first plane list may contain multiple historical planes that can be merged with the current plane. For example, distances L1 and L2 may both be less than a preset distance threshold, while distance L3 may be greater than or equal to the preset distance threshold. In this case, the changes in other attributes of historical plane 1 and historical plane 2 can be compared with the preset thresholds to determine which historical planes can be merged.

[0146] Among them, the changes in other attributes can be the angle between the historical plane normal vector and the current plane normal vector, or the number of points on the historical plane that coincide with the current plane, etc.

[0147] Step S604: If the distance from all historical center points to the current plane is greater than or equal to a preset distance threshold, then add the current plane to the first plane list.

[0148] If there are no historical planes in the first plane list that can be merged with the current plane, for example, if distances L1, L2, and L3 are all greater than or equal to a preset distance threshold, then the current plane can be considered a newly detected plane and can be added to the first plane list.

[0149] Figure 14 A third flowchart illustrating the updating of a first plane list in a head-mounted device according to some embodiments is shown. Figure 14 As shown, the controller of the head-mounted device 500 can also be configured to perform the following steps:

[0150] Step S701: Determine the proportion of the number of overlapping 3D points between the current plane and the historical planes in the first plane list to the total number of overlapping 3D points on the historical planes.

[0151] Since the first plane list may include multiple historical planes, it is necessary to determine the proportion of points overlapping with the current plane for each historical plane.

[0152] Step S702: If the proportion of overlapping points is greater than the preset point threshold, then merge the current plane with the historical plane.

[0153] For example, if historical plane 1 overlaps with the current plane 10 times, and historical plane 1 contains 30 3D points, then the ratio of overlapping points between historical plane 1 and the current plane is 10 / 30 ≈ 0.333; if historical plane 2 overlaps with the current plane 15 times, and historical plane 2 contains 20 3D points, then the ratio of overlapping points between historical plane 2 and the current plane is 15 / 20 = 0.75; if historical plane 3 overlaps with the current plane 5 times, and historical plane 3 contains 40 3D points, then the ratio of overlapping points between historical plane 3 and the current plane is 5 / 40 = 0.125.

[0154] In this embodiment of the application, the preset weight threshold can be selected between 0.5 and 0.6.

[0155] Taking a scenario where only one plane in the first plane list can be merged with the current plane as an example, the proportion of overlapping points between historical plane 1 and the current plane is greater than a preset weight threshold, while the proportions of overlapping points between historical plane 2 and historical plane 3 and the current plane are both less than the preset weight threshold. In this case, the current plane can be merged with historical plane 1, thereby completing the update of the first plane list.

[0156] In some cases, the first plane list may contain multiple historical planes that can be merged with the current plane. For example, the overlap ratio between historical plane 1 and historical plane 2 and the current plane may be greater than a preset weight threshold, while the overlap ratio between historical plane 3 and the current plane may be less than a preset weight threshold. In this case, the changes in other attributes of historical plane 1 and historical plane 2 can be compared with preset thresholds to further determine which historical planes can be merged from historical plane 1 and historical plane 2.

[0157] Among them, the changes in other attributes can be the distance from the center point of the historical plane to the current plane, or the angle between the normal vector of the historical plane and the normal vector of the current plane, etc.

[0158] Step S703: If the number of overlapping points between all historical planes and the current plane is less than or equal to a preset weight threshold, then add the current plane to the first plane list.

[0159] If there are no historical planes in the first plane list that can be merged with the current plane, for example, if the percentage of overlapping points between historical plane 1, historical plane 2, and historical plane 3 and the current plane is less than or equal to a preset percentage threshold, then the current plane can be considered a newly detected plane and can be added to the first plane list.

[0160] In addition to the above-mentioned methods of merging planes based on the angle between normal vectors, the proportion of coincidence points, and the plane distance between center points, this embodiment of the application can also determine whether the bounding boxes of the historical plane and the current plane intersect.

[0161] After updating the first plane list using all current planes in the current frame image, the updated first plane list can be merged pairwise again. The merging process can be performed according to the content of the aforementioned embodiment. Because traversing each pair of planes in the first plane list is time-consuming, this process can usually be performed at certain intervals to reduce the amount of computation.

[0162] As described above, the head-mounted device 500 in this embodiment can be worn on a user's head and continuously acquires real-world scene images based on the direction of the user's head rotation to detect planes in the scene, thereby completing the rendering of virtual objects in the head-mounted device 500. During plane detection, the head-mounted device 500 can update the point cloud-plane relationship based on the 3D point cloud observed in each frame of the image captured by the camera. Then, it detects the plane obtained this time based on the currently observed 3D point cloud and merges the obtained plane with previously detected planes. Through multiple plane detection merging and updating operations, the planes detected by the head-mounted device 500 in this embodiment are more accurate, thereby ensuring the accuracy of plane detection, reducing the deviation between the detected planes and the real-world planes, and improving the user experience of wearing the head-mounted device 500.

[0163] Furthermore, to address the issue of low accuracy in current planar detection methods based on sparse 3D points used in head-mounted devices 500, this application embodiment also provides a planar detection method for head-mounted devices. This method can be applied to the head-mounted device 500 of the aforementioned embodiments and is specifically implemented by the controller of the head-mounted device 500. The method may include the following steps:

[0164] Step S101: Obtain the current 3D point cloud observed in the current frame image;

[0165] Step S102: Update the existing point cloud list using the current 3D point cloud to obtain the current point cloud list;

[0166] Step S103: Based on the point cloud-plane mapping relationship, update the planes in the plane list using the current point cloud list and remove abnormal planes to obtain a first plane list; the plane list stores the planes obtained in each plane detection operation;

[0167] Step S104: Perform plane detection using the current 3D point cloud to obtain several current planes in the current frame image;

[0168] Step S105: Update the first plane list using all current planes in the current frame image.

[0169] Since the plane detection method in the head-mounted device of this application embodiment can be applied to the aforementioned head-mounted device 500, other details regarding the plane detection method in the head-mounted device of this application embodiment can be found in the aforementioned virtual reality and device 500 content, and will not be repeated here. Furthermore, the plane detection method in the head-mounted device of this application embodiment can also make the detected plane more accurate, thereby ensuring the accuracy of plane detection, reducing the deviation between the detected plane and the real-world plane, and improving the user's experience when wearing the head-mounted device.

[0170] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A head-mounted device, characterized in that, include: The camera is configured to capture images of the environment in which the head-mounted device is located; The controller is configured as follows: Obtain the current 3D point cloud observed in the current frame image; The existing point cloud list is updated using the IDs of the 3D points in the current 3D point cloud to obtain the current point cloud list. The update includes updating the position information of the 3D point corresponding to the ID or adding the 3D point corresponding to the ID to the point cloud list. Based on the point cloud-plane mapping relationship, the change value of plane attributes is calculated using the current point cloud list. The planes in the plane list are updated according to the change value of plane attributes, and abnormal planes are removed to obtain a first plane list. The plane list stores the planes obtained in each plane detection operation. The observed 3D point cloud is projected onto the preset plane normal vector by a dot product, and the number of projection points on each projection plane is counted. If the number of projection points on the projection plane exceeds a preset number, then all projection points on the projection plane are determined as a candidate plane point group. Outlier points are removed from the candidate planar point group to obtain the core point set; The core point set is used to calculate several current planes in the current frame image; Update the first plane list using all current planes in the current frame image.

2. The head-mounted device according to claim 1, characterized in that, The controller is also configured to: Obtain the ID of each 3D point in the current 3D point cloud; Determine if the ID exists in the point cloud list; If the ID exists in the point cloud list, then update the position information of the 3D point corresponding to the ID in the point cloud list; If the ID does not exist in the point cloud, then the 3D point corresponding to the ID is added to the point cloud list.

3. The head-mounted device according to claim 1, characterized in that, The controller is also configured to: Based on the point cloud-plane mapping relationship, the current plane attributes are calculated using the 3D points in the current point cloud list, and the historical plane attributes are calculated using the 3D points in the point cloud list before the update; the plane attributes include the plane equation, the plane normal vector, and the plane center point; Calculate the change value of the plane attribute corresponding to the point cloud list using the current plane attribute and the historical plane attribute; When the change value of the plane attribute does not meet the preset condition, the plane corresponding to the plane attribute is removed from the plane list, thereby obtaining the first plane list.

4. The head-mounted device according to claim 1, characterized in that, The controller is also configured to: Calculate the center point of the candidate plane point group, count the distance of each 3D point in the candidate plane to the center point, and remove the 3D points whose distance exceeds the first threshold from the candidate plane point group; In the candidate plane point group after removing 3D points whose distance exceeds the first threshold, the angle between the normal vector of each 3D point and the normal vector of the preset plane is calculated, and the 3D points whose angle exceeds the second threshold are removed from the candidate plane point group.

5. The head-mounted device according to claim 1, characterized in that, The controller is also configured to: Determine the current normal vector of the current plane and the historical normal vectors of the historical planes in the first plane list; Calculate the angle between the historical normal vector and the current normal vector; If the angle between the normal vectors is less than a preset angle threshold, then the current plane is merged with the historical plane.

6. The head-mounted device according to claim 5, characterized in that, The controller is also configured to: Determine the historical center point of the historical plane in the first plane list; Calculate the distance from the historical center point to the current plane; If the distance is less than a preset distance threshold, the current plane is merged with the historical plane.

7. The head-mounted device according to claim 5, characterized in that, The controller is also configured to: Determine the proportion of the number of overlapping 3D points between the current plane and the historical planes in the first plane list to the total number of overlapping 3D points on the historical plane; If the proportion of overlapping points is greater than a preset proportion threshold, then the current plane and the historical plane are merged.

8. The head-mounted device according to any one of claims 5-7, characterized in that, The controller is further configured to: If all the angles between all historical normal vectors and the current normal vector are greater than or equal to a preset angle threshold; Alternatively, if the distance from all historical center points to the current plane is greater than or equal to a preset distance threshold; Alternatively, if the proportion of overlap between all historical planes and the current plane is less than or equal to a preset proportion threshold, then the current plane is added to the first plane list.

9. A planar detection method in a head-mounted device, characterized in that, The method includes: Obtain the current 3D point cloud observed in the current frame image; The existing point cloud list is updated using the IDs of the 3D points in the current 3D point cloud to obtain the current point cloud list. The update includes updating the position information of the 3D point corresponding to the ID or adding the 3D point corresponding to the ID to the point cloud list. Based on the point cloud-plane mapping relationship, the change value of plane attributes is calculated using the current point cloud list. The planes in the plane list are updated according to the change value of plane attributes, and abnormal planes are removed to obtain a first plane list. The plane list stores the planes obtained in each plane detection operation. The observed 3D point cloud is projected onto the preset plane normal vector by a dot product, and the number of projection points on each projection plane is counted. If the number of projection points on the projection plane exceeds a preset number, then all projection points on the projection plane are determined as a candidate plane point group. Outlier points are removed from the candidate planar point group to obtain the core point set; The core point set is used to calculate several current planes in the current frame image; Update the first plane list using all current planes in the current frame image.

10. The method according to claim 9, characterized in that, The method further includes: Obtain the ID of each 3D point in the current 3D point cloud; Determine if the ID exists in the point cloud list; If the ID exists in the point cloud list, then update the position information of the 3D point corresponding to the ID in the point cloud list; If the ID does not exist in the point cloud, then the 3D point corresponding to the ID is added to the point cloud list.

Images