Display device and plane fitting method

Abstract

The application shows a display device and a plane fitting method. The anchor point set is obtained based on the depth image, and the anchor point set includes a plurality of anchor points meeting a preset condition. At least three points are selected from the anchor point set for plane fitting to obtain a plurality of fitting planes. A first fitting plane and a coplanar fitting plane of the first fitting plane are determined from the plurality of fitting planes. The second fitting plane is determined according to the first fitting plane and the coplanar fitting plane. The region growing algorithm is performed on the second fitting plane to obtain a target fitting plane. The plane fitting method indirectly determines the fitting plane through the anchor point set. In essence, the anchor points generated by the contact points of the moving object and the to-be-detected plane are subjected to plane fitting to indirectly determine the to-be-detected plane. Compared with the method of directly using the depth image obtained by the depth camera to determine the to-be-detected plane, the display device can more comprehensively and accurately detect the plane.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a planar fitting method. Background Technology

[0002] Augmented Reality (AR) technology is a technique that blends virtual visuals with real-world scenes. It can simulate and apply generated virtual objects, such as text, images, 3D models, and video footage, to real-world scenes, providing users with a viewing experience that combines virtual and real-world visuals.

[0003] AR applications are commonly found in display devices, used for AR motion-sensing games or AR interior design. Upon entering an AR application, the first step is to perform planar detection on the real-world scene to determine information such as the location of the ground and walls. For example, in AR motion-sensing games, planar detection can be used to depict augmented reality images on the walls of the player's room, or in AR interior design, planar detection can be used to distinguish between floors and obstacles.

[0004] Existing AR applications often use depth cameras, such as Time-of-Flight (TOF) depth cameras, structured-light depth cameras, and binocular depth cameras, to directly detect planes. However, in some scenarios, such as indoor home environments, the ground often has a specular reflection, resulting in incomplete ground position information obtained directly from depth images using depth cameras. This makes it impossible to obtain comprehensive and accurate ground position information. Therefore, there is an urgent need to provide a method that enables display devices to perform plane detection more comprehensively and accurately. Summary of the Invention

[0005] This application provides a display device and a plane fitting method, which can indirectly determine the fitting plane through a set of anchor points. In essence, it uses the anchor points generated by the contact points between the moving object and the plane to be detected to perform plane fitting in order to indirectly determine the plane to be detected. Compared with the method of directly determining the plane to be detected through a depth camera, this application enables the display device to perform plane detection more comprehensively and accurately.

[0006] This application provides a display device, including: a display; an image acquisition device configured to acquire a depth image of a real scene; the depth image including multiple consecutive frame images; and a controller configured to: receive a control command input by a user for entering an augmented reality application; in response to the control command, acquire a set of anchor points determined based on the depth image, the set of anchor points including multiple anchor points that meet preset conditions, the preset conditions being used to determine the contact points between a moving object and a plane to be detected; arbitrarily select at least three points from the set of anchor points for plane fitting to obtain multiple fitting planes; determine a first fitting plane and a coplanar fitting plane of the first fitting plane from the multiple fitting planes; determine a second fitting plane based on the first fitting plane and the coplanar fitting plane; and perform a region growing algorithm on the second fitting plane to obtain a target fitting plane.

[0007] In some embodiments of this application, the controller is further configured to: in the step of obtaining an anchor point set determined based on a depth image, obtain multiple inter-frame difference images of the depth image; the inter-frame difference images are obtained based on two adjacent frames of the depth image; obtain at least one target contour region based on the inter-frame difference images, the target contour region including a centroid and multiple pixels, the centroid being the mass center of the target contour region; select the pixel in the target contour region whose height is lower than the centroid and whose distance from the centroid is the farthest as an anchor point; generate an anchor point set based on all anchor points of the multiple inter-frame difference images, so as to determine a fitting plane through the anchor point set.

[0008] In some embodiments of this application, the controller is further configured to: in the step of determining the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes, obtain the normal vectors of all fitting planes in the multiple fitting planes; calculate the spatial angle between the normal vector of any fitting plane and the normal vectors of all remaining fitting planes pairwise to obtain the spatial angle between each fitting plane and all remaining fitting planes; obtain the number of spatial angles between each fitting plane and all remaining fitting planes that are less than a first angle, and determine the fitting plane with the most spatial angles as the first fitting plane; select from the remaining fitting planes of the first fitting plane the fitting plane with a spatial angle value less than the first angle as the coplanar fitting plane of the first fitting plane.

[0009] In some embodiments of this application, the controller is further configured to: in the step of determining a second fitting plane based on a first fitting plane and a coplanar fitting plane, obtain all anchor points on the first fitting plane and the coplanar fitting plane; perform least squares fitting on all anchor points on the first fitting plane and the coplanar fitting plane to obtain the second fitting plane.

[0010] In some embodiments of this application, the controller is further configured to: in the step of performing a region growing algorithm on the second fitting plane to obtain a target fitting plane, acquire multiple pixels in the depth image; add pixels whose vertical distance to the second fitting plane is less than a first distance to the second fitting plane to make the region of the second fitting plane grow; after the region of the second fitting plane has grown, acquire all anchor points of the second fitting plane; and perform least squares on the region of the second fitting plane to obtain the target plane region.

[0011] In some embodiments of this application, the controller is further configured to: after obtaining the target fitting plane, clear all anchor points in the target plane region and set the target plane region to a verification state; when the target plane region is in the verification state, acquire verification points; the verification points are pixels in the depth image; determine whether the proportion of the verification points in the target plane region is greater than or equal to a proportion threshold. If the proportion of the verification points in the target plane is greater than or equal to the proportion threshold, then the target plane region is the final plane region; if the proportion of the verification points in the target plane is less than the proportion threshold, then reacquire the target plane region.

[0012] In some embodiments of this application, the controller is further configured to: in the step of obtaining verification points, perform the FAST feature point detection algorithm on the target planar region in the color image to obtain multiple feature points; obtain the depth values ​​corresponding to the multiple feature points in the depth image; if the depth value corresponding to any feature point in the depth image is not 0, then the feature point is used as a candidate point; iterate multiple times to obtain multiple candidate points from multiple frames of color images; determine the candidate point attributes by obtaining the BRIEF descriptor of the candidate points; match different candidate point attributes, and determine the candidate point with the smallest change among the multiple frames of color images as the verification point.

[0013] In some embodiments of this application, the controller is further configured to: in the step of acquiring verification points, after clearing all anchor points in the target plane region, reacquire the set of anchor points in the depth image to obtain a new set of anchor points; and determine all anchor points in the new set of anchor points as verification points.

[0014] In some embodiments of this application, a plane fitting method is also provided, comprising: obtaining an anchor point set from a depth image, the anchor point set including multiple anchor points that meet preset conditions, the preset conditions being used to determine the contact points between a moving object and a plane to be detected; arbitrarily selecting at least three points from the anchor point set to perform plane fitting, thereby obtaining multiple fitting planes; determining a first fitting plane and a coplanar fitting plane from the multiple fitting planes; determining a second fitting plane based on the first fitting plane and the coplanar fitting plane; and performing a region growing algorithm on the second fitting plane to obtain a target fitting plane.

[0015] In this way, the plane fitting method indirectly determines the fitting plane through the set of anchor points. In essence, it performs plane fitting on the contact points between the moving object and the plane to be detected to indirectly determine the plane to be detected. Compared with the method of directly determining the plane to be detected through a depth camera, this application enables the display device to perform plane detection more comprehensively and accurately. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating an operational scenario between a display device and a control device according to some embodiments is shown;

[0018] Figure 2 A configuration block diagram of a control device 100 according to some embodiments is shown;

[0019] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0020] Figure 4 A software configuration block diagram of a display device 200 according to some embodiments is shown;

[0021] Figure 5 A schematic diagram of an application icon control interface display of a display device 200 according to some embodiments is shown;

[0022] Figure 6 The diagram illustrates AR application scenarios according to some embodiments;

[0023] Figure 7 A schematic diagram of a planar detection scene using a TOF depth camera is shown according to some embodiments;

[0024] Figure 8 This diagram illustrates the planar detection effect of a TOF depth camera according to some embodiments;

[0025] Figure 9 A comparative diagram of the principles of structured light depth cameras and TOF depth cameras according to some embodiments is shown;

[0026] Figure 10 The following diagram shows a comparison of the scene detection performance of a binocular depth camera and a structured light depth camera according to some embodiments;

[0027] Figure 11 A schematic diagram of a plane detection scene for augmented reality applications is shown according to some embodiments;

[0028] Figure 12 A configuration flowchart of the display device according to some embodiments is shown;

[0029] Figure 13 A flowchart illustrating the display device configuration process is shown according to some embodiments;

[0030] Figure 14 A flowchart illustrating the display device configuration process is shown according to some embodiments;

[0031] Figure 15 A schematic diagram of the inter-frame difference image is shown according to some embodiments;

[0032] Figure 16 A schematic diagram of the target contour region is shown according to some embodiments;

[0033] Figure 17 A schematic diagram of the centroid and anchor points according to some embodiments is shown;

[0034] Figure 18 A schematic diagram of the fitting plane is shown according to some embodiments;

[0035] Figure 19 A schematic diagram of fitting plane normal vectors is shown according to some embodiments. Detailed Implementation

[0036] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0037] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete implementation on its own. It should be noted that the brief descriptions of terminology in this application are merely for the convenience of understanding the embodiments described below, and are not intended to limit the implementation of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0038] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0039] In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to achieve connection and communication via network communication protocols, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronous display. The display device 200 also communicates with the server 400 via various communication methods. The display device 200 can communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactive features to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast television reception functions, the display device 200 can also be equipped with a smart network television function that provides computer support.

[0040] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0041] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0042] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0043] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0044] Figure 4 This is a schematic diagram of the software configuration in a display device 200 according to one or more embodiments of this application, such as... Figure 4As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0045] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 according to one or more embodiments of this application, such as... Figure 5 As shown, the application layer contains at least one application whose corresponding icon control can be displayed on the screen, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV applications can provide live television from different signal sources. Video-on-demand applications can provide video from different storage sources. Unlike live TV applications, video-on-demand provides video display from certain storage sources. Media center applications can provide applications for playing various multimedia content. The application center can provide storage for various applications.

[0046] Figure 6 An exemplary diagram illustrating an AR application scenario according to an exemplary embodiment is shown. Figure 6 As shown, AR application scenarios can involve one or more devices, such as a control device 100, a display device 200, a smart device 300, and an image acquisition device 500. The display device includes a monitor, an image acquisition device, and a controller. The image acquisition device can be fixed to or partially embedded in the housing of the display device, and is used to acquire color and depth images of the real scene directly in front of the display device. Users can enter the AR application by inputting user control commands through the control device or smart device. When entering the AR application, the image acquisition device can capture color and depth images of the real scene in real time (the depth image is the image where the distance from the image acquisition device to each point in the real scene is used as pixel values), and send the color image to the monitor for display as a data stream, and send the depth image to the controller for processing. Both the color and depth images consist of multiple consecutive frames.

[0047] While the display shows the real-world scene, users can input control commands in the AR application to add virtual objects to the real-world scene, ultimately merging them to form an augmented reality image. After acquiring the real-world scene image, the image acquisition device can first send the real-world scene image to the controller, triggering the controller to call up virtual objects and render their images. The virtual object images are then merged with the real-world scene image to form the final augmented reality image. Finally, the augmented reality image is sent to the display for display.

[0048] The virtual objects added to the real-world scene by the controller include, but are not limited to, text, images, 3D models, and video footage. In AR applications, the controller can blend the real-world scene with the virtual object footage, for example... Figure 6 In the scenario shown, when a user uses an AR interior design application, after acquiring the interior scene, they can use a simulated scene to apply AR scene textures to the interior scene. To do this, the AR application must first confirm the specific information of each scene in the real scene image. For example, in the depth image of the home interior scene acquired by the image acquisition device, the depth image of the home interior scene must first be subjected to planar detection to determine the ground position in the home interior scene.

[0049] It should be noted that the image acquisition devices include: TOF depth camera, structured light depth camera, and binocular depth camera.

[0050] Figure 7 This is a schematic diagram of a planar detection scene using a TOF depth camera according to one or more embodiments of this application. Figure 7 As shown, a Time-of-Flight (TOF) depth camera obtains distance by measuring the time of flight of light. It continuously emits laser pulses to the target and uses a sensor to receive the reflected light from these pulses. The exact distance to the target is then calculated by determining the round-trip time of the laser pulses. However, when using a TOF depth camera to capture indoor scenes in AR applications, there is often scattered light entering from windows. Furthermore, the floors in indoor scenes are often smooth surfaces like tiles or wood flooring, causing the scattered light to reflect mirror-like, which interferes with the reflected light from the TOF depth camera. This results in poor planar detection performance of the TOF depth camera on the indoor floor. Figure 8 A schematic diagram illustrating the planar detection effect of a TOF depth camera according to one or more embodiments of this application is shown. Figure 8 As shown, TOF depth cameras can only acquire partial ground location information in indoor home scenes, making it difficult to obtain complete ground location information directly from the depth images acquired by TOF depth cameras.

[0051] Figure 9This is a comparative diagram of the principles of a structured light depth camera and a TOF depth camera according to one or more embodiments of this application. Figure 9 As shown, a structured light depth camera projects light with specific structural features onto the object being photographed using a near-infrared laser, which is then captured by a dedicated infrared camera. This structured light can acquire different image phase information based on different depth regions of the object being photographed, and further, a processing unit converts the image phase information into depth information to obtain the three-dimensional structure of the object. However, because structured light emitted by near-infrared lasers is easily affected by ambient light, it is difficult to acquire image phase information on surfaces with specular reflection. Furthermore, structured light is typically discrete spot light, striped light, or coded structured light, such as… Figure 9 As shown in the principle comparison diagram, for the same plane, the structured light depth camera does not collect as comprehensive planar position information as the TOF depth camera. Therefore, compared with the laser pulses emitted by the TOF depth camera, the light from the structured light is more difficult to use to perform planar detection on the ground of a home indoor scene by directly using the depth image obtained by the structured light depth camera.

[0052] Figure 10 This is a comparison of the scene detection performance of a binocular depth camera and a structured light depth camera according to one or more embodiments of this application. Figure 10 As shown, a binocular depth camera acquires two images of the object being measured from different positions using the parallax principle and an imaging device. By calculating the positional deviation between corresponding points in the images, the three-dimensional structure of the object is obtained. Because binocular depth cameras require feature matching between the left and right cameras, large image deviations can occur when lighting conditions change significantly, leading to matching failures. Therefore, binocular depth cameras are sensitive to ambient light and are not suitable for monotonous scenes lacking texture. Compared to TOF depth cameras and structured light depth cameras, binocular depth cameras are even less suitable for planar detection of mirror-reflective surfaces.

[0053] Therefore, although existing image acquisition devices can directly use depth cameras to obtain depth images of indoor scenes in homes to determine ground location information, the fact that the ground in indoor scenes is often mirror-reflective affects the process of obtaining ground location information by the image acquisition devices. As a result, the image acquisition devices can only obtain partial ground location information by directly performing planar detection on the ground in indoor scenes.

[0054] Therefore, this application provides a display device capable of accurately and effectively detecting the plane of the ground in a home setting, thereby solving the problem of poor ground imaging. It should be noted that, since existing image acquisition devices, particularly TOF depth cameras, provide good results for ground plane detection, this application uses a TOF depth camera as an example for image acquisition. However, the solution presented in this application is not limited to using only a TOF depth camera.

[0055] Figure 11 This is a schematic diagram of a plane detection scene for an augmented reality application according to one or more embodiments of this application. Taking an augmented reality application implementing AR interior design in a display device as an example, the augmented reality application needs to acquire the current real scene image to determine the position of walls, the position of the floor, and the placement of furniture. Therefore, upon entering the augmented reality application, the image acquisition device obtains access permission to acquire color and depth images of the real scene. However, the controller can only determine the floor position information based on the depth image data stream, but the depth image directly captured by the image acquisition device can only obtain partial floor position information. In order to perform plane detection on the floor more comprehensively and accurately, this application illustrates a plane fitting method, which generates anchor points by moving the object and the contact point between the object and the plane to be detected, and performs plane fitting on all anchor points in the anchor point set to determine the floor position information. In specific implementation, when the display device responds to the user's control command to enter the augmented reality application, it immediately enters a calibration procedure to perform plane detection on the floor through the calibration procedure. Taking a person as a moving object as an example, the contact point between the moving object and the ground is obtained by the person walking on the ground directly in front of the image acquisition device as an anchor point. The ground position information is determined based on the anchor point. Compared with the method of directly using the depth image obtained by the depth camera to determine the ground position information, this application can perform more comprehensive and accurate planar detection of the ground.

[0056] It is worth noting that, since display devices are usually in a fixed position in a home setting and users do not move them frequently, and the ground position is fixed, the stored ground position information can be retrieved multiple times by executing a calibration procedure once and storing the ground position information obtained by the calibration procedure.

[0057] The display device shown in this application includes: a display; an image acquisition device configured to acquire color images and depth images of a real scene; the depth image includes multiple consecutive frame images; and a controller configured to execute... Figure 12 A plane fitting method is shown, including steps S1201-S1206 in the following steps;

[0058] Step S1201: Receive control commands input by the user for entering the augmented reality application.

[0059] The display device's controller can receive various control commands input by the user during use. Each control command corresponds to a control function, and some control commands can be used to access augmented reality applications. Users can input control commands to access augmented reality applications in different usage environments. For example, a user can open an AR-enabled application in a home indoor scene to complete the input. This AR-enabled application could be an application that enables AR motion-sensing games or AR interior design functions.

[0060] Some display devices can also have a built-in or external intelligent voice system, allowing users to input control commands to access augmented reality applications. For example, users can control the display device to launch an AR application by inputting voice commands such as "Open ××AR application." In this case, the controller can also receive the control command to access the augmented reality application.

[0061] Step S1202: In response to a control command, obtain a set of anchor points determined based on the depth image. The set of anchor points includes multiple anchor points that meet preset conditions. The preset conditions are used to determine the contact points between the moving object and the plane to be detected.

[0062] In some embodiments, the anchor point set includes at least 10 anchor points, each with coordinates (u, v, X, Y, Z). Where u and v are the pixel coordinates (i.e., planar coordinates) of the anchor point, and X, Y, and Z are the point cloud coordinates (i.e., spatial coordinates of the camera) of the anchor point.

[0063] In the specific implementation of the step of obtaining the set of anchor points determined based on the depth image, such as Figure 13 As shown, the controller is further configured to: acquire multiple inter-frame difference images determined based on the depth image; the inter-frame difference images are obtained based on two adjacent frames in the depth image;

[0064] In some embodiments, after receiving a control command for entering an augmented reality application, the display device may, in response to the control command, acquire an inter-frame difference image of the depth image.

[0065] In specific implementation, such as Figure 14 As shown, the controller is further configured to include the following steps in the process of acquiring the inter-frame difference image of the depth image: Step S1401, acquiring any two adjacent frames of the depth image. After receiving a control command for entering the augmented reality application, the controller sends a request command to the image acquisition device to request the depth image. The image acquisition device sends the acquired color image and depth image to the controller, enabling the controller to acquire the depth image. In order to acquire the inter-frame difference image, the controller first needs to acquire any two frames of the depth image.

[0066] To transmit request commands and depth images, the image acquisition device needs to establish a specific form of communication connection with the controller. This communication connection can be based on wired or wireless communication. For example, for wired communication, the image acquisition device's communicator can be a USB interface, allowing the device to connect to the display device via a USB cable and establish a communication connection based on the USB transmission protocol. For wireless communication, the communicator can be a module with built-in Bluetooth, infrared, or Wi-Fi network capabilities, establishing a wireless connection through the corresponding transmission protocol.

[0067] During the process of the controller acquiring depth images, the request command and the depth image can be based on the same communication connection method or on different communication connection methods. For example, the controller can send an image acquisition request to the image acquisition device via Bluetooth, but because Bluetooth connection has low transmission efficiency, the image acquisition device sends the depth image to the controller via a Wi-Fi network after receiving the image acquisition request.

[0068] It should be noted that when acquiring depth images, the communication connection between the image acquisition device and the controller can also be detected. When the image acquisition device and the controller can interact with each other, the controller can send an image acquisition request to the image acquisition device through the communicator. When the image acquisition device and the controller cannot interact with each other, the controller can display the connection interface between the controller and the image acquisition device on the display to establish a communication connection.

[0069] Figure 15 The diagram illustrates an inter-frame difference image according to one or more embodiments of this application. For fast-moving objects, a smaller frame acquisition time interval needs to be selected. If the selected frame acquisition time interval is inappropriate, when the moving object has no overlap between two consecutive frames, only two separate objects can be obtained through the inter-frame difference image, and the motion contour of the moving object cannot be obtained. For slow-moving objects, a larger frame acquisition time interval needs to be selected. If the selected frame acquisition time interval is inappropriate, when the object almost completely overlaps between two consecutive frames, the moving object cannot be detected. Therefore, the time interval for acquiring any two frames in the depth image can be set according to the actual situation based on the moving speed of the moving object, such as... Figure 15 As shown, taking a person as the moving object as an example, the frame acquisition time interval can be set to 500ms (milliseconds) to acquire two adjacent frames as shown. Figure 15 The depth image on the left side of the image is obtained by subtracting the depth images from the two frames. Figure 15 Only one depth image frame is shown in the image; adjacent depth images are not shown because the differences are too small to be distinguished by the naked eye. This allows us to obtain images such as... Figure 15 The inter-frame difference image shown on the right side of the image.

[0070] Step S1402: Determine whether the two adjacent frames of images meet the filtering conditions;

[0071] It should be noted that the inter-frame difference method can obtain the outline of a moving object by performing a difference operation on two adjacent frames in a video image sequence. When a moving object exists in the monitored scene, there will be a significant difference between frames. Subtracting two frames yields the absolute value of the difference between the two frames. This absolute value is then used to analyze the motion characteristics of the video or image sequence and determine whether a moving object exists in the image sequence. Furthermore, if, in two adjacent frames, the moving object in the later depth image occludes the background of the previous depth image by less than a certain threshold, then a moving object is present in the inter-frame difference image of those two adjacent frames.

[0072] In some embodiments, each frame of a depth image includes multiple pixels, and each pixel includes pixel coordinates (u, v) and the pixel depth corresponding to those pixel coordinates. In the two adjacent frames of the acquired image, the latter frame is f. n The previous frame image is f n-1 Where n is the sequence number of the frame image, Let (u, v) be the pixel depth of the pixel point (u, v) in the next frame of two adjacent frames. The abs() function is used to calculate the absolute value of the pixel depth (u, v) of the next frame in two adjacent frames. delta is the acquisition error of the depth image, and beta is the distance threshold for the moving object to occlude the background. The order of the previous and next frames is determined according to the acquisition order of the depth images.

[0073] Therefore, the filter criteria are set as follows:

[0074] >0 and abs ( - ) < delta; or >0 and > +beta.

[0075] The frame difference images filtered according to the above filtering criteria contain the motion contours of moving objects.

[0076] Step S1403: If two adjacent frames meet the filtering criteria, perform inter-frame difference analysis on the two adjacent frames that meet the filtering criteria to obtain the inter-frame difference image of the two adjacent frames. The filtering criteria are used to determine whether there is a moving object in the inter-frame difference image.

[0077] After filtering out two adjacent frames using the filtering criteria, Marked as 0, the inter-frame difference image f' (n) =f n -f n-1 .

[0078] In some embodiments, when using a TOF depth camera, the acquisition error of the depth image can be set to 20 mm based on the absolute accuracy of the TOF depth camera. In specific implementations, the acquisition error of the depth image can be set according to actual conditions. Furthermore, in two adjacent depth image frames, if there is a moving object and background in the previous frame, and the position of the moving object in the subsequent frame changes compared to the previous frame, the background it occludes will inevitably change. In essence, for a pixel in a given frame, if there is an occlusion, the depth coordinates of the occluder (e.g., a person) at that pixel in the current frame will necessarily be smaller than the depth value at the same pixel position in the previous frame. Therefore, by setting a distance threshold for the moving object occluding the background, it can be determined whether the inter-frame difference image contains a moving object. This distance threshold can be set according to the actual situation of a home indoor scene; for example, it can be set to 100 mm. Therefore, when the filtering conditions are met, it indicates that a moving object exists in the inter-frame difference image.

[0079] In some embodiments, the controller is also configured to perform step S1404, deleting two adjacent depth images if they do not meet the filtering criteria.

[0080] If two adjacent depth images do not meet the filtering criteria, it means that there is no moving object in the inter-frame difference image of the two adjacent depth images or the moving object in the two adjacent images has not moved. Therefore, the contact point between the moving object and the plane to be detected cannot be obtained from the inter-frame difference image of the two adjacent depth images. The controller can delete depth images that do not meet the filtering criteria to reduce memory usage.

[0081] At least one target contour region is obtained from the inter-frame difference image. The target contour region includes a centroid and multiple pixels, and the centroid is the mass center of the target contour region.

[0082] Figure 16 An exemplary schematic diagram of the target contour region is shown in one or more embodiments of this application. In some embodiments, the controller is further configured to:

[0083] In the step of obtaining at least one target contour region based on the inter-frame difference image, the first gradient image of the inter-frame difference image is obtained;

[0084] Inter-frame difference image f' （u，v）It is a discrete two-dimensional function, and the gradient represents the maximum value of the function at a given point along that direction. Therefore, the inter-frame difference image f' （u，v） The gradient at pixel (u, v) is a two-dimensional vector composed of the partial derivatives of the two-dimensional function at that point in the u direction and the partial derivatives in the v direction. However, the gradient definition only applies to continuous gradients. For images, the gradient needs to be approximated using the finite difference method. For example, the central difference method of the finite difference method can be used to approximate the gradient to obtain the image gradient along the u direction and the image gradient along the v direction. Based on the image gradient along the u direction, the vertical edge of the motion contour of the moving object in the inter-frame difference image can be obtained, and based on the image gradient along the v direction, the horizontal edge of the motion contour of the moving object in the inter-frame difference image can be obtained.

[0085] In the specific implementation, the inter-frame difference image includes multiple pixels, each pixel including pixel coordinates (u, v); the controller is further configured as follows:

[0086] In the step of obtaining the first gradient image of the inter-frame difference image, the first gradient coordinates (u', v) of the pixel coordinates (u, v) of each pixel point in the inter-frame difference image are obtained by taking the derivative along the u direction and the second gradient coordinates (u, v') of the derivative along the v direction.

[0087] The second gradient image generated based on the first gradient coordinates (u', v) and the third gradient image generated based on the second gradient coordinates (u, v') are superimposed to obtain the first gradient image of the inter-frame difference image.

[0088] The pixels with approximate gradients in the first gradient image are clustered according to the clustering algorithm. The first gradient image corresponds to the pixels in the inter-frame difference image to obtain multiple clustering contour regions in the inter-frame difference image.

[0089] Clustering algorithms include various methods such as K-means clustering, DBSCAN clustering, hierarchical clustering, and SOM clustering. This application can use DBSCAN clustering to cluster pixels in the total gradient image. It should be noted that this application is not limited to using only DBSCAN clustering.

[0090] Taking DBSCAN clustering as an example, Density-Based Special Clustering of Applications with Noise (DBSCAN) defines a cluster as the largest set of density-connected points. It can divide regions with sufficiently high density into clusters and discover clusters of arbitrary shapes in a noisy spatial database. For multiple sample points in a 2D image, the scan radius and minimum number of contained points are first determined. Starting with an unvisited point, all nearby points within the scan radius are selected. If the number of nearby points is greater than or equal to the minimum number of contained points, the current point and its nearby points form a cluster, and the unvisited point is marked as visited. This process continues recursively, processing all unmarked points within the cluster in the same way, thus expanding the cluster. If the number of nearby points is less than the minimum number of contained points, the unmarked point is marked as a noise point. If the cluster is sufficiently expanded, i.e., all points within the cluster are marked as visited, the unvisited points are then processed using the same algorithm. The DBSCAN clustering method can divide a two-dimensional image into multiple clusters based on density among multiple coordinate points, with each cluster being a clustering contour region.

[0091] Detect whether the clustered contour regions meet the constraints;

[0092] In some embodiments, the controller is further configured to convert the inter-frame difference image into a point cloud image in the step of detecting whether the clustered contour region meets the constraints;

[0093] In the specific implementation, each pixel in the inter-frame difference image also includes a depth value. ;

[0094] The controller is further configured as follows:

[0095] In the step of converting the inter-frame difference image into a point cloud image, the pixel coordinates (u, v) of each pixel in the inter-frame difference image are converted into point cloud coordinates (X, Y, Z) to obtain the point cloud image; Z=d (u,v) / s, X=Z (uc x ) / f x Y=Z (vc y ) / f y ;

[0096] Among them, f x f y c x ,c y is the camera's intrinsic parameter matrix, and s is the scaling factor of the depth image.

[0097] Obtain the local normal vector of each cluster contour region in the point cloud image;

[0098] If the angle between the local normal vector of any clustered contour region and the y-axis in the coordinate system of the point cloud image is within the constraint threshold, then the clustered contour region meets the constraint condition.

[0099] The controller is further configured to: in the step of acquiring the local normal vector of each cluster contour region in the point cloud image, divide each cluster region into sub-regions according to a specified size;

[0100] Calculate the local normal vector for each sub-region.

[0101] In some embodiments, the specified size is 4 A 4-pixel grid.

[0102] Taking a human as an example, the motion contour of a human can be obtained according to the aforementioned steps, and the motion contour of a human can be further divided into multiple cluster contour regions. Since each part of a human has its unique characteristics, such as the head being round and the legs being cylindrical, the leg region can be selected according to the constraints.

[0103] In some embodiments, the constraints are:

[0104] The angle between the local normal vector of the clustered contour region and the y-axis in the coordinate system of the point cloud image is within the range of -90°±10°.

[0105] If any cluster contour region meets the constraint conditions, then the contour region is determined as the target contour region, where the constraint conditions are used to determine the angle of the local normal vector of the cluster contour region.

[0106] It should be noted that, since there are multiple cluster contour regions, there may be more than one cluster contour region that meets the constraints, such as... Figure 16 As shown, in the embodiment where a person is moving, the target contour region is the leg region. When the person's movement in the inter-frame difference image is that both legs overlap, only one target contour region can be obtained. When the person's movement in the inter-frame difference image is that they are taking a step, two target contour regions can be obtained. Due to different constraints, the target contour region can be the entire leg or the lower leg region. The specific setting method of the target contour region can be set according to the actual situation.

[0107] Select the pixel in the target contour region whose height is lower than the centroid and whose distance from the centroid is the farthest as the anchor point.

[0108] Figure 17 This is a schematic diagram of the centroid and anchor point shown in one or more embodiments of this application, as follows: Figure 17As shown, according to the target contour region acquisition method disclosed in this application, each inter-frame difference image includes at least one target contour region. Taking a person walking with both legs in the inter-frame difference image as an example, it includes two target contour regions, namely the regions where the person's two legs are located. Therefore, the inter-frame difference image includes two centroids, which are the mass centers of the two legs respectively. Correspondingly, in each of the two target contour regions, there is a point in each target contour region that is farthest from the centroid as an anchor point.

[0109] It should be noted that the centroid of the image described in the embodiments of this application is also called the center of gravity of the image. The pixel value of each point in the image can be understood as the mass of that pixel. Unlike the center of gravity of an object in space, since the image is two-dimensional, the centroid is found independently on the x-axis and y-axis of the image. That is, for the centroid in the x-axis direction, the sum of the pixels on the left and right sides of the centroid is equal; for the centroid in the y-axis direction, the sum of the pixels on the top and bottom sides of the centroid is equal.

[0110] Taking a moving human as an example, the target contour area is usually the entire area from the legs to the feet. Therefore, during the process of walking, the point farthest from the center of mass is usually the contact point between the heel, sole, or toe and the ground. Thus, selecting the point farthest from the center of mass in the target contour area as the anchor point is equivalent to selecting the contact point between the person and the ground, so as to further indirectly perform planar detection on the ground through multiple contact points between the person and the ground.

[0111] It should be noted that, depending on the constraints, the leg area can be defined as the entire leg area from the thigh to the foot, or it can be defined as the entire calf area from the knee to the foot. The leg area can be selected according to the actual constraints.

[0112] An anchor point set is generated based on all anchor points of the multiple inter-frame difference images, and the anchor points in the anchor point set are used for plane fitting.

[0113] In some embodiments, the controller is further configured to: in the step of generating an anchor set based on all anchors of multiple inter-frame difference images, determine whether the number of anchors in the anchor set is greater than a first threshold.

[0114] In some embodiments, if the number of anchor points reaches a first threshold, it means that the number of anchor points collected can be used for plane detection. For example, the first threshold can be set to 10.

[0115] If the number of anchor points in the anchor point set is greater than the first threshold, then obtain the maximum distance difference of the point cloud coordinates of all anchor points in the anchor point set along the x-axis and the maximum distance difference along the z-axis.

[0116] In some embodiments, in order to further determine whether all the obtained anchor points are usable, the spatial movement distance of the moving object on the plane can be determined by obtaining the maximum distance difference along the x-axis and the maximum distance difference along the Z-axis of the point cloud coordinates of all anchor points in the anchor point set. If the spatial movement distance reaches a certain range, it proves that the spatial distribution distance of the anchor points in the anchor point set is appropriate, and it is further determined that the obtained anchor points can be used for plane fitting.

[0117] Determine whether the maximum distance difference along the x-axis and the maximum distance difference along the z-axis of the point cloud coordinates of all anchor points are both greater than the second threshold.

[0118] In some embodiments, when the maximum distance difference between the point cloud coordinates of all anchor points along the x-axis and the maximum distance difference along the z-axis are both greater than or equal to the second threshold, it can be determined that the distribution of all anchor points in the anchor point set is at a suitable spatial distance. For example, the second threshold can be set to 1000mm.

[0119] If the maximum distance difference along the x-axis and the maximum distance difference along the z-axis of the point cloud coordinates of all anchor points are both greater than the second threshold, then the anchor point region formed by all anchor points in the anchor point set is divided into N intervals (N is a natural number).

[0120] In some embodiments, if the maximum distance difference between the point cloud coordinates of all anchor points along the x-axis and the maximum distance difference along the z-axis are both greater than a second threshold (taking a second threshold of 1000 mm as an example), then on the X0Z plane, the set of anchor points forms at least a 1000 mm... 1000mm 2 The spatial region is defined along the XOZ plane at 500 degrees. 500mm 2 Dividing the data into intervals results in at least four intervals.

[0121] Determine whether the number of anchor points in each of the N intervals is greater than 0;

[0122] If the number of anchor points in each of the N intervals is greater than 0, then the anchor points contained in the anchor point set can be used for plane fitting.

[0123] In some embodiments, taking the above embodiment as an example, if the anchor point region is divided into 4 intervals on the XOZ plane, and if the number of anchor points in each interval is greater than 0, then it is determined that the anchor point distribution in each interval is uniform, and it can be determined that the set of anchor points can be used to determine the fitting plane.

[0124] According to the above embodiments, anchor points in multiple frames of depth images are determined, and anchor point sets are generated from the anchor points in the multiple frames of depth images. Since the anchor points in the anchor point set are all close to the ground, it is urgent to provide a plane fitting method to perform plane fitting on the anchor points in the anchor point set to obtain the plane that is closest to the ground, and then perform plane detection on the ground.

[0125] Step S1203: Select at least three points from the anchor point set for plane fitting to obtain multiple fitting planes.

[0126] Figure 18 This is a schematic diagram of a fitting plane according to one or more embodiments of this application. In this diagram, three points are randomly sampled from the anchor point set a times, and plane fitting is performed based on each sampled point, resulting in a fitting planes. Figure 18 As shown, taking a=3 as an example, the first random sampling of 3 points yields the first fitting plane a1, the second random sampling of 3 points yields the second fitting plane a2, and the third random sampling of 3 points yields the third fitting plane a3.

[0127] Step S1204: Determine the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes;

[0128] In some embodiments, the controller is further configured to: in the step of determining the first fitting plane and the coplanar fitting plane of the first fitting plane from a plurality of fitting planes, obtain the normal vectors of all fitting planes in the plurality of fitting planes;

[0129] Taking the above embodiment as an example of obtaining a fitting planes, the normal vectors of all fitting planes in the a fitting planes can be obtained to get a normal vectors.

[0130] Calculate the spatial angle between the normal vector of any fitted plane and the normal vectors of all the remaining fitted planes to obtain the spatial angle between each fitted plane and all the remaining fitted planes.

[0131] Taking the a normal vectors in the a fitting planes in the above embodiment as an example, the spatial angle between the normal vector of any fitting plane and the normal vectors of the remaining a-1 fitting planes is calculated pairwise, so that each plane has a-1 angle values.

[0132] Figure 19 The schematic diagram of the fitting plane normal vector according to one or more embodiments of this application shows that, taking a=3 as an example, the normal vector of the first fitting plane a1 is a1', the normal vector of the second fitting plane a2 is a2', and the normal vector of the third fitting plane a3 is a3', wherein, as shown in the figure... Figure 19As shown, a1' forms two spatial angles with a2' and a3' respectively, and the values ​​of the two spatial angles are calculated.

[0133] The number of spatial angles between each fitted plane and all other fitted planes that are less than the first angle is obtained, and the fitted plane with the most spatial angles is determined as the first fitted plane.

[0134] In some embodiments, if the spatial angle between a certain fitted plane and all the remaining fitted planes is less than a certain angle, then the fitted plane is closer to the ground. For example, the first angle is set to 2°, and the fitted plane with the most spatial angles that meet the condition is determined as the first fitted plane.

[0135] In some embodiments, the controller is further configured to:

[0136] The steps of determining the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes include:

[0137] From the remaining fitting planes of the first fitting plane, select the fitting plane whose spatial angle with the first fitting plane is less than the first angle as the coplanar fitting plane of the first fitting plane.

[0138] In some embodiments, since the first fitting plane is only a plane that approximates the ground, in order to more accurately detect the actual ground plane, coplanar fitting planes of the first fitting plane are further selected. Fitting planes whose spatial angle with the first fitting plane is less than a first angle are used as coplanar fitting planes, resulting in multiple coplanar fitting planes of the first fitting plane. For example, the first angle can be set to 2°.

[0139] Step S1205: Determine the second fitting plane based on the first fitting plane and the coplanar fitting plane;

[0140] In some embodiments, the controller is further configured to: in the step of determining the second fitting plane based on the first fitting plane and the coplanar fitting plane, acquire all anchor points on the first fitting plane and the coplanar fitting plane;

[0141] Perform least squares fitting on all anchor points on the first fitting plane and the coplanar fitting plane to obtain the second fitting plane.

[0142] Least squares is a mathematical optimization technique that finds the best function fit for data by minimizing the sum of squared errors. It is commonly used for plane fitting. The specific steps for plane fitting using least squares will not be elaborated upon here.

[0143] Step S1206: Perform a region growing algorithm on the second fitting plane to obtain the target fitting plane.

[0144] In some embodiments, the controller is further configured to: acquire multiple pixels in the depth image during the step of performing a region growing algorithm on the second fitting plane to obtain the target fitting plane;

[0145] Pixels whose vertical distance from the second fitting plane is less than the first distance are added to the second fitting plane to increase the region of the second fitting plane.

[0146] The first distance is set based on the measurement error of the TOF depth camera. It should be noted that the depth value measured by TOF has a certain error range, including absolute and relative accuracy. The absolute accuracy is 0.5%-1% error; if the measurement distance is 2cm, the error will be between 1-2cm. The relative accuracy is 0.5%-1% relative error; if multiple pixels at the same depth are continuously sampled, the depth value of that pixel will fluctuate, ranging from 0.5%-1%. Therefore, for example, the first distance can be set to 5cm, which corresponds to the range between the TOF depth camera's measurement error and the foot thickness.

[0147] In some embodiments, the controller is further configured to: in the step of performing a region growing algorithm on the second fitting plane to obtain a target fitting plane, after the region growing of the second fitting plane is completed, obtain all anchor points of the second fitting plane;

[0148] The second fitted plane region is subjected to least squares to obtain the target plane region. The target plane region is the ground region detected by the plane.

[0149] To further verify the target planar region, the controller is also used to execute a planar verification method, including:

[0150] Clear all anchor points in the target plane area and set the target plane area to the verification state;

[0151] Since the target planar region is a planar region indirectly obtained by using a set of anchor points for planar fitting, in order to further confirm that the target planar region is the ground region detected by the technical solution shown in this application, the target planar region is further verified.

[0152] When the target planar region is in the verification state, the verification point is obtained; the verification point is a pixel in the depth image.

[0153] In some embodiments, the pixels on the color image acquired by the image acquisition device correspond to the pixels in the depth image, and the pixels in the depth image correspond to the pixels in the inter-frame difference image. The controller is further configured to: in the step of acquiring verification points, perform the FAST feature point detection algorithm on the target planar region in the color image to obtain multiple feature points.

[0154] Obtain the depth values ​​of multiple feature points in a depth image;

[0155] If any feature point has a non-zero depth value in the depth image, then the feature point is used as a candidate point.

[0156] Multiple candidate points are obtained from multiple frames of color images through multiple iterations; the BRIEF descriptors of the candidate points are obtained to determine the attributes of the candidate points.

[0157] By matching different candidate point attributes, the candidate point with the least change among multiple color images is determined as the verification point.

[0158] In some embodiments, the controller is further configured to: in the step of obtaining verification points, obtain a new set of anchor points, and determine all anchor points in the new set of anchor points as verification points.

[0159] Determine whether the proportion of the verification point within the target plane region is greater than or equal to the proportion threshold.

[0160] If the proportion of verification points within the target plane is greater than or equal to the proportion threshold, then the target plane region is the final plane region.

[0161] If the proportion of the verification points within the target plane is less than the proportion threshold, the target plane region is reacquired.

[0162] In some embodiments, if the point-to-surface distance between the verification point and the target plane region is less than a first distance, the verification point is considered an interior point.

[0163] Calculate the proportion of inliers in the verification points. If the proportion of inliers in the verification points is greater than or equal to the proportion threshold, then the target plane region is the final plane region.

[0164] For example, the percentage threshold is 60%.

[0165] Based on the aforementioned display device, this application further provides a planar fitting method, including:

[0166] An anchor point set is obtained from the depth image. The anchor point set includes multiple anchor points that meet preset conditions. The preset conditions are used to determine the contact points between the moving object and the plane to be detected.

[0167] At least three sampling points are randomly selected from the set of anchor points for plane fitting to obtain multiple fitting planes;

[0168] Determine the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes;

[0169] The second fitting plane is determined based on the first fitting plane and the coplanar fitting plane;

[0170] Perform a region growing algorithm on the second fitted plane to obtain the target fitted plane.

[0171] It should be understood that the specific implementation of each step in the above-described plane fitting method can be found in the aforementioned display device embodiments, and will not be repeated here. As can be seen from the above embodiments, the plane fitting method shown in this application indirectly determines the fitting plane through a set of anchor points. In essence, it performs plane fitting on the anchor points generated by the contact points between the moving object and the plane to be detected to indirectly determine the plane to be detected. Compared with the method of directly determining the plane to be detected through a depth camera, this application enables the display device to perform plane detection more comprehensively and accurately.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0173] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: monitor; An image acquisition device is configured to acquire depth images of a real scene. The depth image includes multiple consecutive frame images; The controller is configured as follows: Receive user input for control commands to access augmented reality applications; In response to the control command, a set of anchor points determined based on the depth image is obtained. The set of anchor points includes multiple anchor points that meet preset conditions. The preset conditions are used to determine the contact points between the moving object and the plane to be detected. At least three points are randomly selected from the set of anchor points for plane fitting to obtain multiple fitting planes; Determine the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes; The second fitting plane is determined based on the first fitting plane and the coplanar fitting plane; Perform a region growing algorithm on the second fitted plane to obtain the target fitted plane.

2. The display device according to claim 1, characterized in that, The controller is further configured to: In the step of obtaining the set of anchor points determined based on the depth image, multiple inter-frame difference images of the depth image are obtained; the inter-frame difference images are obtained based on two adjacent frames in the depth image; At least one target contour region is obtained from the inter-frame difference image. The target contour region includes a centroid and multiple pixels. The centroid is the mass center of the target contour region. Select the pixel in the target contour region that is lower than the centroid and has the greatest distance from the centroid as the anchor point; An anchor point set is generated based on all anchor points of the multiple inter-frame difference images to determine the fitting plane using the anchor point set.

3. The display device according to claim 1, characterized in that, The controller is further configured to: In the step of determining the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes, the normal vectors of all fitting planes in the multiple fitting planes are obtained; Calculate the spatial angle between the normal vector of any fitted plane and the normal vectors of all the remaining fitted planes to obtain the spatial angle between each fitted plane and all the remaining fitted planes. The number of spatial angles between each fitted plane and all other fitted planes that are less than the first angle is obtained, and the fitted plane with the most spatial angles is determined as the first fitted plane. From the remaining fitting planes of the first fitting plane, select the fitting plane whose spatial angle with the first fitting plane is less than the first angle as the coplanar fitting plane of the first fitting plane.

4. The display device according to claim 1, characterized in that, The controller is further configured to: In the step of determining the second fitting plane based on the first fitting plane and the coplanar fitting plane, all anchor points on the first fitting plane and the coplanar fitting plane are obtained; The least squares method is performed on all anchor points on the first fitting plane and the coplanar fitting plane to perform plane fitting, so as to obtain the second fitting plane.

5. The display device according to claim 1, characterized in that, The controller is further configured to: In the step of performing a region growing algorithm on the second fitting plane to obtain the target fitting plane, multiple pixels in the depth image are obtained; Pixels whose vertical distance from the second fitting plane is less than the first distance are added to the second fitting plane to increase the region of the second fitting plane; After the second fitted plane region has finished growing, obtain all anchor points of the second fitted plane; The second fitted plane region is subjected to least squares to obtain the target plane region.

6. The display device according to claim 1, characterized in that, The controller is also configured to: After obtaining the target fitting plane, clear all anchor points in the target plane region and set the target plane region to the verification state; When the target planar region is in the verification state, a verification point is acquired; the verification point is a pixel in the depth image. Determine whether the proportion of the verification points within the target plane area is greater than or equal to a proportion threshold; If the proportion of the verification points within the target plane is greater than or equal to the proportion threshold, then the target plane region is the final plane region; If the proportion of the verification point within the target plane is less than the proportion threshold, then the target plane region is reacquired.

7. The display device according to claim 6, characterized in that, The controller is further configured to: In the step of obtaining verification points, the FAST feature point detection algorithm is performed on the target planar region in the color image to obtain multiple feature points; Obtain the depth values ​​corresponding to multiple feature points in the depth image; If the depth value corresponding to any of the feature points in the depth image is not 0, then the feature point is used as a candidate point; Multiple candidate points are obtained from multiple frames of color images through multiple iterations; the attributes of the candidate points are determined by obtaining the BRIEF descriptors of the candidate points. By matching different candidate point attributes, the candidate point with the smallest change among multiple frames of color images is determined as the verification point.

8. The display device according to claim 6, characterized in that, The controller is further configured to: In the step of obtaining the verification point, after clearing all anchor points in the target plane region, the anchor point set in the depth image is re-obtained to obtain a new anchor point set; All anchor points in the new set of anchor points are identified as the verification points.

9. A plane fitting method, characterized in that, include: An anchor point set is obtained from the depth image. The anchor point set includes multiple anchor points that meet preset conditions. The preset conditions are used to determine the contact points between the moving object and the plane to be detected. At least three points are randomly selected from the set of anchor points for plane fitting to obtain multiple fitting planes; Determine the first fitting plane and the coplanar fitting plane of the first fitting plane from multiple fitting planes; The second fitting plane is determined based on the first fitting plane and the coplanar fitting plane; Perform a region growing algorithm on the second fitted plane to obtain the target fitted plane.

10. The fitting method according to claim 9, characterized in that, Also includes: Clear all anchor points in the target plane area and set the target plane area to the verification state; When the target planar region is in the verification state, a verification point is acquired; the verification point is a pixel in the depth image. Determine whether the proportion of the verification point within the target plane area is greater than or equal to a proportion threshold; If the proportion of the verification point within the target plane is greater than or equal to the proportion threshold, then the target plane region is the final plane region. If the proportion of the verification points within the target plane is less than a proportion threshold, then the target plane region is reacquired.

Images