Display device and anchor point determination method

Abstract

The application shows a display device and an anchor point determination method. The anchor point determination method comprises the following steps: receiving a control instruction input by a user for entering an augmented reality application; obtaining a plurality of interframe difference images of a depth image in response to the control instruction; obtaining at least one target contour region from the interframe difference images, the target contour region comprising a centroid and a plurality of pixel points; selecting a pixel point in the target contour region as an anchor point, the pixel point being the farthest from the centroid and having a lower height than the centroid; and generating an anchor point set from all anchor points of the plurality of interframe difference images, the anchor points in the anchor point set being used for plane fitting. The anchor point determination method indirectly determines a fitting plane through the anchor point set, and essentially performs plane fitting on anchor points generated by contact points between a moving object and a to-be-detected plane to indirectly determine the to-be-detected plane. Compared with a method of directly determining a plane by using a depth camera to obtain a depth image, the application can enable the display device to more comprehensively and accurately perform plane detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display devices, and particularly relates to a display device and an anchor point determination method. BACKGROUND

[0002] Augmented Reality (AR) technology is a technology of fusing virtual pictures with real scenes. Virtual objects, such as text, images, three-dimensional models and video pictures, can be simulated and applied in real scene pictures to bring users a viewing experience of combining virtual pictures with real pictures.

[0003] An AR application is usually arranged in a display device to perform AR motion sensing games or AR indoor design. When entering the AR application, the AR application first needs to perform plane detection on a real scene picture to determine ground position information and wall position information, etc. For example, in an AR motion sensing game, plane detection can be used to draw augmented reality images on real world walls in a player's room, or in AR indoor design, plane detection can be used to distinguish floors from obstacles.

[0004] When performing plane detection, an anchor point can be used to indirectly determine a fitting plane, so that the display device can perform plane detection more comprehensively and accurately. SUMMARY

[0005] The present application provides a display device and an anchor point determination method, which can indirectly determine a fitting plane through an anchor point set. In essence, anchor points generated by contact points of a moving object and a to-be-detected plane are used to perform plane fitting to indirectly determine the to-be-detected plane. Compared with a method of directly using a depth image obtained by a depth camera to determine a plane, the present application can make the display device perform plane detection more comprehensively and accurately.

[0006] The application provides a display device, comprising: a display; an image acquisition device configured to acquire a depth image of a real scene picture; the depth image comprises a plurality of continuous frame images; a controller configured to: receive a control instruction input by a user for entering an augmented reality application; in response to the control instruction, acquire a plurality of inter-frame difference images of the depth image; wherein the inter-frame difference images are obtained according to two adjacent frame images in the depth image; acquire at least one target contour region according to the inter-frame difference images, the target contour region comprising a mass center and a plurality of pixel points, the mass center being a mass center of the target region; select a pixel point in the target contour region as an anchor point, the pixel point being the farthest from the mass center and having a lower height than the mass center; generate an anchor point set according to all anchor points of the plurality of inter-frame difference images, the anchor points in the anchor point set being used for plane fitting. According to the embodiment of the application, the fitting plane can be determined through the anchor point set. The anchor point determination method indirectly determines the fitting plane through the anchor point set, and in essence, the contact points of the moving object and the to-be-detected plane are fitted to indirectly determine the to-be-detected plane. Compared with the method of directly determining the to-be-detected plane through a depth camera, the application can enable the display device to more comprehensively and accurately detect the plane.

[0007] In some embodiments of the application, the controller is further configured to: in the step of acquiring the plurality of inter-frame difference images of the depth image, acquire any two adjacent frame images in the depth image; determine whether the two adjacent frame images meet a screening condition; if the two adjacent frame images meet the screening condition, perform an inter-frame difference method on the two adjacent frame images meeting the screening condition to obtain an inter-frame difference image of the two adjacent frame images; wherein the screening condition is used to determine whether a moving object exists in the inter-frame difference image. According to the implementation, whether a moving object exists in the inter-frame difference image can be determined through the inter-frame difference method, which is further used to determine the contact points of the moving object and the to-be-detected plane.

[0008] In some embodiments of the application, the controller is further configured to: in the step of acquiring at least one target contour region according to the inter-frame difference image, acquire a first gradient image of the inter-frame difference image; the first gradient image comprises pixel points having approximate gradients; the pixel points having approximate gradients in the first gradient image are clustered according to a clustering algorithm, the pixel points in the first gradient image corresponding to the pixel points in the inter-frame difference image, to obtain a plurality of clustered contour regions in the inter-frame difference image; determine whether the clustered contour regions meet a constraint condition; if any one of the clustered contour regions meets the constraint condition, determine the clustered contour region as the target contour region, wherein the constraint condition is used to determine the angle of the local normal vector of the clustered contour region. According to the implementation, the target contour region can be acquired according to the angle of the local normal vector of the clustered contour region. In the embodiments shown in the application, the target contour region is a leg region of the moving object, which is used to further determine the contact points of the moving object and the to-be-detected plane.

[0009] In some embodiments of the present application, each frame of the depth image comprises a plurality of pixels, and each pixel comprises a pixel coordinate (u, v) and a pixel depth d corresponding to the pixel coordinate (u，v) ; the screening condition is: and or and wherein n is the serial number of the frame image, is the pixel depth corresponding to the pixel point of the latter frame image of the adjacent two frame images, is the pixel depth corresponding to the pixel point of the former frame image of the adjacent two frame images, the abs() function is used to obtain the absolute value, delta is the acquisition error of the depth image, beta is the distance threshold of the moving object shielding the background, and the former frame image and the latter frame image are determined in front of and behind according to the depth image acquisition sequence. By using the implementation manner, the frame image not meeting the acquisition error of the depth image can be screened out, and the frame image not containing the moving object can also be screened out.

[0010] In some embodiments of the present application, the inter-frame difference image comprises a plurality of pixels, and each pixel comprises a pixel coordinate (u, v); the controller is further configured to: in the step of obtaining the first gradient image of the inter-frame difference image, obtain a first gradient coordinate (u', v) derived along the u direction and a second gradient coordinate (u, v') derived along the v direction of the pixel coordinate (u, v) of each pixel in the inter-frame difference image; and superimpose a second gradient image generated according to the first gradient coordinate (u', v) and a third gradient image generated according to the second gradient coordinate (u, v') to obtain the first gradient image of the inter-frame difference image.

[0011] In some embodiments of the present application, the controller is further configured to: in the step of detecting whether the cluster contour region meets the constraint condition, convert the inter-frame difference image into a point cloud image; obtain a local normal vector of each cluster contour region in the point cloud image; and if the local normal vector of any one cluster contour region is within a constraint threshold with respect to the y-axis in the coordinate system of the point cloud image, the cluster contour region meets the constraint condition. By using the implementation manner, since the normal vector of each region of the moving object is different, the leg region of the moving object can be determined according to the local normal vector, which is used to further determine the contact point of the moving object and the detection plane.

[0012] In some embodiments of the present application, each pixel in the inter-frame difference image further comprises a depth value d (u，v) ; the controller is further configured to: in the step of converting the inter-frame difference image into a point cloud image, convert the pixel coordinate (u, v) of each pixel in the inter-frame difference image into a point cloud coordinate (X, Y, Z) to obtain the point cloud image; Z=d (u,v) / s, X=Z*(u-cx ) / f x , Y = Z * (v - c y ) / f y ; wherein, f x f y c x ,c y is an intrinsic matrix of the camera, and s is a scaling factor of the depth image.

[0013] In some embodiments of the present application, the controller is further configured to: in the step of generating the anchor point set according to all anchor points of the plurality of inter-frame difference images, determine whether the number of anchor points in the anchor point set is greater than a first threshold value; if the number of anchor points in the anchor point set is greater than the first threshold value, obtain a maximum distance difference value of the point cloud coordinates of all anchor points in the anchor point set along the x-axis direction and a maximum distance difference value of the point cloud coordinates of all anchor points in the anchor point set along the z-axis direction; determine whether the maximum distance difference value of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of the point cloud coordinates of all anchor points along the z-axis direction are both greater than a second threshold value; if the maximum distance difference value of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of the point cloud coordinates of all anchor points along the z-axis direction are both greater than the second threshold value, divide an anchor point region formed by all anchor points in the anchor point set into N intervals (N is a natural number); determine whether the number of anchor points in each interval of the N intervals is greater than 0; if the number of anchor points in each interval of the N intervals is greater than 0, the anchor points contained in the anchor point set can be used for plane fitting.

[0014] In some embodiments of the present application, a method for determining an anchor point is also provided, comprising: obtaining an inter-frame difference image according to two adjacent frames of images in a depth image; obtaining at least one target contour region from the inter-frame difference image; determining a centroid of the target contour region; selecting a pixel point in the target contour region as an anchor point, the pixel point being the lowest in height and farthest from the centroid, the anchor point being used for plane fitting.

[0015] In this way, the anchor point determination method can indirectly determine the fitting plane through the anchor point set, and in essence, the contact points between the moving object and the anchor points generated by the to-be-detected plane are subjected to plane fitting to indirectly determine the to-be-detected plane. Compared with the method of directly determining the plane by using the depth image obtained by the depth camera, the present application can enable the display device to more comprehensively and accurately perform plane detection. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0017] Figure 1 A schematic diagram of an operation scenario between a display device and a control device according to some embodiments is shown.

[0018] Figure 2 A configuration block diagram of the control device 100 is shown according to some embodiments;

[0019] Figure 3 A hardware configuration block diagram of the display device 200 is shown according to some embodiments;

[0020] Figure 4 A software configuration block diagram of the display device 200 is shown according to some embodiments;

[0021] Figure 5 An application icon control interface display diagram of the display device 200 is shown according to some embodiments;

[0022] Figure 6 An AR application scene diagram is shown according to some embodiments;

[0023] Figure 7 A TOF depth camera plane detection scene diagram is shown according to some embodiments;

[0024] Figure 8 A TOF depth camera plane detection effect diagram is shown according to some embodiments;

[0025] Figure 9 A comparison diagram of the principle of a structured light depth camera and a TOF depth camera is shown according to some embodiments;

[0026] Figure 10 A comparison diagram of the scene detection effect of a binocular depth camera and a structured light depth camera is shown according to some embodiments;

[0027] Figure 11 An augmented reality application plane detection scene diagram is shown according to some embodiments;

[0028] Figure 12 A configuration flowchart of the display device is shown according to some embodiments;

[0029] Figure 13 A display device configuration flowchart is shown according to some embodiments;

[0030] Figure 14 An inter-frame difference image diagram is shown according to some embodiments;

[0031] Figure 15 A display device configuration flowchart is shown according to some embodiments;

[0032] Figure 16 A centroid and anchor point diagram is shown according to some embodiments. DETAILED DESCRIPTION

[0033] For the purposes of the present application, the exemplary embodiments described herein are intended to be illustrative only and not in a limiting sense as numerous modifications and variations are possible in light of the above teachings. Needlessly, the terms used in the description of the present application are briefly explained, only for the convenience of understanding the following described embodiments, and are not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and customary meanings.

[0034] Based on the exemplary embodiments described in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the claims appended to the present application. In addition, although the disclosure in the present application is introduced according to one or more examples, it should be understood that each aspect of these disclosures can also constitute a complete embodiment. It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and customary meanings.

[0035] Figure 1 For the purposes of the present application, the exemplary embodiments described herein are intended to be illustrative only and not in a limiting sense as numerous modifications and variations are possible in light of the above teachings. Needlessly, the terms used in the description of the present application are briefly explained, only for the convenience of understanding the following described embodiments, and are not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and customary meanings. Figure 1 As shown in FIG. 1, a user can operate a display device 200 through a mobile terminal 300 and a control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired means to control the display device 200. The user can input user instructions through the keys on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a notebook computer, and other smart devices can also be used to control the display device 200.

[0036] In some embodiments, the mobile terminal 300 can install a software application with the display device 200, and achieve connection communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The mobile terminal 300 can also display audio and video content on the display device 200 to achieve a synchronous display function. The display device 200 also communicates data with a server 400 through various communication means. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interaction to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide a smart network television function with computer support function.

[0037] Figure 2An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0038] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3 The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0039] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0040] In some embodiments, a "user interface" is a medium interface between an application program or an operating system and a user for interaction and information exchange, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of user interface is a graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, window, control, etc. interface element displayed in the display screen of an electronic device, wherein the control can include at least one of the following visual interface elements: icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc.

[0041] Figure 4 As shown in the schematic diagram of the software configuration in the display device 200 according to one or more embodiments of the present application, Figure 4 the system is divided into four layers from top to bottom, namely, an application layer (referred to as "application layer" for short), an application framework layer (referred to as "framework layer" for short), an Android runtime and system library layer (referred to as "system runtime library layer" for short), and a kernel layer. The kernel layer at least includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0042] Figure 5 As shown in the schematic diagram of the icon control interface display of the application program in the display device 200 according to one or more embodiments of the present application, Figure 5 the application layer includes at least one application program, which can display a corresponding icon control in the display, such as: a live TV application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control, etc. The live TV application program can provide live TV through different signal sources. The video on demand application program can provide videos from different storage sources. Unlike the live TV application program, the video on demand provides video display from certain storage sources. The media center application program can provide various multimedia content playing application programs. The application center can provide storage of various application programs.

[0043] Figure 6 An exemplary AR application scenario schematic diagram according to an exemplary embodiment is shown. As Figure 6As shown, AR application scenarios can involve one or more devices, such as a control device 100, a display device 200, a smart device 300, and an image acquisition device 500. The display device includes a monitor, an image acquisition device, and a controller. The image acquisition device can be fixed to or partially embedded in the housing of the display device, and is used to acquire color and depth images of the real scene directly in front of the display device. Users can enter the AR application by inputting user control commands through the control device or smart device. When entering the AR application, the image acquisition device can capture color and depth images of the real scene in real time (the depth image is the image where the distance from the image acquisition device to each point in the real scene is used as pixel values), and send the color image to the monitor for display as a data stream, and send the depth image to the controller for processing. Both the color and depth images consist of multiple consecutive frames.

[0044] While the display shows the real-world scene, users can input control commands in the AR application to add virtual objects to the real-world scene, ultimately merging them to form an augmented reality image. After acquiring the real-world scene image, the image acquisition device can first send the real-world scene image to the controller, triggering the controller to call up virtual objects and render their images. The virtual object images are then merged with the real-world scene image to form the final augmented reality image. Finally, the augmented reality image is sent to the display for display.

[0045] The virtual objects added to the real-world scene by the controller include, but are not limited to, text, images, 3D models, and video footage. In AR applications, the controller can blend the real-world scene with the virtual object footage, for example... Figure 6 In the scenario shown, when a user uses an AR interior design application, after acquiring the interior scene, they can use a simulated scene to apply AR scene textures to the interior scene. To do this, the AR application must first confirm the specific information of each scene in the real scene image. For example, in the depth image of the home interior scene acquired by the image acquisition device, the depth image of the home interior scene must first be subjected to planar detection to determine the ground position in the home interior scene.

[0046] It should be noted that the image acquisition devices include: TOF depth camera, structured light depth camera, and binocular depth camera.

[0047] Figure 7 This is a schematic diagram of a planar detection scene using a TOF depth camera according to one or more embodiments of this application. Figure 7As shown, the TOF depth camera acquires distance by measuring the time of flight of light, by continuously emitting laser pulses to the measured target, and using sensors to receive the reflected light of the laser pulses to obtain the exact distance of the target by calculating the round-trip time of flight of the laser pulses. When the TOF depth camera is started to shoot a home indoor scene in an AR application, there is often scattered light from the window in the home indoor scene, and the ground in the home indoor scene is mostly smooth ground of ceramic tile or floor material. The scattered light is often specularly reflected on the ground, which will interfere with the reflected light of the TOF depth camera, resulting in poor plane detection effect of the TOF depth camera on the ground of the home indoor scene. Figure 8 A schematic diagram of the plane detection effect of the TOF depth camera according to one or more embodiments of the present application is shown. As shown in Figure 8 As shown, the TOF depth camera can only acquire part of the ground position information in the home indoor scene, and it is difficult to obtain complete ground position information directly from the depth image acquired by the TOF depth camera.

[0048] Figure 9 A principle comparison diagram of the structured light depth camera and the TOF depth camera according to one or more embodiments of the present application is shown. As shown in Figure 9 As shown, the structured light depth camera projects light with certain structural features onto the photographed object by a near-infrared laser, and then collects it by a special infrared camera. The structured light can collect different image phase information according to different depth regions of the photographed object, and further convert the image phase information into depth information by an operation unit to obtain the three-dimensional structure of the photographed object. Since the structured light emitted by the near-infrared laser is easily disturbed by ambient light, it is difficult to collect image phase information on the specularly reflecting ground, and the structured light is usually discrete light spots, striped light and coded structured light, such as Figure 9 As shown in the principle comparison diagram, for the same plane, the structured light depth camera does not collect as comprehensive plane position information as the TOF depth camera, so compared with the laser pulses emitted by the TOF depth camera, the light of the structured light is more difficult to perform plane detection on the ground of the home indoor scene by directly using the depth image acquired by the structured light depth camera.

[0049] Figure 10 A same scene detection effect comparison diagram of the binocular depth camera and the structured light depth camera according to one or more embodiments of the present application is shown. As shown in Figure 10As shown, the binocular depth camera obtains two images of the measured object from different positions by the parallax principle and using imaging devices, and obtains the three-dimensional structure of the photographed object by calculating the positional deviation between the corresponding points of the images. Since the binocular depth camera needs to perform feature matching of the left and right cameras, when the light changes greatly, the image deviation of the binocular depth camera is large, which can cause matching failure. Therefore, the binocular depth camera is sensitive to the ambient light and is not suitable for monotonous and texture-lacking scenes. Compared with the TOF depth camera and the structured light depth camera, the binocular depth camera is more unsuitable for plane detection of the ground that is specular reflection.

[0050] Therefore, it can be seen that although the existing image acquisition device can directly use the depth camera to obtain the depth image of the indoor scene of the home to judge the ground position information, since the ground of the indoor scene of the home is often specular reflection, the acquisition process of the ground position information of the indoor scene of the home by the image acquisition device is affected, and the image acquisition device directly performs plane detection on the ground of the indoor scene of the home can only obtain part of the ground position information.

[0051] Therefore, the display device provided in the present application can accurately and effectively perform plane detection on the ground in the home scene, so as to solve the problem of poor imaging effect of the ground. It should be noted that since the TOF depth camera in the existing image acquisition device has good plane detection effect on the ground, the image acquisition device in the embodiments of the present application takes the TOF depth camera as an example for image acquisition, but the scheme shown in the present application is not limited to only using the TOF depth camera.

[0052] Figure 11A schematic diagram for a plane detection scene of an augmented reality application according to one or more embodiments of the present application. Taking an augmented reality application for implementing AR indoor design in a display device as an example, the augmented reality application needs to acquire a current real scene picture to determine a wall position, a ground position and a furniture placement position. Therefore, when entering the augmented reality application, an image acquisition device acquires an opening permission to acquire a color image and a depth image of the real scene picture. However, a controller can only determine ground position information according to a data stream of the depth image. However, only part of the ground position information can be acquired in the depth image directly captured by the image acquisition device. In order to more comprehensively and accurately perform plane detection on the ground, the present application shows an anchor point determination method to indirectly determine the ground position information through anchor points generated by contact points of a moving object and a to-be-detected plane. In a specific implementation, when the display device responds to a control instruction of a user entering the augmented reality application, a calibration program is immediately entered to perform plane detection on the ground through the calibration program. Taking a person as a moving object as an example, the person walks on the ground directly in front of the image acquisition device to acquire contact points of the moving object and the ground as anchor points, and the ground position information is determined according to the anchor points. Compared with a manner of directly determining the ground position information by using a depth camera to acquire a depth image, the present application can more comprehensively and accurately perform plane detection on the ground.

[0053] It is worth noting that, in a home indoor scene, the display device is usually in a fixed position and the user will not frequently move it, and the ground position is fixed and immovable. Therefore, after performing a calibration program and storing ground position information acquired by the calibration program, the stored ground position information can be called multiple times.

[0054] The display device shown in the present application includes a display, an image acquisition device configured to acquire a color image and a depth image of a real scene picture, the depth image includes a plurality of continuous frame images, and a controller can be configured to perform Figure 12 The anchor point determination method shown includes S1201-S1205 in the following steps.

[0055] In step S1201, a control instruction for entering an augmented reality application input by a user is received.

[0056] The controller of the display device can receive a plurality of control instructions input by the user during use. Each control instruction corresponds to a control function, and part of the control instructions can be used to enter the augmented reality application. The user can input the control instruction for entering the augmented reality application in different use environments, for example, the user opens an application program with AR function in a home indoor scene to complete the input. The application program with AR function can be an application program for implementing AR body sense game or AR indoor design function.

[0057] For some display devices, a smart voice system can also be built-in or externally connected, so that the user can also input the control instruction for entering the augmented reality application through the smart voice system. For example, the user can input voice content such as "open ×× AR application" to control the display device to start the AR application, and at this time, the controller can also receive the control instruction for entering the augmented reality application.

[0058] In step S1202, an inter-frame difference image of the depth image is obtained in response to the control instruction; the inter-frame difference image is obtained according to any two adjacent frames of images in the depth image.

[0059] In some embodiments, after receiving the control instruction for entering the augmented reality application, the display device can obtain an inter-frame difference image of the depth image in response to the control instruction.

[0060] In a specific implementation, as shown in Figure 13 In the step of obtaining the inter-frame difference image of the depth image, the controller is further configured to:

[0061] In step S1202-1, any two adjacent frames of images in the depth image are obtained.

[0062] After receiving the control instruction for entering the augmented reality application, the controller sends a request instruction for requesting the depth image to the image acquisition device, and the image acquisition device sends the captured color image and depth image to the controller, so that the controller can obtain the depth image; in order to obtain the inter-frame difference image, the controller first needs to obtain any two frames of images in the depth image.

[0063] In order to transmit the request instruction and the depth image, the image acquisition device needs to establish a specific form of communication connection relationship with the controller. The communication connection relationship can be based on wired communication or wireless communication, for example, for wired communication, the communicator of the image acquisition device can be a USB interface, and the image acquisition device can be connected to the display device through a USB data line and establish a communication connection relationship based on the USB transmission protocol. For wireless communication, the communicator can be a built-in Bluetooth, infrared, WIFI network, etc. Communication function module, and establish a wireless connection relationship through the corresponding transmission protocol.

[0064] In the process of obtaining the depth image by the controller, the request instruction and the depth image can be based on the same communication connection mode, or can be based on different communication connection modes respectively. For example, the controller can send an image acquisition request to the image acquisition device through Bluetooth connection, but due to the low transmission efficiency of Bluetooth connection, the image acquisition device sends the depth image to the controller through the WIFI network after receiving the image acquisition request.

[0065] It should be noted that when the depth image is acquired, the communication connection relationship between the image acquisition device and the controller can also be detected. When the image acquisition device and the controller can interact with each other, the controller can send an image acquisition request to the image acquisition device through the communicator. When the image acquisition device and the controller cannot interact with each other, the controller can display a connection interface of the controller and the image acquisition device through the display, so as to establish the communication connection relationship.

[0066] Figure 14 For a moving object with fast motion, a smaller time interval needs to be selected when setting the frame acquisition time interval. If the selected frame acquisition time interval is not appropriate, when the moving object has no overlapping part in the two frames, only two separate objects can be obtained through the inter-frame difference image, and the motion profile of the moving object cannot be obtained. For a moving object with slow motion, a larger time interval needs to be selected when setting the frame acquisition time interval. If the selected frame acquisition time interval is not appropriate, when the object almost completely overlaps in the two frames, the moving object cannot be detected. Therefore, the time interval for acquiring any two frames of the depth image can be set according to the actual situation according to the moving speed of the moving object, for example, as shown in Figure 14 Taking a person as the moving object, the frame acquisition time interval can be set to 500 ms (milliseconds), and the depth images of the adjacent two frames are as shown in Figure 14 The two depth images are subtracted Figure 14 only one depth image is shown, and the adjacent depth image cannot be recognized by the naked eye because of the small difference, and therefore is not shown), and the inter-frame difference image as shown in Figure 14 can be obtained.

[0067] In step S1202-2, it is determined whether the adjacent two frames of images meet the screening condition.

[0068] It should be noted that the inter-frame difference method can obtain the profile of the moving object by performing difference operation on the adjacent two frames of the video image sequence. When there is a moving object in the monitored scene, there will be a relatively obvious difference between the frames. The absolute value of the difference between the two frames is obtained by subtracting the two frames, and it is determined whether the absolute value of the difference between the two frames is within a certain threshold range to analyze the motion characteristics of the video or image sequence, and determine whether there is a moving object in the image sequence. At the same time, if the moving object in the depth image of the latter frame of the adjacent two frames of images blocks the background of the depth image of the former frame by less than a certain threshold, the inter-frame difference image of the adjacent two frames of images has the moving object.

[0069] In some embodiments, each frame of the depth image includes a plurality of pixel points, and each pixel point includes a pixel coordinate (u, v) and a pixel depth d (u，v); the latter one of the two adjacent frames of images is f n , and the former one of the two adjacent frames of images is f n-1 ; wherein n is the serial number of the frame of image, is the pixel depth corresponding to the pixel point (u, v) of the latter one of the two adjacent frames of images, is the pixel depth corresponding to the pixel point (u, v) of the latter one of the two adjacent frames of images, the abs() function is used to obtain the absolute value, delta is the acquisition error of the depth image, beta is the distance threshold of the moving object shielding the background, and the former frame of image and the latter frame of image are determined according to the depth image acquisition sequence.

[0070] Therefore, the screening condition is set as:

[0071] and or and

[0072] The inter-frame difference image screened according to the above screening condition contains the motion contour of the moving object.

[0073] In step S1202-3, if the two adjacent frames of images meet the screening condition, the inter-frame difference method is performed on the two adjacent frames of images meeting the screening condition, so as to obtain the inter-frame difference image of the two adjacent frames of images. The screening condition is used to determine whether the moving object exists in the inter-frame difference image.

[0074] When the two adjacent frames of images are screened through the screening condition, the value of f is marked as 0, the inter-frame difference image f (n) = f n -f n-1 .

[0075] In some embodiments, when a TOF depth camera is used, the acquisition error of the depth image can be set to 20 mm (millimeters) according to the absolute accuracy of the TOF depth camera. In a specific implementation, the acquisition error of the depth image can be set according to actual conditions. Meanwhile, in the adjacent two frames of depth images, if there are moving objects and backgrounds in the previous frame of image, and the positions of the moving objects in the previous frame change in the next frame of image, the changed backgrounds occluded by the moving objects are bound to change. In essence, for a pixel point of a frame of image, if there is an occlusion object, the depth coordinate of the current frame of occlusion object (for example, a person) at the pixel point is bound to be smaller than the depth value of the same pixel point in the previous frame. Therefore, by setting the distance threshold of the background occluded by the moving object, whether the moving object is included in the inter-frame difference image can be determined. The distance threshold of the background occluded by the moving object can be set according to the actual conditions of the indoor scene of the home, for example, the distance threshold of the background occluded by the moving object can be set to 100 mm. Therefore, when the screening condition is met, it indicates that the moving object is included in the inter-frame difference image.

[0076] In some embodiments, the controller is further configured to perform step S1202-4, if the adjacent two frames of depth images do not meet the screening condition, deleting the adjacent two frames of images.

[0077] If the adjacent two frames of depth images do not meet the screening condition, it indicates that the inter-frame difference image of the adjacent two frames of depth images does not include the moving object or the moving object does not move in the adjacent two frames of images. Therefore, the contact point between the moving object and the detection plane cannot be obtained according to the inter-frame difference image of the adjacent two frames of depth images. The controller can delete the depth images that do not meet the screening condition to reduce the memory occupation.

[0078] Step S1203, obtaining at least one target contour region according to the inter-frame difference image, the target contour region including a centroid and a plurality of pixel points, and the centroid being a mass center of the target region;

[0079] Figure 15 An exemplary schematic diagram of a target contour region in one or more embodiments of the present application is shown. In some embodiments, the controller is further configured to, in the step of obtaining at least one target contour region according to the inter-frame difference image, include:

[0080] Step S1203-1, obtaining a first gradient image of the inter-frame difference image;

[0081] Inter-frame difference image f' (u，v) is a discrete two-dimensional function, and the gradient represents the maximum value of the direction of a certain function at the point along the direction. Therefore, the inter-frame difference image f' (u，v)The gradient at the pixel point (u, v) is a two-dimensional vector composed of the partial derivative of the two-dimensional function of the point in the u direction and the partial derivative in the v direction, however, the gradient definition is only applicable to continuous gradient, and for images, the gradient needs to be calculated approximately using finite difference method, for example, the central difference method of finite difference method can be used to approximate the gradient to obtain the image gradient along the u direction and the image gradient along the v direction; the vertical edge of the motion contour of the moving object in the inter-frame difference image can be obtained according to the image gradient along the u direction, and the horizontal edge of the motion contour of the moving object in the inter-frame difference image can be obtained according to the image gradient along the v direction.

[0082] In a specific implementation, the inter-frame difference image includes a plurality of pixel points, and each pixel point includes pixel coordinates (u, v); the controller is further configured to: in the step of obtaining the first gradient image of the inter-frame difference image, obtain first gradient coordinates (u', v) derived along the u direction and second gradient coordinates (u, v') derived along the v direction of the pixel coordinates (u, v) of each pixel point in the inter-frame difference image;

[0083] The second gradient image generated according to the first gradient coordinates (u', v) and the third gradient image generated according to the second gradient coordinates (u, v') are superimposed to obtain the first gradient image of the inter-frame difference image.

[0084] In step S1203-2, the pixel points with approximate gradients in the first gradient image are clustered according to a clustering algorithm, the first gradient image corresponds to the pixel points in the inter-frame difference image, to obtain a plurality of clustered contour regions in the inter-frame difference image;

[0085] The clustering algorithm includes K-means clustering, DBSCAN clustering, hierarchical clustering, SOM clustering and the like, and the DBSCAN clustering can be used to cluster the pixel points in the first gradient image, and it should be noted that the application is not limited to only using the DBSCAN clustering.

[0086] For example, the DBSCAN clustering method, the density-based spatial clustering of applications with noise (DBSCAN) defines a cluster as the maximal set of density-connected points, and can divide regions with sufficiently high density into clusters and find clusters of arbitrary shape in noisy spatial databases. For a plurality of sample points in a two-dimensional image, a scanning radius and a minimum number of contained points are first determined, and an unvisited point is optionally selected as a starting point. All nearby points within the scanning radius from the unvisited point are screened. If the number of nearby points is greater than or equal to the minimum number of contained points, a cluster is formed by the current point and the nearby points, and the unvisited point is marked as visited. The unvisited points in the cluster are recursively processed in the same way to expand the cluster. If the number of nearby points is less than the minimum number of contained points, the unmarked point is marked as a noise point. If the cluster is sufficiently expanded, i.e., all points in the cluster are marked as visited, the unvisited points are processed in the same way. According to the DBSCAN clustering method, a plurality of clusters can be divided among a plurality of coordinate points in a two-dimensional image according to density, and each cluster is a cluster contour region.

[0087] In step S1203-3, it is detected whether the cluster contour region meets the constraint condition.

[0088] In some embodiments, the controller is further configured to, in the step of detecting whether the cluster contour region meets the constraint condition, convert the inter-frame difference image into a point cloud image.

[0089] In a specific implementation, each pixel point in the inter-frame difference image further includes a depth value d (u，v) .

[0090] The controller is further configured to, in the step of converting the inter-frame difference image into a point cloud image, convert the pixel coordinates (u, v) of each pixel point in the inter-frame difference image into point cloud coordinates (X, Y, Z) to obtain a point cloud image; Z = d (u,v) / s, X = Z*(u-c x ) / f x , Y = Z*(v-c y ) / f y .

[0091] Wherein, f x f y c x ,c y is the intrinsic matrix of the camera, and s is the scaling factor of the depth image.

[0092] A local normal vector of each cluster contour region in the point cloud image is obtained.

[0093] If the included angle between the local normal vector of any one of the clustered contour regions and the y-axis in the coordinate system of the point cloud image is within a constraint threshold, the clustered contour region meets the constraint condition.

[0094] The controller is further configured to: in the step of obtaining the local normal vector of each clustered contour region in the point cloud image, divide each clustered region into sub-regions according to a specified size;

[0095] The local normal vector of each sub-region is calculated.

[0096] In some embodiments, the specified size is a 4*4 pixel grid.

[0097] Taking a person as a moving object, the motion contour of the person can be obtained according to the foregoing steps, and the motion contour of the person is further divided into a plurality of clustered contour regions. Since each part of the person has its unique characteristics, such as the head being round and the legs being cylindrical, the leg region can be screened out according to the constraint condition.

[0098] In some embodiments, the constraint condition is:

[0099] The included angle between the local normal vector of the clustered contour region and the y-axis in the coordinate system of the point cloud image is within the range of -90°±10°.

[0100] Step S1203-4, if any one of the clustered contour regions meets the constraint condition, determining the contour region as a target contour region, wherein the constraint condition is used to determine the angle of the local normal vector of the clustered contour region.

[0101] It should be noted that, since there are a plurality of clustered contour regions, there can be more than one clustered contour region meeting the constraint condition, for example, Figure 15 As shown in the embodiment of taking a person as a moving object, the target contour region is the leg region of the person. When the person moves in the inter-frame difference image by overlapping the two legs, only one target contour region can be obtained. When the person moves in the inter-frame difference image by taking a step, two target contour regions can be obtained. Since the constraint conditions are different, the target contour region can be the entire leg or the calf region. The specific setting mode of the target contour region can be set according to the actual situation.

[0102] Step S1204, selecting a pixel point with the lowest height and the farthest distance from the centroid in the target contour region as an anchor point.

[0103] Figure 16 The centroid and anchor point schematic diagram shown in one or more embodiments of the present application is as shown in Figure 16As shown, according to the target contour region acquisition method shown in the present application, each inter-frame difference image includes at least one target contour region, and taking a walking person as an example of the moving object included in the inter-frame difference image, two target contour regions are included, i.e., the regions where the two legs of the person are located. Therefore, two mass centers are included in the inter-frame difference image, i.e., the mass centers of the two legs, and correspondingly, in the two target contour regions, a point farthest from the mass center in each target contour region is taken as an anchor point.

[0104] It should be noted that the mass center of the image, also referred to as the center of gravity of the image, in the embodiments of the present application, the pixel value of each point in the image can be understood as the mass of the pixel point, and the difference between the mass center of the object in space is that, since the image is two-dimensional, the mass center is found independently on the x-axis and y-axis of the image, i.e., for the mass center in the x-axis direction, the sum of the pixels on the left and right sides of the mass center in the image is equal; for the mass center in the y-axis direction, the sum of the pixels on the upper and lower sides of the mass center in the image is equal.

[0105] Taking a person as an example of a moving object, the target contour region is usually the entire region from the leg to the foot, and therefore, during the walking process of the person, the point farthest from the mass center is usually the contact point of the heel, sole or toe with the ground, and therefore, selecting the point farthest from the mass center in the target contour region as the anchor point is to select the contact point of the person with the ground, so as to further indirectly detect the ground plane through multiple contact points of the person with the ground.

[0106] It should be noted that, according to different constraints, the leg region can be the entire leg region from the thigh to the foot, or can be defined as the entire calf region from the knee to the foot, and the leg region can be selected according to the actual situation according to different actual constraints.

[0107] In step S1205, an anchor point set is generated according to all the anchor points of the plurality of inter-frame difference images, and the anchor points included in the anchor point set are used for plane fitting.

[0108] In some embodiments, the controller is further configured to, in the step of generating an anchor point set according to all the anchor points of the plurality of inter-frame difference images, judge whether the number of anchor points in the anchor point set is greater than a first threshold value.

[0109] In some embodiments, if the number of anchor points reaches the first threshold value, it means that the number of collected anchor points can be used for plane detection, and for example, the first threshold value can be set to 10.

[0110] If the number of anchor points in the anchor point set is greater than the first threshold value, the maximum distance difference value of the point cloud coordinates of all the anchor points in the anchor point set along the x-axis direction and the maximum distance difference value along the z-axis direction are obtained.

[0111] In some embodiments, in order to further determine whether the obtained anchor points are all available, the maximum distance difference of the point cloud coordinates of all anchor points in the anchor point set along the x-axis direction and the maximum distance difference along the z-axis direction are obtained, and the spatial movement distance of the mobile object on the plane is determined. If the spatial movement distance reaches a certain range, it is proved that the anchor points in the anchor point set are distributed at a suitable spatial distance, and it is further determined that the obtained anchor points can be used for plane fitting.

[0112] determining whether the maximum distance difference of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference along the z-axis direction are both greater than a second threshold value;

[0113] In some embodiments, when the maximum distance difference of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference along the z-axis direction are both greater than or equal to the second threshold value, it can be determined that the distribution of all anchor points in the anchor point set is at a suitable spatial distance. For example, the second threshold value can be set to 1000 mm.

[0114] If the maximum distance difference of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference along the z-axis direction are both greater than the second threshold value, the anchor point region formed by all anchor points in the anchor point set is divided into N intervals (N is a natural number);

[0115] In some embodiments, if the maximum distance difference of the point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference along the z-axis direction are both greater than the second threshold value, and the second threshold value is set to 1000 mm, for example, on the XOZ plane, the anchor point set forms at least one 1000*1000 mm 2 space region, and the region is divided into intervals along the XOZ plane with a size of 500*500 mm 2 , at least 4 intervals are formed.

[0116] determining whether the number of anchor points in each interval of the N intervals is greater than 0;

[0117] If the number of anchor points in each interval of the N intervals is greater than 0, the anchor points contained in the anchor point set can be used for plane fitting.

[0118] In some embodiments, taking the above-mentioned embodiments as an example, if the anchor point region is divided into 4 intervals on the XOZ plane, and the number of anchor points in each interval is greater than 0, it is determined that the anchor points in each interval are uniformly distributed, and it can be determined that the anchor point set can be used to determine the fitting plane.

[0119] Based on the above display device, the application provides an anchor point determination method, comprising:

[0120] obtaining an inter-frame difference image according to two adjacent images in the depth image;

[0121] obtaining at least one target contour region from the inter-frame difference image;

[0122] determining a centroid of the target contour region; the centroid is the mass center of the target region;

[0123] selecting a pixel point in the target contour region as an anchor point, the anchor point being used for plane fitting, the pixel point being the pixel point with the lowest height and the farthest distance from the centroid;

[0124] The method further comprises:

[0125] generating an anchor point set according to all anchor points of the plurality of inter-frame difference images;

[0126] determining whether the number of anchor points in the anchor point set is greater than a first threshold value;

[0127] if the number of anchor points in the anchor point set is greater than the first threshold value, obtaining a maximum distance difference value along an x-axis direction and a maximum distance difference value along a z-axis direction of point cloud coordinates of all anchor points in the anchor point set;

[0128] determining whether the maximum distance difference value along the x-axis direction and the maximum distance difference value along the z-axis direction of the point cloud coordinates of all anchor points are both greater than a second threshold value;

[0129] if the maximum distance difference value along the x-axis direction and the maximum distance difference value along the z-axis direction of the point cloud coordinates of all anchor points are both greater than the second threshold value, dividing an anchor point region formed by all anchor points in the anchor point set into N intervals (N is a natural number);

[0130] determining whether the number of anchor points in each interval of the N intervals is greater than 0;

[0131] if the number of anchor points in each interval of the N intervals is greater than 0, the anchor points contained in the anchor point set can be used for plane fitting.

[0132] It should be understood that the specific implementation of each step in the above anchor point determination method can refer to the foregoing display device embodiments, which will not be described here. As can be seen from the above embodiments, the anchor point determination method shown in the present application indirectly determines the fitting plane through the anchor point set, which is essentially to perform plane fitting on the anchor points generated by the contact points of the moving object and the to-be-detected plane to indirectly determine the to-be-detected plane. Compared with the method of directly determining the plane by using the depth image obtained by the depth camera, the present application can enable the display device to more comprehensively and accurately perform plane detection.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0134] The foregoing description has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or can be acquired from practice of the embodiments. The described embodiments were chosen and described in order to best explain the principles of the embodiments and its practical application. This enables others skilled in the art to best use the embodiments in various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A display device, characterized by comprising: The application relates to an augmented reality device, comprising: a display; an image acquisition device configured to acquire a depth image of a real scene picture; the depth image comprises a plurality of continuous frame images; a controller configured to: receive a control instruction input by a user for entering an augmented reality application; in response to the control instruction, acquire a plurality of inter-frame difference images of the depth image; wherein the inter-frame difference images are obtained according to adjacent two frame images in the depth image; acquire at least one target contour region according to the inter-frame difference images, the target contour region comprises a mass center and a plurality of pixel points, and the mass center is the mass center of the target contour region; select a pixel point in the target contour region as an anchor point, the pixel point is the farthest from the mass center and has a lower height than the mass center; generate an anchor point set according to all anchor points of a plurality of inter-frame difference images, and the anchor points in the anchor point set are used for plane fitting.

2. The display device according to claim 1, wherein The controller is further configured to: in the step of acquiring a plurality of inter-frame difference images of the depth image, acquire any adjacent two frame images in the depth image; determine whether the adjacent two frame images meet a screening condition; if the adjacent two frame images meet the screening condition, perform an inter-frame difference method on the adjacent two frame images meeting the screening condition to obtain an inter-frame difference image of the adjacent two frame images; wherein the screening condition is used to determine whether a moving object exists in the inter-frame difference image.

3. The display device of claim 1, wherein, The controller is further configured to: in the step of acquiring at least one target contour region according to the inter-frame difference images, acquire a first gradient image of the inter-frame difference image; the first gradient image comprises pixel points with approximate gradients; cluster the pixel points with approximate gradients in the first gradient image according to a clustering algorithm, the pixel points in the first gradient image correspond to pixel points in the inter-frame difference image, to obtain a plurality of clustering contour regions in the inter-frame difference image; determine whether the clustering contour regions meet a constraint condition; if any one of the clustering contour regions meets the constraint condition, the clustering contour region is determined as a target contour region, wherein the constraint condition is used to determine the angle of a local normal vector of the clustering contour region.

4. The display device of claim 2, wherein, Each of the depth images comprises a plurality of pixels, each pixel comprising a pixel coordinate (u, v) and a pixel depth d corresponding to the pixel coordinate (u，v) ; The screening condition is that: and or and wherein n is the serial number of the frame image, is the pixel depth corresponding to the pixel point of the latter frame image in the two adjacent frame images, is the pixel depth corresponding to the pixel point of the former frame image in the two adjacent frame images, the abs() function is used to obtain the absolute value, delta is the acquisition error of the depth image, beta is the distance threshold of the moving object shielding the background, and the former frame image and the latter frame image are determined in front of and behind according to the depth image acquisition sequence.

5. The display device of claim 3, wherein, the inter-frame difference image comprises a plurality of pixel points, and each pixel point comprises pixel coordinates (u, v); the controller is further configured to: in the step of acquiring the first gradient image of the inter-frame difference image, acquire a first gradient coordinate (u', v) obtained by deriving the pixel coordinates (u, v) of each pixel point in the inter-frame difference image along a u direction and a second gradient coordinate (u, v') obtained by deriving the pixel coordinates (u, v) along a v direction; superimpose a second gradient image generated according to the first gradient coordinate (u', v) and a third gradient image generated according to the second gradient coordinate (u, v') to acquire the first gradient image of the inter-frame difference image.

6. The display device of claim 5, wherein, The controller is further configured to: in the step of determining whether the clustering contour regions meet a constraint condition, convert the inter-frame difference image into a point cloud image; acquire a local normal vector of each clustering contour region in the point cloud image; If an angle between a local normal vector of any one of the cluster contour regions and a y-axis in a coordinate system of the point cloud image is within a constraint threshold, the cluster contour region meets a constraint condition.

7. The display device of claim 6, wherein, each pixel point in the inter-frame difference image further comprises a depth value d (u，v) ; The controller is further configured to: In the step of converting the inter-frame difference image into a point cloud image, the pixel coordinates (u, v) of each pixel point in the inter-frame difference image are converted into point cloud coordinates (X, Y, Z) to obtain a point cloud image; Z = d (u,v) / s, X = Z * (u - c x ) / f x , Y = Z * (v - c y ) / f y ; where f x f y c x ,c y is the intrinsic matrix of the camera and s is the scaling factor of the depth image.

8. The display device of claim 7, wherein, The controller is further configured to: In the step of generating an anchor point set according to all anchor points of the plurality of inter-frame difference images, it is determined whether the number of anchor points in the anchor point set is greater than a first threshold value; If the number of anchor points in the anchor point set is greater than the first threshold value, a maximum distance difference value of point cloud coordinates of all anchor points in the anchor point set along an x-axis direction and a maximum distance difference value of point cloud coordinates of all anchor points in the anchor point set along a z-axis direction are obtained; It is determined whether the maximum distance difference value of point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of point cloud coordinates of all anchor points along the z-axis direction are both greater than a second threshold value; If the maximum distance difference value of point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of point cloud coordinates of all anchor points along the z-axis direction are both greater than the second threshold value, an anchor point region formed by all anchor points in the anchor point set is divided into N intervals; It is determined whether the number of anchor points in each interval of the N intervals is greater than 0; If the number of anchor points in each interval of the N intervals is greater than 0, the anchor points contained in the anchor point set can be used for plane fitting.

9. An anchor point determination method, characterized by, The method comprises: obtaining an inter-frame difference image according to two adjacent frames in a depth image; obtaining at least one target contour region from the inter-frame difference image; determining a centroid of the target contour region; the centroid is a mass center of the target contour region; selecting a pixel point with a lowest height in the target contour region and farthest from the centroid as an anchor point, the anchor point being used for plane fitting.

10. The anchor point determination method of claim 9, wherein, The method further comprises: generating an anchor point set according to all anchor points of the plurality of inter-frame difference images; determining whether the number of anchor points in the anchor point set is greater than a first threshold value; If the number of anchor points in the anchor point set is greater than the first threshold value, a maximum distance difference value of point cloud coordinates of all anchor points in the anchor point set along an x-axis direction and a maximum distance difference value of point cloud coordinates of all anchor points in the anchor point set along a z-axis direction are obtained; It is determined whether the maximum distance difference value of point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of point cloud coordinates of all anchor points along the z-axis direction are both greater than a second threshold value; If the maximum distance difference value of point cloud coordinates of all anchor points along the x-axis direction and the maximum distance difference value of point cloud coordinates of all anchor points along the z-axis direction are both greater than the second threshold value, an anchor point region formed by all anchor points in the anchor point set is divided into N intervals; It is determined whether the number of anchor points in each interval of the N intervals is greater than 0; If the number of anchor points in each interval of the N intervals is greater than 0, the anchor points contained in the anchor point set can be used for plane fitting.

Images