Plane reconstruction method and display device

Abstract

The present application relates to the field of VR / AR technology, and provides a plane reconstruction method and a display device, which obtain a depth image captured by a depth camera, extract three-dimensional point cloud data containing multiple 3D points based on the depth image, select two 3D points from the multiple 3D points, and determine the distance between the two 3D points on an axis perpendicular to the plane to be reconstructed, traverse all 3D points in the multiple 3D points, generate a 3D point set of the plane to be reconstructed according to the determined distances, and reconstruct a plane for displaying an enhanced image according to the 3D point set. The 3D point set of the plane to be reconstructed is generated according to the height difference between the two 3D points on the axis perpendicular to the plane to be reconstructed, which is simple to calculate, efficient, and has a higher accuracy in reconstructing the plane.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality (VR) and augmented reality (AR), and in particular to a plane reconstruction method and a display device. Background Art

[0002] Virtual Reality (VR) technology uses computer simulation technology to generate an interactive, fully digital three-dimensional field of view to simulate a virtual environment, giving people a sense of environmental immersion. Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking, intelligent interaction, and sensing. After simulating the generated virtual information such as text, images, three-dimensional models, music, and videos, it is applied to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0003] VR / AR displays are widely used in various industries, such as education and training, fire drills, virtual driving, real estate, marketing and other industries, bringing users an immersive visual feast. Through AR technology, scene effects are superimposed on video content, fully demonstrating the unique visual impact experience brought by intelligence. Videos with personalized special effects will make users have interesting dynamic effects of body response, and show the advantage of high stability in extreme environments.

[0004] Based on visual tracking detection technology, the surrounding environment is understood according to the images taken by the camera, and the plane used to display the enhanced image is reconstructed, thereby achieving AR enhanced special effects.

[0005] At present, all planes are generally reconstructed based on point cloud information, and then the angle between the normal vector of the plane and the direction of gravity is used to further determine whether the plane is horizontal or vertical. However, the calculation of the angle between the normal vector and the direction of gravity is complex and time-consuming, and the accuracy of the reconstructed plane is low. Summary of the invention

[0006] The embodiments of the present application provide a plane reconstruction method and a display device for quickly and accurately reconstructing a plane.

[0007] In a first aspect, an embodiment of the present application provides a plane reconstruction method, comprising:

[0008] Acquire the acquired depth image, and extract three-dimensional point cloud data based on the depth image, wherein the three-dimensional point cloud data includes a plurality of 3D points;

[0009] Selecting two 3D points from the multiple 3D points, and determining a distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, wherein the distance between the two selected 3D points is less than a set distance threshold;

[0010] Traversing all 3D points in the plurality of 3D points, and generating a 3D point set of the to-be-reconstructed plane according to the determined distances;

[0011] A plane to be reconstructed for displaying an enhanced image is reconstructed according to the 3D point set.

[0012] In a second aspect, an embodiment of the present application provides a display device, including a display, a memory, and a controller:

[0013] The display, connected to the controller, is configured to display the enhanced image on the reconstructed plane;

[0014] The memory is connected to the controller and configured to store computer program instructions;

[0015] The controller is configured to perform the following operations according to the computer program instructions:

[0016] Acquire the acquired depth image, and extract three-dimensional point cloud data based on the depth image, wherein the three-dimensional point cloud data includes a plurality of 3D points;

[0017] Selecting two 3D points from the multiple 3D points, and determining a distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, wherein the distance between the two selected 3D points is less than a set distance threshold;

[0018] Traversing all 3D points in the plurality of 3D points, and generating a 3D point set of the to-be-reconstructed plane according to the determined distances;

[0019] A plane to be reconstructed for displaying an enhanced image is reconstructed according to the 3D point set.

[0020] In a third aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the plane reconstruction method provided in an embodiment of the present application.

[0021] In the above-mentioned embodiment of the present application, three-dimensional point cloud data containing multiple 3D points are extracted through the depth image captured by the camera, all 3D points are traversed, two 3D points are selected from the multiple 3D points each time, and the distance between the two 3D points on the axis perpendicular to the plane to be reconstructed is determined. After the traversal, a 3D point set of the plane to be reconstructed is generated according to the determined distances, and the plane for displaying the enhanced image is reconstructed according to the 3D point set. The 3D point set of the plane to be reconstructed is generated according to the height difference between the two 3D points on the axis perpendicular to the plane to be reconstructed. With respect to reconstructing all planes, the normal vector and the angle of the gravity direction are used to determine the selection of the required plane. The embodiment of the present application is simple to calculate, efficient, and has higher accuracy in reconstructing the plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 The system architecture diagram provided by the embodiment of the present application is exemplarily shown;

[0024] Figure 2 The hardware structure diagram of the display device provided by the embodiment of the present application is exemplarily shown;

[0025] Figure 3 The plane reconstruction method provided by the embodiment of the present application is exemplarily shown;

[0026] Figure 4 The point cloud diagram of the extracted three-dimensional point cloud data provided by the embodiment of the present application is exemplarily shown;

[0027] Figure 5 A schematic diagram of selecting 3D points corresponding to a horizontal plane from multiple 3D points provided in an embodiment of the present application is exemplarily shown;

[0028] Figure 6 A schematic diagram of selecting 3D points corresponding to a vertical surface from multiple 3D points provided in an embodiment of the present application is exemplarily shown;

[0029] Figure 7 A schematic diagram of screening 3D points corresponding to an inclined surface from multiple 3D points provided in an embodiment of the present application is exemplarily shown;

[0030] Figure 8 The reconstructed plane effect diagram provided by the embodiment of the present application is exemplarily shown;

[0031] Fig. 9 The functional structure diagram of the display device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0032] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0033] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0034] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to orders other than those given in the diagrams or descriptions of the embodiments of this application.

[0035] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0036] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0037] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the display device has a camera for collecting depth images in a real environment. The embodiment of the present application does not restrict the type of camera, for example, it can be an RGBD camera, a binocular camera, etc. The display device processes the depth image collected by the camera to obtain a 3D point set for reconstructing a plane, reconstructs the plane based on the obtained 3D point set, and displays an enhanced image on the reconstructed plane, such as Figure 1As shown, a cute dinosaur is displayed on the reconstructed horizontal plane, and the dinosaur has added an enhanced special effect of "breathing fire", which enhances the sense of immersion and authenticity of the picture, thereby bringing a strong and shocking visual impact to the user, and producing interesting dynamic effects of limb response, for example, the body naturally avoids the fire-breathing special effect.

[0038] It should be noted that Figure 1 The reconstructed horizontal plane is only an example. The present application can be applied to reconstruct various planes such as vertical planes and inclined planes, and there is no restrictive requirement on the shape of the reconstructed plane. It can be a regular shape such as a rectangle, a circle, or an irregular polygon.

[0039] Take the display device as a smart TV as an example. Figure 2 The structure diagram of the display device provided by the embodiment of the present application is exemplarily shown. Figure 2 As shown, the first terminal 100 includes at least one of a controller 250, a tuner and demodulator 210, a communicator 220, a detector 230, an input / output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, a user interface 265, and an external device interface 240.

[0040] In some embodiments, the display 275 includes a display screen component for presenting images, and a driving component for driving image display, and a component for receiving an image signal output from the first processor to display video content and images as well as a menu control interface.

[0041] In some embodiments, the display 275 is a projection display, and may also include a projection device and a projection screen.

[0042] In some embodiments, the communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

[0043] In some embodiments, the first terminal 100 can establish control signal and data signal transmission and reception with the external device through the communicator 220.

[0044] In some embodiments, the user interface 265 may be configured to receive a control signal from an external device.

[0045] In some embodiments, the detector 230 includes a light receiver, an image collector, a temperature sensor, a sound collector, etc., for collecting signals of the external environment or interactions with the outside.

[0046] In some embodiments, the input / output interface 255 is configured to perform data transmission between the controller 250 and other external devices or other controllers 250, such as receiving video signal data and audio signal data, or command instruction data from external devices.

[0047] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as a high-definition multimedia interface HDMI interface, an analog or digital high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. It is also possible that the above multiple interfaces form a composite input / output interface.

[0048] In some embodiments, the tuner-demodulator 210 is configured to receive broadcast television signals via wired or wireless reception, and can perform modulation and demodulation processing such as amplification, mixing and resonance, and demodulate audio and video signals from multiple wireless or wired broadcast television signals. The audio and video signals may include television audio and video signals carried in the television channel frequency selected by the user, as well as EPG data signals.

[0049] In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, which can send a control signal based on the user's selection to make the modem respond to the TV signal frequency selected by the user and modulate and demodulate the TV signal carried by the frequency.

[0050] In some embodiments, the controller 250 controls the operation of the first terminal and responds to the user's operation through various software control programs stored in the memory. The controller 250 can control the overall operation of the first terminal 100. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 can perform an operation related to the object selected by the user command.

[0051] like Figure 2 As shown, the controller 250 includes at least one of a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor 270, an audio processor 280, other processors 253 (for example: a graphics processor (Graphics Processing Unit, GPU), a central processing unit 254 (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus). Among them, the communication bus connects various components.

[0052] In some embodiments, RAM 251 is used to store temporary data for the operating system or other running programs.

[0053] In some embodiments, ROM 252 is used to store various system startup instructions.

[0054] In some embodiments, ROM 252 is used to store a basic input / output system, called Basic Input Output System (BIOS), which is used to complete the power-on self-test of the system, the initialization of each functional module in the system, the basic input / output driver of the system, and the booting operating system.

[0055] In some embodiments, when receiving the power-on signal, the power of the first terminal 100 starts to start, and the CPU runs the system startup instruction in the ROM 252, and copies the temporary data of the operating system stored in the memory to the RAM 251, so as to start or run the operating system. After the operating system is started, the CPU copies the temporary data of various applications in the memory to the RAM 251, and then starts or runs various applications.

[0056] In some embodiments, the CPU processor 254 is used to execute operating system and application instructions stored in the memory, and to execute various applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video contents.

[0057] In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the first terminal 100 in the pre-power-on mode and / or display the screen in the normal mode. The one or more sub-processors are used for an operation in the standby mode and the like.

[0058] In some embodiments, the graphics processor 253 is used to generate various graphic objects, such as icons, operation menus, and user input command display graphics, etc. It includes an operator that performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes. It also includes a renderer that renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

[0059] In some embodiments, the video processor 270 is configured to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal, and obtain a signal that can be directly displayed or played on the first terminal 100.

[0060] In some embodiments, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc.

[0061] In some embodiments, the audio processor 280 is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to the standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

[0062] The power supply 290, under the control of the controller 250, uses the power input from the external power supply to provide power supply support for the first terminal 100. The power supply 290 may include a built-in power supply circuit installed inside the first terminal 100, or may be an external power supply installed on the first terminal 100, providing a power interface for an external power supply in the first terminal 100.

[0063] The user interface 265 is used to receive a user input signal, and then send the received user input signal to the controller 250. The user input signal may be a remote control signal received by an infrared receiver, and may receive various user control signals through a network communication module.

[0064] The memory 260 includes storing various software modules for driving the first terminal 100. For example, the various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

[0065] It should be noted that Figure 1-Figure 2 This is just an example. The display device in this application can be a terminal with video playback and interactive functions, such as a smart phone, a laptop, a desktop, a tablet, AR glasses, VR glasses, etc.

[0066] based on Figure 1 The scene shown, Figure 3 The following is an exemplary flow chart of the planar reconstruction method provided in the embodiment of the present application. Figure 3 As shown, the process is executed by the display device and mainly includes the following steps:

[0067] S301: Acquire a collected depth image, and extract three-dimensional point cloud data based on the depth image, where the three-dimensional point cloud data includes multiple 3D points.

[0068] In this step, the display device turns on the camera to collect a depth image of the environment in which the display device is located, and extracts three-dimensional point cloud data based on the collected depth image.

[0069] Figure 4The point cloud diagram of the extracted three-dimensional point cloud data provided by the embodiment of the present application is exemplarily shown. Figure 4 As shown, three-dimensional point data extracted from a frame of depth image contains multiple 3D points.

[0070] S302: Select two 3D points from a plurality of 3D points, and determine the distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, wherein the distance between the two selected 3D points is less than a set distance threshold.

[0071] In this step, two 3D points whose distance is less than a set distance threshold are randomly selected from multiple 3D points, and the distance between the two selected 3D points on the axis perpendicular to the plane to be reconstructed is determined. In the embodiments of the present application, the method of determining the distance between the two 3D points on the axis perpendicular to the plane to be reconstructed is different depending on the plane to be reconstructed.

[0072] In an optional embodiment, when the plane to be reconstructed is a horizontal plane or a vertical plane, during the execution of S302, the coordinates of the two selected 3D points on the first axis parallel to the plane to be reconstructed are respectively set to preset values, and the coordinates of the two 3D points on the second axis parallel to the plane to be reconstructed are respectively reduced according to a preset reduction ratio to obtain the updated first coordinates of the two 3D points, and based on the updated first coordinates, the distance between the two 3D points on the axis perpendicular to the plane to be reconstructed is determined.

[0073] Take the horizontal plane as an example, Figure 5As shown, the Y axis is perpendicular to the horizontal plane XOZ, and whether the two 3D points are 3D points corresponding to the horizontal plane is determined according to the height difference between the two 3D points and the horizontal plane, that is, the 3D points corresponding to the horizontal plane are selected from the 3D points according to the distance between the two selected 3D points on the axis perpendicular to the plane to be reconstructed. In specific implementation, two 3D points whose distance is less than the set distance threshold are arbitrarily selected from the multiple extracted 3D points, and are respectively recorded as A and B, where the coordinates of point A are represented by (x1, y1, z1), and the coordinates of point B are represented by (x2, y2, z2). Optionally, the two selected 3D points are the two 3D points with the closest distance. Set the coordinate values ​​x1 and x2 of the two 3D points A and B on the X axis to 0, and reduce the coordinate values ​​of the two 3D points A and B on the Z axis by 10,000 times, and obtain the updated first coordinate of point A as (0, y1, z1 / 1000), and the first coordinate of point B as (0, y2, z2 / 1000). Since the coordinates on the Z axis after reduction are smaller, their influence on whether the two selected 3D points belong to the horizontal plane can be ignored. According to the first coordinates of the two 3D points A and B, determine the distance between the two 3D points A and B on the Y axis, that is, determine the height difference between the two 3D points A and B to the horizontal plane. The distance between the two 3D points A and B on the Y axis dy = |y2-y1| Compare dy with the preset height difference ay. If dy < ay, it indicates that the two 3D points A and B are two points on the horizontal plane. If dy ≥ ay, it indicates that the two 3D points A and B do not belong to the horizontal plane.

[0074] Take the plane to be reconstructed as a vertical plane as an example, Figure 6As shown, the Z axis is perpendicular to the vertical plane XOY, and the height difference between the two 3D points and the vertical plane is used to determine whether the two selected 3D points are 3D points corresponding to the vertical plane, that is, the 3D points corresponding to the vertical plane are selected from the 3D points according to the distance between the two selected 3D points on the axis perpendicular to the plane to be reconstructed. In specific implementation, two 3D points whose distance is less than the set distance threshold are arbitrarily selected from the multiple extracted 3D points, and are denoted as C and D respectively, where the coordinates of point C are represented by (x3, y3, z3), and the coordinates of point D are represented by (x4, y4, z4). Optionally, the two selected 3D points are the two 3D points with the closest distance. Set the coordinate values ​​x3 and x4 of the two 3D points C and D on the X axis to 0, and reduce the coordinate values ​​of the two 3D points C and D on the Y axis by 10,000 times, and obtain the updated first coordinate of point C as (0, y3 / 1000, z3), and the first coordinate of point D as (0, y4 / 1000, z4). According to the first coordinates of the two 3D points C and D, determine the distance between the two 3D points C and D on the Z axis, that is, determine the height difference between the two 3D points C and D and the vertical plane. The distance between the two 3D points C and D on the Z axis is dz = |z4-z3|, and dz is compared with the preset height difference value az. If dz < az, it indicates that the two 3D points C and D are two points on the vertical plane. If dy ≥ az, it indicates that the two 3D points C and D do not belong to the vertical plane.

[0075] In another optional implementation, when the plane to be reconstructed is an inclined plane, in the process of executing S302, the axis corresponding to the plane to be reconstructed is determined according to the inclination angle of the plane to be reconstructed relative to the horizontal plane and the vertical plane. Specifically, if the inclination angle of the plane to be reconstructed relative to the horizontal plane is greater than the inclination angle of the plane to be reconstructed relative to the vertical plane, the axis corresponding to the plane to be reconstructed is determined to be the axis perpendicular to the vertical plane; if the inclination angle of the plane to be reconstructed relative to the horizontal plane is less than the inclination angle of the plane to be reconstructed relative to the vertical plane, the axis corresponding to the plane to be reconstructed is determined to be the axis perpendicular to the horizontal plane; if the inclination angle of the plane to be reconstructed relative to the horizontal plane is equal to the inclination angle of the plane to be reconstructed relative to the vertical plane, any one of the axis perpendicular to the horizontal plane and the axis perpendicular to the vertical plane is determined as the axis corresponding to the plane to be reconstructed.

[0076] Furthermore, according to a preset reduction ratio, the coordinates of the two selected 3D points on two coordinate axes perpendicular to the determined axis are reduced respectively to obtain updated second coordinates of the two 3D points, and the distance between the two 3D points on the axis is determined based on the updated second coordinates.

[0077] Take the plane to be reconstructed as the inclined surface determined by the diagonal of a cube as an example, Figure 7As shown, the inclined plane XYZ has the same inclination angle relative to the horizontal plane XOZ and the vertical plane XOZ, and the Y axis or the Z axis is used as the axis corresponding to the inclined plane XYZ. Assuming that the Y axis is selected as the axis corresponding to the inclined plane XYZ, the height difference between the two 3D points and the inclined plane is used to determine whether the two selected 3D points are the 3D points corresponding to the inclined plane, that is, the 3D points corresponding to the inclined plane are selected from the 3D points according to the distance between the two selected 3D points on the Y axis. In the specific implementation, two 3D points whose distance is less than the set distance threshold are randomly selected from the extracted multiple 3D points, and are respectively recorded as E and F, wherein the coordinates of point E are represented by (x5, y5, z5), and the coordinates of point F are represented by (x6, y6, z6). The coordinate values ​​of the two 3D points E and F on the X-axis and Z-axis are reduced by 10,000 times, respectively, and the first coordinate of the updated point C is (x5 / 1000, y5, z5 / 1000), and the first coordinate of point D is (x6 / 1000, y6, z6 / 1000). According to the second coordinates of the two 3D points E and F, the distance dw between the two 3D points E and F on the Y-axis is determined. Compare dw with the preset height difference aw. If dw<aw, it indicates that the two 3D points E and F are two points on the inclined surface. If dw≥aw, it indicates that the two 3D points C and D do not belong to the inclined surface.

[0078] It should be noted that the sizes of ay, az, and aw can be set according to experimental data, and their sizes can be the same or different.

[0079] S303: Traverse all 3D points in the plurality of 3D points, and generate a 3D point set of the plane to be reconstructed according to the determined distances.

[0080] In this step, after traversing the multiple 3D points extracted in S302, multiple distances are determined. For each of the distances, the following operations are performed: the distance is compared with a preset height difference value, and if the distance is less than the preset height difference value, it indicates that the two 3D points corresponding to the distance are points on the plane to be reconstructed, and therefore, the two 2D points corresponding to the distance are taken as two points in the 3D point set of the plane to be reconstructed.

[0081] For example, assuming that the plane to be reconstructed is a horizontal plane, and the distance between the two 3D points A and B on the Y axis is less than ay, then the two 3D points A and B are retained, and the distance between the two 3D points C and D on the Y axis is greater than ay, then the two 3D points C and D are eliminated, until all 3D points are traversed to obtain the 3D point set corresponding to the horizontal plane.

[0082] It should be noted that, when traversing multiple 3D points, if the number of the multiple 3D points is an odd number, for a remaining 3D point, the plane to which the remaining 3D point belongs can be determined in combination with the 3D point closest to it.

[0083] S304: Reconstructing a plane to be reconstructed for displaying an enhanced image according to the 3D point set.

[0084] In this step, the plane to be reconstructed is reconstructed according to the 3D point set corresponding to the plane to be reconstructed, without reconstructing other planes, thereby improving the efficiency and accuracy of plane reconstruction. Among them, a plane to be reconstructed may contain multiple planes of different shapes. When executing S304, clustering is performed according to the distance between 3D points in the 3D point set to obtain a plane to be reconstructed of at least one shape category.

[0085] Take the horizontal plane as an example, Figure 8 As shown, after clustering according to the distances between the 3D points in the 3D point set corresponding to the horizontal plane, four horizontal planes with different shapes are obtained, which are respectively recorded as plane 1, plane 2, plane 3, and plane 4.

[0086] After the plane to be reconstructed is reconstructed, the enhanced video image can be displayed on the reconstructed plane, thereby realizing functions such as zooming in, zooming out, enhancing the effect, and having fun interactions on the displayed video image.

[0087] The embodiments of the present application do not impose restrictive requirements on clustering algorithms, including but not limited to the mean clustering (K-means) algorithm, the mean shift clustering algorithm, the density-based clustering algorithm (DBSCAN), and the Gaussian mixture model (GMM) clustering algorithm.

[0088] In the above-mentioned embodiments of the present application, by determining the distance between any two 3D points on the axis perpendicular to the plane to be reconstructed, that is, the height difference between any two 3D points and the plane to be reconstructed, a set of 3D points corresponding to the plane to be reconstructed is screened out from multiple 3D points, and the plane to be reconstructed is clustered according to the distance between the 3D points in the 3D point set to obtain at least one shape of the plane to be reconstructed, thereby displaying AR special effects. This method can quickly reconstruct the required horizontal plane, vertical plane, inclined plane, etc., without the need to reconstruct all planes based on all 3D points, and then screen the required plane from all planes. It is highly efficient, and because the 3D point set used for reconstruction is the 3D point corresponding to the plane to be reconstructed, the accuracy of the reconstructed plane is higher. In environments such as AR / VR, the reconstructed plane can be used to display enhanced video images, which brings a strong and shocking visual impact to users, enhances the user's experience and sense of reality, and generates body responses corresponding to special effects to enhance the experience.

[0089] Based on the same technical concept, the embodiment of the present application provides a display device, which can realize Figure 3 The plane reconstruction method shown can achieve the same technical effect.

[0090] See also Fig. 9, the display device includes an acquisition module 901, a distance determination module 902, a screening module 903, and a reconstruction module 904;

[0091] An acquisition module 901 is used to acquire a collected depth image and extract three-dimensional point cloud data based on the depth image, where the three-dimensional point cloud data includes multiple 3D points;

[0092] A distance determination module 902 is used to select two 3D points from a plurality of 3D points and determine the distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, wherein the distance between the two selected 3D points is less than a set distance threshold;

[0093] A screening module 903 is used to traverse all 3D points in the plurality of 3D points and generate a set of 3D points of a plane to be reconstructed according to each determined distance;

[0094] The reconstruction module 904 is used to reconstruct the plane to be reconstructed for displaying the enhanced image according to the 3D point set.

[0095] Optionally, the plane to be reconstructed includes a horizontal plane or a vertical plane, and the distance determination module 902 is specifically used to:

[0096] The coordinates of the two selected 3D points on the first axis parallel to the plane to be reconstructed are respectively set to preset values, and the coordinates of the two 3D points on the second axis parallel to the plane to be reconstructed are respectively reduced according to a preset reduction ratio to obtain updated first coordinates of the two 3D points;

[0097] According to the updated first coordinates, the distance between the two 3D points on an axis perpendicular to the plane to be reconstructed is determined.

[0098] Optionally, the plane to be reconstructed includes an inclined plane, and the distance determination module 902 is specifically used to:

[0099] Determine the axis corresponding to the plane to be reconstructed according to the inclination angle of the plane to be reconstructed relative to the horizontal plane and the vertical plane;

[0100] According to a preset reduction ratio, the coordinates of the two selected 3D points on the two coordinate axes perpendicular to the axis are reduced respectively to obtain updated second coordinates of the two 3D points;

[0101] According to the updated second coordinate, the distance between the two 3D points on the axis is determined.

[0102] Optionally, the distance determination module 902 is specifically configured to:

[0103] If the inclination angle of the to-be-reconstructed plane relative to the horizontal plane is greater than the inclination angle of the to-be-reconstructed plane relative to the vertical plane, then determining the axis corresponding to the to-be-reconstructed plane to be the axis perpendicular to the vertical plane; or

[0104] If the inclination angle of the to-be-reconstructed plane relative to the horizontal plane is smaller than the inclination angle of the to-be-reconstructed plane relative to the vertical plane, then determining the axis corresponding to the to-be-reconstructed plane to be the axis perpendicular to the horizontal plane; or

[0105] If the inclination angle of the to-be-reconstructed plane relative to the horizontal plane is equal to the inclination angle of the to-be-reconstructed plane relative to the vertical plane, any one of the axis perpendicular to the horizontal plane and the axis perpendicular to the vertical plane is determined as the axis corresponding to the to-be-reconstructed plane.

[0106] Optionally, the screening module 903 is specifically used for:

[0107] For each of the distances, if the distance is less than a preset height difference value, two 2D points corresponding to the distance less than the preset height difference value are used as two points in the 3D point set of the plane to be reconstructed.

[0108] Optionally, the reconstruction module 904 is specifically used for:

[0109] Clustering is performed according to the distances between the 3D points in the 3D point set to obtain a plane to be reconstructed of at least one shape category.

[0110] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0111] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0113] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A planar reconstruction method, characterized in that: include: Acquire the acquired depth image, and extract three-dimensional point cloud data based on the depth image, wherein the three-dimensional point cloud data includes a plurality of 3D points; Traversing the multiple 3D points, each time selecting two 3D points from the multiple 3D points, where a distance between the two selected 3D points is less than a set distance threshold; Determining the distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, specifically comprising: if the plane to be reconstructed is a horizontal plane or a vertical plane, setting the coordinates of the two selected 3D points on a first axis parallel to the plane to be reconstructed to preset values, and reducing the coordinates of the two 3D points on a second axis parallel to the plane to be reconstructed according to a preset reduction ratio, to obtain updated first coordinates of the two 3D points; determining the distance between the two 3D points on the axis perpendicular to the plane to be reconstructed according to the updated first coordinates; Generating a 3D point set of the plane to be reconstructed according to the determined distances; A plane to be reconstructed for displaying an enhanced image is reconstructed according to the 3D point set.

2. The method according to claim 1, characterized in that If the plane to be reconstructed is an inclined plane, determining the distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed includes: Determining the axis corresponding to the plane to be reconstructed according to the inclination angle of the plane to be reconstructed relative to the horizontal plane and the vertical plane; According to a preset reduction ratio, the coordinates of the two selected 3D points on two coordinate axes perpendicular to the axis are reduced respectively to obtain updated second coordinates of the two 3D points; According to the updated second coordinates, the distance between the two 3D points on the axis is determined.

3. The method according to claim 2, characterized in that The step of determining the axis corresponding to the plane to be reconstructed according to the inclination angle of the plane to be reconstructed relative to the horizontal plane and the vertical plane comprises: If the inclination angle of the to-be-reconstructed plane relative to the horizontal plane is greater than the inclination angle of the to-be-reconstructed plane relative to the vertical plane, then determining the axis corresponding to the to-be-reconstructed plane to be the axis perpendicular to the vertical plane; or If the inclination angle of the to-be-reconstructed plane relative to the horizontal plane is smaller than the inclination angle of the to-be-reconstructed plane relative to the vertical plane, then determining the axis corresponding to the to-be-reconstructed plane to be the axis perpendicular to the horizontal plane; or If the inclination angle of the plane to be reconstructed relative to the horizontal plane is equal to the inclination angle of the plane to be reconstructed relative to the vertical plane, any one of the axis perpendicular to the horizontal plane and the axis perpendicular to the vertical plane is determined as the axis corresponding to the plane to be reconstructed.

4. The method according to any one of claims 1 to 3, characterized in that The traversing all 3D points in the plurality of 3D points and generating the 3D point set of the to-be-reconstructed plane according to the determined distances includes: For each of the distances, if the distance is smaller than a preset height difference value, two 2D points corresponding to the distances smaller than the preset height difference value are used as two points in the 3D point set of the plane to be reconstructed.

5. The method according to any one of claims 1 to 3, characterized in that The reconstructing the plane to be reconstructed according to the 3D point set comprises: Clustering is performed according to the distances between the 3D points in the 3D point set to obtain a plane to be reconstructed of at least one shape category.

6. A display device, characterized in that: Including display, memory, controller: The display, connected to the controller, is configured to display the enhanced image on the reconstructed plane; The memory is connected to the controller and configured to store computer program instructions; The controller is configured to perform the following operations according to the computer program instructions: Acquire the acquired depth image, and extract three-dimensional point cloud data based on the depth image, wherein the three-dimensional point cloud data includes a plurality of 3D points; Traversing the multiple 3D points, each time selecting two 3D points from the multiple 3D points, where a distance between the two selected 3D points is less than a set distance threshold; Determining the distance between the two selected 3D points on an axis perpendicular to the plane to be reconstructed, specifically comprising: if the plane to be reconstructed is a horizontal plane or a vertical plane, setting the coordinates of the two selected 3D points on a first axis parallel to the plane to be reconstructed to preset values, and reducing the coordinates of the two 3D points on a second axis parallel to the plane to be reconstructed according to a preset reduction ratio, to obtain updated first coordinates of the two 3D points; determining the distance between the two 3D points on the axis perpendicular to the plane to be reconstructed according to the updated first coordinates; Generating a 3D point set of the plane to be reconstructed according to the determined distances; A plane to be reconstructed for displaying an enhanced image is reconstructed according to the 3D point set.

7. The display device according to claim 6, characterized in that If the plane to be reconstructed is an inclined plane, the controller is specifically configured as follows: Determining the axis corresponding to the plane to be reconstructed according to the inclination angle of the plane to be reconstructed relative to the horizontal plane and the vertical plane; According to a preset reduction ratio, the coordinates of the two selected 3D points on two coordinate axes perpendicular to the axis are reduced respectively to obtain updated second coordinates of the two 3D points; According to the updated second coordinates, the distance between the two 3D points on the axis is determined.

8. The display device according to any one of claims 6 to 7, characterized in that: The controller reconstructs the plane to be reconstructed according to the 3D point set, and is specifically configured as follows: Clustering is performed according to the distances between the 3D points in the 3D point set to obtain a plane to be reconstructed of at least one shape category.

Images