Flat panel display method, flat panel display device, electronic equipment and medium

Abstract

The invention discloses a flat panel display method, a flat panel display device, electronic equipment and a medium, and belongs to the technical field of computers. The flat panel display method comprises the following steps: acquiring a target depth image and target pose information associated with the target depth image, wherein the target pose information represents the pose of an image acquisition component when the target depth image is acquired; the target depth image is uniformly divided into N depth image blocks, N normal vectors are determined based on pixel information of center pixel points of the N depth image blocks, each depth image block corresponds to one normal vector, and N is a positive integer; based on the N normal vectors and the target pose information, at least one piece of first plane information is determined, and each piece of first plane information represents a first plane where the at least one depth image block is located in the three-dimensional space; and projecting the first plane onto a two-dimensional plane based on the pose information to obtain a projection plane, and displaying the projection plane.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a flat display method, a flat display device, an electronic device and a storage medium. Background Art

[0002] At present, augmented reality (AR) applications based on mobile devices are becoming more and more common, and are favored by major mobile device manufacturers and users. When integrating virtual objects into the real environment, it is necessary to determine the anchor point of the virtual object based on the plane, so as to render the virtual object at the determined anchor point. Therefore, plane detection is an important function in the calculation process of augmented reality applications based on mobile devices.

[0003] In the related technology, multiple images can be processed based on the SLAM system to obtain point cloud data, and then plane detection can be performed through the point cloud data to obtain the plane in space. Since the point cloud is obtained by extracting feature points from multiple images, the point cloud contains many feature points. Therefore, this method has high computational complexity, which makes the plane detection efficiency low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a flat display method, a flat display device, an electronic device and a storage medium, which can improve the efficiency of flat surface detection.

[0005] In a first aspect, an embodiment of the present application provides a planar display method, which is applied to an image acquisition component, and the method includes:

[0006] Acquire a target depth image and target pose information associated with the target depth image, wherein the target pose information represents a pose of an image acquisition component when acquiring the target depth image;

[0007] The target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer;

[0008] Based on the N normal vectors and the target pose information, determine at least one first plane information, each first plane information represents a first plane where at least one depth image block is located in the three-dimensional space;

[0009] Based on the posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed.

[0010] In a second aspect, an embodiment of the present application provides a flat display device, which is applied to an image acquisition component. The device includes: an acquisition module, a processing module, and a display module; wherein:

[0011] An acquisition module, used to acquire a target depth image and target position information associated with the target depth image, wherein the target posture information represents the posture of the image acquisition component when acquiring the target depth image;

[0012] A processing module, used to evenly divide the target depth image acquired by the acquisition module into N depth image blocks, and determine N normal vectors based on the pixel information of the central pixel points of the N depth image blocks, each depth image block corresponds to a normal vector, and N is a positive integer;

[0013] The processing module is further used to determine at least one first plane information based on the N normal vectors and the target pose information, each first plane information represents a first plane where at least one depth image block is located in the three-dimensional space;

[0014] The display module is used to project the first plane onto a two-dimensional plane based on the target posture information to obtain a projection plane and display the projection plane.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the planar display method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the planar display method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the planar display method as described in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the planar display method as described in the first aspect.

[0019] In an embodiment of the present application, a target depth image and target posture information associated with the target depth image are obtained, and the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, at least one first plane information is determined, and each first plane information represents the first plane where at least one depth image block is located in three-dimensional space; based on the posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed. In this way, in the present application, only the pixel information of the central pixel points of the N depth image blocks is needed to determine the N normal vectors, and then the N normal vectors and the target posture information are used to determine at least one first plane information, thereby greatly reducing the amount of data calculation in the plane detection process, thereby improving the efficiency of plane detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 1 ;

[0021] Figure 2 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 2 ;

[0022] Figure 3 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 3 ;

[0023] Figure 4 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 4 ;

[0024] Figure 5 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 5 ;

[0025] Figure 6 The process diagram of the flat display method provided by the embodiment of the present application is as follows Figure 6 ;

[0026] Figure 7 is a schematic diagram of a depth image block provided in an embodiment of the present application;

[0027] Figure 8 is a schematic structural diagram of a flat-panel display device provided in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0033] The flat panel display method, flat panel display device, electronic device and storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0034] The flat display method, flat display device, electronic device and storage medium provided in the embodiments of the present application can be applied to AR scenarios.

[0035] In AR scenarios, it is usually necessary to integrate virtual objects into the real environment. When integrating virtual objects into the real environment, it is often necessary to determine the anchor point of the virtual object based on the plane, so as to render the virtual object at the determined anchor point. Therefore, plane detection is an important function in the calculation process of augmented reality applications based on mobile devices.

[0036] In the related technology, multiple images can be processed based on the SLAM system to obtain point cloud data in three-dimensional space, and then plane detection can be performed through the point cloud data to obtain the plane in space. Since the point cloud is obtained by extracting feature points from multiple images, the point cloud contains a large number of feature points. Therefore, when the related technology is used for plane detection, the amount of data calculation is large, which makes the calculation complexity of plane detection high, and then makes the plane detection efficiency low.

[0037] To this end, the embodiment of the present application provides a plane display method, a plane display device, an electronic device and a storage medium, obtains a target depth image and target posture information associated with the target depth image, the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, at least one first plane information is determined, each first plane information represents the first plane where at least one depth image block is located in three-dimensional space; based on the posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed. In this way, in the present application, only the pixel information of the central pixel points of the N depth image blocks is needed to determine the N normal vectors, and then the N normal vectors and the target posture information are used to determine at least one first plane information, thereby greatly reducing the amount of data calculation in the plane detection process, thereby improving the efficiency of plane detection.

[0038] The execution subject of the planar display method provided in the embodiment of the present application may be a planar display device. For example, the planar display device may be an image acquisition component in an electronic device, or may be a functional component or functional entity in the electronic device. The planar display method provided in the embodiment of the present application will be described exemplarily below by taking the execution subject as an electronic device as an example.

[0039] Figure 1 is a flow chart of a planar display method provided in an embodiment of the present application, such as Figure 1 As shown, the flat display method provided in the embodiment of the present application may include the following steps 101 to 104.

[0040] Step 101: The electronic device obtains a target depth image and target pose information associated with the target depth image.

[0041] In some embodiments of the present application, the target posture information represents the posture of an image acquisition component on the electronic device when acquiring the target depth image.

[0042] In some embodiments of the present application, in some embodiments of the present application, the image acquisition component may be a camera, and the electronic device may acquire the target depth image through the camera.

[0043] In some embodiments of the present application, a depth image is also referred to as a distance image, which refers to an image in which the distance from an image collector to each point in a scene is used as a pixel value.

[0044] In some embodiments of the present application, the camera may be a time-of-flight (TOF) camera. Among them, TOF technology is a high-precision, high-frame-rate technology in photoelectric distance measurement, that is, using laser pulses generated by a laser to perform photoelectric distance measurement and calculate the distance between all objects in the scene and the TOF-based camera. On this basis, by integrating these distance information, a relatively accurate depth image can be obtained.

[0045] In some embodiments of the present application, the above-mentioned posture includes the rotation matrix of the camera and the position of the camera when acquiring the above-mentioned target depth image, and the position of the camera is the coordinate position of the camera in the three-dimensional space.

[0046] In some embodiments of the present application, the target posture information includes matrix information R of the rotation matrix of the camera when acquiring the target depth image and camera position information t, where the camera position information t is the coordinate information of the camera in three-dimensional space.

[0047] In some embodiments of the present application, the electronic device may also obtain an intrinsic parameter K of the camera when acquiring the above-mentioned target depth image.

[0048] In some embodiments of the present application, after the electronic device acquires the target depth image, it may perform denoising processing on the target depth image to remove noise from the target depth image and improve the image quality of the target depth image.

[0049] Step 102: The electronic device evenly divides the target depth image into N depth image blocks, and determines N normal vectors based on pixel information of central pixels of the N depth image blocks.

[0050] Among them, each depth image block corresponds to a normal vector, and N is a positive integer.

[0051] In some embodiments of the present application, the normal vector corresponding to each of the depth image blocks may be understood as the normal vector of the plane where each depth image block is located in the three-dimensional space.

[0052] In some embodiments of the present application, the above pixel information includes the depth value and coordinate information of the central pixel point; Figure 1 ,like Figure 2As shown, the “determining N normal vectors based on the pixel information of the central pixel points of the N depth image blocks” in the above step 102 can be implemented by the following steps 102a to 102c:

[0053] Step 102a: The electronic device obtains a depth value of a target central pixel of a target depth image block and first coordinate information of the target central pixel in the target depth image.

[0054] The target depth image block is any one of the N depth image blocks.

[0055] In some embodiments of the present application, the depth value is the pixel value of the target center pixel, that is, the distance between the point corresponding to the target center pixel in three-dimensional space and the camera.

[0056] In some embodiments of the present application, the electronic device may establish an image coordinate system on the target depth image, and then obtain first coordinate information of a target center pixel in the target depth image.

[0057] For example, the electronic device can use the pixel point at the lower left corner of the target depth image as the coordinate origin of the image coordinate system, the straight line where the bottom row of pixels of the target depth image is located as the horizontal axis of the image coordinate system, and the straight line where the leftmost column of pixels of the target depth image is located as the vertical axis of the image coordinate system to determine the image coordinate system.

[0058] Step 102b: The electronic device determines second coordinate information of the target center pixel point in the three-dimensional space based on the depth value of the target center pixel point, the first coordinate information, and the target posture information.

[0059] In some embodiments of the present application, the electronic device projects each pixel into the three-dimensional space using the depth value of the target center pixel, the camera intrinsic parameter K, the camera rotation matrix R, and the camera position information to form a camera rotation translation matrix [Rt] to obtain the second coordinate information Xw of each pixel in the three-dimensional space. Exemplarily, the electronic device can use the following formula to determine the second coordinate information of the pixel in the three-dimensional space:

[0060] Xw=K_inv*[Rt]*(j,i,1)*deepth; (1);

[0061] Wherein, Xw represents the second coordinate information of the target center pixel in three-dimensional space, deepth represents the depth value of the target center pixel, and (j, i) represents the first coordinate information of the target center pixel. For example, j represents that the target center pixel is located in the jth column of the target depth image, and i represents that the target center pixel is located in the ith row of the target depth image.

[0062] Step 102c: The electronic device determines a normal vector corresponding to the target depth image block based on the second coordinate information and the first coordinate information.

[0063] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 102c can be implemented by the following step 102c1 and the following step 102c2:

[0064] Step 102c1, the electronic device calculates a first gradient based on the first coordinate information, the second coordinate information, and the offset of other pixel points relative to the target center pixel point in the first direction, and calculates a second gradient based on the first coordinate information, the second coordinate information, and the offset of other pixel points relative to the target center pixel point in the second direction.

[0065] In some embodiments of the present application, the other pixel points are pixel points other than the target center pixel point in the target depth image block.

[0066] In some embodiments of the present application, the first gradient represents the rate of change of coordinates of the other pixel points relative to the target center pixel point in the first direction; the second gradient represents the rate of change of coordinates of the other pixel points relative to the target center pixel point in the second direction.

[0067] In some embodiments of the present application, the first direction may be the X-axis (i.e., horizontal axis) direction of the image coordinate system, and the second direction may be the Y-axis (i.e., vertical axis) direction of the image coordinate system. The electronic device may determine at least one group of symmetrical pixels from all other pixels except the target center pixel in the target depth image block, wherein a group of symmetrical pixels includes two pixels, and the two pixels of a group of symmetrical pixels are on a straight line with the target center pixel, and the distances between the two pixels and the target center pixel are the same. Then, the electronic device may calculate the first gradient of each group of symmetrical pixels in the X-axis direction based on the first coordinate information, the second coordinate information, and the offset of each group of symmetrical pixels in the target depth image block relative to the target center pixel in the X-axis direction, and calculate the first gradient of each group of symmetrical pixels in the Y-axis direction based on the first coordinate information, the second coordinate information, and the offset of each group of symmetrical pixels in the target depth image block relative to the first center pixel in the Y-axis direction.

[0068] In some embodiments of the present application, the electronic device may use the following formula to calculate the first gradient of a group of symmetrical pixels in the Y-axis direction:

[0069]

[0070] Among them, g_ X Represents the first gradient of a set of symmetrical pixels in the X-axis direction, Xt It represents the gradient of the pixel on the left side of the target center pixel in a set of symmetrical pixels in the X-axis direction. b It represents the gradient of the pixel on the right side of the target center pixel in a set of symmetrical pixels in the X-axis direction, a represents the offset of a set of symmetrical pixels relative to the target center pixel in the X-axis direction, and width represents the width of the target depth image.

[0071] For example, when a group of symmetrical pixels are the left adjacent pixels and the right adjacent pixels of the center pixel, a is 1.

[0072] In some embodiments of the present application, the electronic device may use the following formula to calculate the second gradient of a group of symmetrical pixels in the Y-axis direction:

[0073]

[0074] Among them, g_ Y Represents the second gradient of a set of symmetrical pixels in the Y-axis direction, X l It represents the gradient of the pixel below the target center pixel in a set of symmetrical pixels in the Y-axis direction, r It represents the gradient of the pixel above the target center pixel in a group of symmetrical pixels in the Y-axis direction, and b represents the offset of a group of symmetrical pixels relative to the target center pixel in the Y-axis direction.

[0075] For example, when a group of symmetrical pixel points are upper adjacent pixel points and lower adjacent pixel points of a central pixel point, b is 1.

[0076] Step 102c2: The electronic device determines a normal vector corresponding to the target depth image block based on the first gradient and the second gradient.

[0077] In some embodiments of the present application, the electronic device may average all first gradients to obtain a first average gradient, and average all second gradients to obtain a second average gradient. Then, the first average gradient and the second average gradient are cross-multiplied, and the vector obtained by the cross-multiplication operation is normalized to obtain a normal vector corresponding to the target depth image block.

[0078] In this way, the electronic device can calculate the first gradient only through the first coordinate information, the second coordinate information and the offset of other pixels relative to the target center pixel in the first direction; the second gradient can be calculated only through the first coordinate information, the second coordinate information and the offset of other pixels relative to the first center pixel in the second direction of the target center pixel, thereby reducing the amount of data calculation in the gradient calculation process; the electronic device can then accurately and quickly determine the normal vector corresponding to the target depth image block based on the first gradient and the second gradient, thereby improving the detection efficiency of the normal vector, and then improving the plane detection efficiency.

[0079] In this way, the electronic device determines the second coordinate information of the target center pixel in the three-dimensional space based on the depth value of the target center pixel, the first coordinate information and the target posture information; and determines the normal vector corresponding to the target depth image block based on the second coordinate information and the first coordinate information, so that when the electronic device determines the normal vector, it does not need the pixel information of too many pixels to participate in the calculation, thereby reducing the amount of data calculation in the normal vector calculation process, thereby improving the detection efficiency of the normal vector, and then improving the plane detection efficiency.

[0080] Step 103: The electronic device determines at least one first plane information based on N normal vectors and target pose information.

[0081] In some embodiments of the present application, each piece of first plane information represents a first plane where at least one depth image block is located in the three-dimensional space.

[0082] In some embodiments of the present application, the first plane information includes a plane equation and plane contour information; Figure 1 ,like Figure 4 As shown, the above step 103 can be implemented by the following steps 103a to 103e:

[0083] Step 103a: The electronic device clusters the N normal vectors based on the first angle value between every two normal vectors in the N normal vectors to obtain at least one cluster.

[0084] In some embodiments of the present application, each cluster includes at least one normal vector whose first angle value is less than or equal to a first threshold.

[0085] In some embodiments of the present application, the electronic device may perform K-means clustering on the N normal vectors based on the first angle value between every two normal vectors in the N normal vectors. K-means clustering is an unsupervised learning algorithm for dividing objects in a data set into K different groups or clusters, where each cluster is represented by the average value (i.e., centroid) of the objects within it. The algorithm iteratively redistributes data points to the nearest centroid and updates the centroid position until a certain termination condition is reached, such as the centroid no longer changes or the maximum number of iterations is reached.

[0086] In some embodiments of the present application, when the electronic device performs K-means clustering on N normal vectors based on the first angle value between every two normal vectors in the N normal vectors, K normal vectors may be randomly selected as the initial centroids. Then, the electronic device may calculate the first angle value between each normal vector and each centroid, and assign it to the cluster where the centroid with the smallest first angle value is located. Finally, the electronic device recalculates the average normal vector of each cluster to obtain a new centroid. The electronic device repeats the steps of calculating the first angle value between each normal vector and each centroid, and recalculating the average normal vector of each cluster until the centroid no longer changes or the maximum number of cycles is reached. Wherein, K is an integer greater than or equal to 1 and less than or equal to N.

[0087] Step 103b: The electronic device determines at least one depth image block corresponding to the target cluster based on at least one normal vector in the target cluster.

[0088] In some embodiments of the present application, the target cluster is any cluster among at least one cluster.

[0089] In some embodiments of the present application, the electronic device may use an image block corresponding to each normal vector in at least one normal vector in the target cluster as the image block corresponding to the target cluster to determine at least one depth image block corresponding to the target cluster.

[0090] Step 103c: determine the target plane equation based on the depth value of the central pixel point corresponding to the target cluster and at least one normal vector in the target cluster.

[0091] In some embodiments of the present application, the target plane equation is a plane equation of a first plane of the depth image block corresponding to the target cluster in the three-dimensional space.

[0092] In some embodiments of the present application, the central pixel point corresponding to the target cluster may be understood as: the central pixel point of all depth image blocks in at least one depth image block corresponding to the target cluster.

[0093] In some embodiments of the present application, the electronic device determines the target plane equation based on the average value of the depth value of the central pixel point corresponding to the above-mentioned target cluster and the first mean vector of at least one normal vector in the above-mentioned target cluster.

[0094] In some embodiments of the present application, the electronic device may first obtain the depth value of the central pixel point of each depth image block in at least one depth image block corresponding to the target cluster cluster, and then calculate the average depth value of the central pixel point of all depth image blocks in at least one depth image block corresponding to the target cluster cluster.

[0095] In some embodiments of the present application, the electronic device may calculate the average value of the depth values ​​of all depth image blocks in at least one depth image block corresponding to the target cluster using the following formula:

[0096] d=(deepth1+deepth2+...deepth M ) / M; (4)

[0097] Among them, d represents the average depth value of the central pixel corresponding to the target cluster, deepth1 represents the depth value of the central pixel of the first depth image block in the M depth image blocks corresponding to the target cluster, and deepth2 represents the depth value of the central pixel of the second depth image block in the M depth image blocks corresponding to the target cluster. M Represents the depth value of the central pixel of the Mth depth image block among the M depth image blocks corresponding to the target cluster, where M is an integer greater than or equal to 2.

[0098] In some embodiments of the present application, the electronic device may calculate a first mean vector of at least one normal vector in the target cluster using the following formula:

[0099] [a,b,c]=[(a1+a2+...+a M ) / M,(b1+b2+...+b M ) / M,(c1+c2+...+c M ) / M]; (5)

[0100] Among them, [a, b, c] represents the first mean vector, [a1, b1, c1] represents the first normal vector in the target cluster, [a2, b2, c2] represents the second normal vector in the first cluster...[a M ,b M ,c M ] represents the Mth normal vector in the target cluster.

[0101] In some embodiments of the present application, the target plane equation may be as follows:

[0102] ax+by+cz+d=0; (6)

[0103] Step 103d: The electronic device performs labeling and contour detection on at least one depth image block corresponding to the target cluster to obtain a target plane label image corresponding to the target cluster.

[0104] In some embodiments of the present application, the target plane label image includes pixel contour points of at least one depth image block corresponding to the target cluster.

[0105] In some embodiments of the present application, the electronic device marks at least one depth image block corresponding to the target cluster as the target color, and obtains a target plane label image corresponding to the target cluster, wherein the pixel value of each pixel in the target plane label image is the pixel value corresponding to the target color.

[0106] In some embodiments of the present application, the electronic device may also mark at least one depth image block corresponding to each other cluster as a color, and at least one depth image block corresponding to different clusters is marked with a different color, to obtain a plane label image corresponding to each other cluster. In this way, the target depth image can be marked as a label image, and the label image includes at least one plane label image.

[0107] In some embodiments of the present application, the electronic device may perform contour detection on the target plane label image on the label image to obtain pixel contour points of at least one depth image block corresponding to a target cluster.

[0108] In some embodiments of the present application, the electronic device may use a fixed-size pixel window, such as a 3*3 pixel window, to perform contour detection on the target plane label image on the label image. Exemplarily, if the pixel value of the central pixel point of the pixel window is the pixel value corresponding to the target color, and the pixel value of at least one other pixel point of the pixel window is different from the pixel value of the central pixel point, then the central pixel point of the pixel window is the above-mentioned pixel contour point.

[0109] In this way, the electronic device clusters N normal vectors based on the first angle value of every two normal vectors in the N normal vectors to obtain at least one cluster cluster; determines at least one depth image block corresponding to the target cluster cluster based on at least one normal vector in the target cluster cluster; determines the target plane equation based on the depth value of the central pixel point corresponding to at least one depth image block corresponding to the target cluster cluster and at least one normal vector in the target cluster cluster, so as to detect the plane corresponding to each cluster cluster, and then detect any plane in the space.

[0110] Step 103e: The electronic device determines first plane contour information based on the target plane label image and the target posture information.

[0111] In some embodiments of the present application, the first plane contour information is contour information of a first plane in a three-dimensional space of a depth image block corresponding to the target cluster.

[0112] In some embodiments of the present application, Figure 4 ,like Figure 5 As shown, the above step 103e can be implemented through the following steps 103e1 to 103e3:

[0113] Step 103e1: The electronic device simplifies the pixel contour points in the target plane label image to obtain simplified pixel contour points.

[0114] In some embodiments of the present application, the electronic device may use any existing simplification method, such as the Douglas Peucker method, to simplify the pixel contour points in the target plane label image, which is not specifically limited in this embodiment.

[0115] Step 103e2: The electronic device projects the simplified pixel contour points into a three-dimensional space based on the target posture information to obtain three-dimensional contour points of the depth image block corresponding to the target cluster.

[0116] It should be noted that the implementation process of the above step 103e2 is similar to the above step 102b. To avoid repetition, this embodiment will not be repeated here.

[0117] Step 103e3: The electronic device determines first plane contour information based on the coordinate information of the three-dimensional contour points in the three-dimensional space, the target posture information and the target plane equation.

[0118] In this way, the electronic device simplifies the pixel contour points in the target plane label image to obtain simplified pixel contour points, which can effectively reduce the number of pixel contour points, and then project the simplified pixel contour points into three-dimensional space based on the target posture information to obtain the three-dimensional contour points of the depth image block corresponding to the target cluster, which can greatly reduce the number of projected pixel contour points. Therefore, the first plane contour information can be determined based on the projected pixel contour points, which greatly reduces the calculation time and improves the detection efficiency of the first plane contour information.

[0119] In some embodiments of the present application, the above step 103e3 may be implemented by the following steps A1 and A2:

[0120] Step A1: The electronic device determines a plurality of first straight line equations based on coordinate information of three-dimensional contour points in three-dimensional space and target posture information.

[0121] In some embodiments of the present application, each first straight line equation corresponds to a contour point among the three-dimensional contour points.

[0122] In some embodiments of the present application, the electronic device determines a plurality of first straight line equations based on coordinate information of the three-dimensional contour points in the three-dimensional space and the camera position information t.

[0123] In some embodiments of the present application, a first straight line can be understood as a straight line where a three-dimensional contour point and the camera are located.

[0124] Step A2: The electronic device maps the plurality of first straight line equations to a target plane equation to obtain first plane contour information.

[0125] In some embodiments of the present application, the first plane contour information includes coordinate information of contour points of the depth image block corresponding to the target cluster in the first plane in the three-dimensional space.

[0126] In some embodiments of the present application, the plurality of first straight line equations may be vector equations. For example, the vector equation of a first straight line may be:

[0127]

[0128] In some embodiments of the present application, the electronic device can substitute the above vector equation into the target plane equation to calculate t, and then substitute it back into the vector equation to obtain the value of (x, y, z), where the value of (x, y, z) is the coordinate information of the contour point.

[0129] For example, when the plane equation is x+y+2z=5, substitute formula (6) into x+2y+2z=5 to obtain t=0.8, and then substitute t into formula (6) to determine the coordinate information of the contour point.

[0130] Step 104: The electronic device projects the first plane onto a two-dimensional plane based on the target posture information to obtain a projection plane, and displays the projection plane.

[0131] In some embodiments of the present application, the electronic device may project the contour points of the first plane onto a two-dimensional plane based on the above-mentioned posture information, and then use the plane composed of the contour points projected onto the two-dimensional plane as the projection plane.

[0132] In some embodiments of the present application, when displaying the above-mentioned projection planes, the electronic device may display the projection planes corresponding to different first planes in different colors.

[0133] In the plane display method provided in the present application, by obtaining the target depth image and the target posture information associated with the target depth image, the target posture information represents the posture of the image acquisition component when the target depth image is collected; the target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, at least one first plane information is determined, and each first plane information represents the first plane where at least one depth image block is located in three-dimensional space; based on the target posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed. In this way, in the present application, only the pixel information of the central pixel points of the N depth image blocks is needed to determine the N normal vectors, and then the N normal vectors and the target posture information are used to determine at least one first plane information, thereby greatly reducing the amount of data calculation in the plane detection process, thereby improving the efficiency of plane detection.

[0134] In some embodiments of the present application, after the above step 103, the flat display method provided by the present application further includes the following steps 105 to 108:

[0135] Step 105: The electronic device calculates a second angle value between the normal vector of the target plane and the normal vector of the second plane.

[0136] In some embodiments of the present application, the target plane is any first plane.

[0137] In some embodiments of the present application, the electronic device may calculate the second angle value between the normal vector of the target plane and the normal vector of the second plane by using the existing method of calculating the angle between two vectors, which is not described in detail in this embodiment.

[0138] Step 106: The electronic device calculates a first distance between the first center point and the second center point based on the coordinate information of the first center point of the target plane and the coordinate information of the second center point of the second plane.

[0139] In some embodiments of the present application, the second plane is any plane of at least one plane where the previous frame depth image of the target depth image is located in the three-dimensional space.

[0140] In some embodiments of the present application, after the electronic device obtains the coordinate information of the contour points of the target plane, it can calculate the coordinate information of the first center point based on the coordinate information of the contour points of the target plane.

[0141] In some embodiments of the present application, the electronic device may calculate the coordinate information of the first center point using the following formula:

[0142] (x0,y0,z0)=((x1+x2+...+x M ) / L,(y1+y2+...+y L ) / L,(z1+z2+...+z L ) / L); (7)

[0143] Among them, (x0, y0, z0) represents the coordinate information of the first center point, (x1, y1, z1) represents the coordinate information of the first contour point of the target plane, (x2, y2, z2) represents the coordinate information of the second contour point of the target plane...(x L ,y L ,z L ) represents the coordinate information of the Lth contour point of the target plane. L is an integer greater than or equal to 2.

[0144] Step 107: When the first distance is less than or equal to the second threshold and the second angle value is less than or equal to the third threshold, the electronic device determines the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point, and the coordinate information of the second center point.

[0145] In some embodiments of the present application, the above-mentioned fusion plane equation is a plane equation of a fusion plane corresponding to the above-mentioned target plane and the above-mentioned second plane.

[0146] In some embodiments of the present application, the step 107 of “determining the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point, and the coordinate information of the second center point” can be implemented by the following steps 107a and 107b:

[0147] Step 107a: Calculate the coordinate information of the third center point of the fusion plane based on the coordinate information of the first center point and the coordinate information of the second center point.

[0148] It should be noted that the implementation process of "calculating the coordinate information of the third center point of the fusion plane" in the above step 107a can refer to the implementation process of "calculating the coordinate information of the first center point" in the above step 106. To avoid repetition, this embodiment will not be repeated here.

[0149] Step 107b: The electronic device determines a fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, and the coordinate information of the third center point.

[0150] In some embodiments of the present application, the electronic device can calculate the normal vector of the target plane and the second mean vector of the normal vector of the second plane, and use the above second mean vector as the normal vector of the fusion plane. Then the electronic device can determine the point normal plane equation of the fusion plane in combination with the coordinate information of the third center point.

[0151] Step 108: The electronic device determines contour information of the fused plane based on the fused plane equation, the coordinate information of the contour points of the target plane, and the coordinate information of the contour points of the second plane.

[0152] In some embodiments of the present application, the above step 108 may be implemented by the following step 108a and the following step 108b:

[0153] Step 108a, the electronic device maps the contour points of the target plane to the fused plane equation based on the coordinate information of the contour points of the target plane and the target posture information to obtain second plane contour information, and maps the contour points of the second plane to the fused plane equation based on the coordinate information and posture information of the contour points of the second plane to obtain third plane contour information;

[0154] In some embodiments of the present application, the above step 108a of "based on the coordinate information and pose information of the contour points of the target plane, mapping the contour points of the target plane to the fusion plane equation to obtain the second plane contour information" can be implemented by the following steps 108a1 and 108a2, and the above step 108a of "based on the coordinate information and target pose information of the contour points of the second plane, mapping the contour points of the second plane to the fusion plane equation to obtain the third plane contour information" can be implemented by the following steps 108a3 and 108a4:

[0155] Step 108a1: The electronic device determines a plurality of second straight line equations based on the coordinate information of the contour points of the target plane and the target posture information.

[0156] In some embodiments of the present application, each second straight line equation corresponds to a contour point of the target plane.

[0157] It should be noted that the implementation process of the above step 108a1 is similar to the above step A1, and to avoid repetition, it will not be described again in this embodiment.

[0158] Step 108a2: The electronic device maps the multiple second straight line equations to the fused plane equation to obtain second plane contour information.

[0159] In some embodiments of the present application, the second plane contour information includes coordinate information of at least one first contour point of the fusion plane, and each first contour point corresponds to a contour point of the target plane.

[0160] It should be noted that the implementation process of the above step 108a2 is similar to the above step A2. To avoid repetition, this embodiment will not be repeated here.

[0161] Step 108a3: The electronic device determines a plurality of third straight line equations based on the coordinate information of the contour points of the second plane and the target posture information.

[0162] In some embodiments of the present application, each third straight line equation corresponds to a contour point of the second plane.

[0163] It should be noted that the implementation process of the above step 108a3 is similar to the above step A1, and to avoid repetition, it will not be described again in this embodiment.

[0164] Step 108a4: The electronic device maps the plurality of third straight line equations to a fused plane equation to obtain third plane contour information.

[0165] In some embodiments of the present application, the third plane contour information includes coordinate information of at least one second contour point of the fused plane, and each second contour point corresponds to a contour point of the second plane.

[0166] It should be noted that the implementation process of the above step 108a4 is similar to the above step A2, and to avoid repetition, this embodiment will not be repeated here.

[0167] Step 108b: The electronic device determines contour information of the fused plane based on the second plane contour information, the third plane contour information, and the normal vector of the fused plane.

[0168] In some embodiments of the present application, the above step 108b may be implemented by the following steps 108b1 to 108b4:

[0169] Step 108b1, the electronic device performs matrix conversion processing on the normal vector of the fusion plane to obtain a rotation matrix;

[0170] In some embodiments of the present application, the electronic device may use the following formula to perform matrix transformation processing on the normal vector of the fusion plane to obtain a rotation matrix:

[0171]

[0172] Where R represents the rotation matrix and [nx,ny,nz] represents the normal vector of the fusion plane.

[0173] Step 108b2: The electronic device rotates at least one first contour point onto a two-dimensional plane based on the rotation matrix and the coordinate information of at least one first contour point, and rotates at least one second contour point onto a two-dimensional plane based on the rotation matrix and the coordinate information of at least one second contour point.

[0174] In some embodiments of the present application, the electronic device may perform a product operation on the coordinate information of the first contour point and the rotation matrix to rotate at least one first contour point to a two-dimensional plane, so that at least one first contour point can be rotated into a two-dimensional contour point.

[0175] In some embodiments of the present application, the electronic device may multiply the coordinate information of the second contour point with the rotation matrix to rotate at least one second contour point to a two-dimensional plane, so that at least one second contour point can be rotated into a two-dimensional contour point.

[0176] Step 108b3: The electronic device performs convex hull detection on at least one first contour point in the two-dimensional plane and at least one second contour point in the two-dimensional plane to obtain a convex hull.

[0177] In some embodiments of the present application, the convex hull represents the maximum contour where the contour point is located on the two-dimensional plane.

[0178] In some embodiments of the present application, the electronic device may use any existing convex hull detection algorithm to perform convex hull detection on at least one first contour point in a two-dimensional plane and at least one second contour point in the two-dimensional plane, and this embodiment is not specifically limited here.

[0179] Step 108b4: The electronic device reversely rotates the contour points on the convex hull to the fusion plane based on the inverse matrix of the rotation matrix and the coordinate information of the contour points on the convex hull to obtain the contour information of the fusion plane.

[0180] In some embodiments of the present application, the contour information of the fusion plane includes coordinate information of contour points of the fusion plane.

[0181] In some embodiments of the present application, the electronic device may perform a product operation on the inverse matrix of the above rotation matrix and the coordinate information of the contour points on the convex hull to reversely rotate the contour points on the convex hull to the fusion plane to obtain the contour information of the fusion plane.

[0182] In this way, the electronic device obtains the convex hull by performing convex hull detection on at least one first contour point in the two-dimensional plane and at least one second contour point in the two-dimensional plane, and can accurately determine the maximum contour of the two-dimensional plane; based on the inverse matrix of the rotation matrix and the coordinate information of the contour points on the convex hull, the contour points on the convex hull are reversely rotated to the fusion plane to obtain the contour information of the fusion plane, thereby being able to accurately determine the contour information of the fusion plane.

[0183] In this way, the electronic device determines the fused plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point and the coordinate information of the second center point when the first distance is less than or equal to the second threshold and the second angle value is less than or equal to the third threshold. It determines the contour information of the fused plane based on the fused plane equation, the coordinate information of the contour points of the target plane and the coordinate information of the contour points of the second plane. It can continuously update the plane equation and the three-dimensional boundary of the plane in combination with the historical plane information, so that the final calculated spatial plane has higher accuracy and the spatial plane contour is more in line with the actual environment, thereby improving the effect of augmented display and spatial interaction of mixed reality.

[0184] Figure 6 6 is a flow chart of a flat display method provided in an embodiment of the present application. The method may include the following steps 601 to 610.

[0185] Step 601: The electronic device evenly divides the target depth image into blocks to obtain N depth image blocks.

[0186] For example, Figure 7 As shown, the electronic device can use X pixels in the row and column directions of the target depth image as a depth image block, where X is a positive integer. The electronic device can determine the size of X according to the resolution of the target depth image. For example, when the resolution of the target depth image is 300*200, X=20; when the resolution of the target depth image is 600*400, X=40.

[0187] Step 602: The electronic device detects the normal vector corresponding to each depth image block, and obtains the plane equation and plane label image of the first plane corresponding to each cluster cluster by performing k-means clustering on the normal vector.

[0188] Exemplarily, different plane label images have different colors.

[0189] Step 603: The electronic device detects the two-dimensional pixel contour points of each plane label image, and simplifies the two-dimensional pixel contour points using the Douglas Peucker method.

[0190] Step 604: The electronic device projects the simplified two-dimensional pixel contour points into three-dimensional space using the target position information to obtain three-dimensional contour points.

[0191] Exemplarily, the target position information represents the position and posture of an image acquisition component on an electronic device when acquiring a target depth image.

[0192] Step 605: The electronic device projects the three-dimensional contour points onto the corresponding first plane using the target posture information, and determines the contour point information of the first plane.

[0193] Step 606: The electronic device calculates the plane distance between the target plane and the second plane and the angle between the target plane and the second plane.

[0194] Exemplarily, the target plane is any first plane, and the second plane is a history plane, that is, any plane in at least one plane where a previous depth image of the first depth image is located in the three-dimensional space.

[0195] Exemplarily, the plane distance between the target plane and the second plane may be a distance between a center point of the target plane and a center point of the second plane.

[0196] Exemplarily, the angle between the target plane and the second plane may be the angle between a normal vector of the target plane and a normal vector of the second plane.

[0197] Step 607: When the plane distance is less than or equal to the first threshold and the angle is less than or equal to the second threshold, the electronic device fuses the plane equation of the target plane and the plane equation of the second plane to obtain a fused plane equation of the fused plane.

[0198] Exemplarily, the fusion plane may be a fusion plane corresponding to the target plane and the second plane.

[0199] Exemplarily, the fusion process may be to average the normal vectors and center points of the two planes to generate a new point normal equation.

[0200] Step 608: The electronic device projects the contour points of the two planes onto the fusion plane, and uses the normal vector of the fusion plane to rotate the contour points projected onto the fusion plane to obtain two-dimensional contour points.

[0201] Step 609: The electronic device obtains a convex hull by performing convex hull detection on the two-dimensional contour points.

[0202] Step 610: The electronic device uses the normal vector of the fusion plane to reversely rotate and obtain the contour point information of the corresponding fusion plane.

[0203] The plane display method provided in the embodiment of the present application can determine the contour point boundary of the plane by projecting very few two-dimensional pixel contour points, thereby greatly reducing the time consumption. In addition, the plane display method provided in the embodiment of the present application can use the depth image to quickly reconstruct high-quality global plane information. Compared with the previous plane reconstruction scheme, the reconstruction speed is faster and the plane quality is better. It can not only support the reconstruction of horizontal and vertical planes, but also can reconstruct any plane information in space.

[0204] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0205] The flat display method provided in the embodiment of the present application can be executed by a flat display device. In the embodiment of the present application, a flat display device executing the flat display method is taken as an example to illustrate the flat display device provided in the embodiment of the present application.

[0206] Figure 8 800 is a schematic diagram of the structure of a flat display device provided in an embodiment of the present application. The flat display device 800 includes: an acquisition module 801, a processing module 802 and a display module 803; wherein,

[0207] The acquisition module 801 is used to acquire a target depth image and target position information associated with the target depth image, and the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the processing module 802 is used to evenly divide the target depth image acquired by the acquisition module into N depth image blocks, and determine N normal vectors based on the pixel information of the central pixel points of the N depth image blocks, each depth image block corresponds to a normal vector, and N is a positive integer; the processing module 802 is also used to determine at least one first plane information based on the N normal vectors and the target posture information, and each first plane information represents a first plane where at least one depth image block is located in a three-dimensional space; the display module 803 is used to project the first plane onto a two-dimensional plane based on the target posture information to obtain a projection plane, and display the projection plane.

[0208] In some embodiments of the present application, the pixel information includes the depth value and coordinate information of the center pixel; the acquisition module 801 is specifically used to: obtain the depth value of the target center pixel of the target depth image block and the first coordinate information of the target center pixel in the target depth image; based on the depth value, the first coordinate information and the target posture information of the target center pixel, determine the second coordinate information of the target center pixel in the three-dimensional space; based on the second coordinate information and the first coordinate information, determine the normal vector corresponding to the target depth image block; wherein the target depth image block is any depth image block among the N depth image blocks.

[0209] In some embodiments of the present application, the first plane information includes a plane equation and plane contour information; the processing module 802 is specifically used to: cluster the N normal vectors based on the first angle value of each two normal vectors in the N normal vectors to obtain at least one cluster cluster; determine at least one depth image block corresponding to the target cluster cluster based on at least one normal vector in the target cluster cluster, and the target cluster cluster is any cluster cluster in the at least one cluster cluster; determine the target plane equation based on the depth value of the central pixel point corresponding to the target cluster cluster and at least one normal vector in the target cluster cluster, and the target plane equation is the plane equation of the first plane of the depth image block corresponding to the target cluster cluster in the three-dimensional space; mark and contour detect the at least one depth image block corresponding to the target cluster cluster to obtain a target plane label image corresponding to the target cluster cluster, and the target plane label image includes the pixel contour points of at least one depth image block corresponding to the target cluster cluster; determine the first plane contour information based on the target plane label image and the target posture information, and the first plane contour information is the contour information of the depth image block corresponding to the target cluster cluster in the first plane in the three-dimensional space.

[0210] In some embodiments of the present application, the processing module 802 is specifically used to: simplify the pixel contour points in the target plane label image to obtain simplified pixel contour points; based on the target posture information, project the simplified pixel contour points into the three-dimensional space to obtain the three-dimensional contour points of the depth image block corresponding to the target cluster cluster; based on the coordinate information of the three-dimensional contour points in the three-dimensional space, the target posture information and the target plane equation, determine the first plane contour information.

[0211] In some embodiments of the present application, the processing module 802 is specifically used to: determine multiple first straight line equations based on the coordinate information of the three-dimensional contour points in the three-dimensional space and the target posture information, each first straight line equation corresponds to a contour point in the three-dimensional contour points; map the multiple first straight line equations to the target plane equation to obtain first plane contour information, the first plane contour information including the coordinate information of the contour points of the depth image block corresponding to the target cluster in the first plane in the three-dimensional space.

[0212] In some embodiments of the present application, each first plane information includes a target plane equation of the first plane and coordinate information of the contour points of the first plane; the processing module 802 is also used to: after determining at least one first plane information based on N normal vectors and target posture information, calculate the second angle value between the normal vector of the target plane and the normal vector of the second plane, where the target plane is any first plane; calculate the first distance between the first center point and the second center point based on the coordinate information of the first center point of the target plane and the coordinate information of the second center point of the second plane, where the second plane is any plane in at least one plane in which the previous frame depth image of the target depth image is located in the three-dimensional space; when the first distance is less than or equal to the second threshold value and the second angle value is less than or equal to the third threshold value, determine the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point and the coordinate information of the second center point, where the fusion plane equation is the plane equation of the fusion plane corresponding to the target plane and the second plane; determine the contour information of the fusion plane based on the fusion plane equation, the coordinate information of the contour points of the target plane and the coordinate information of the contour points of the second plane.

[0213] In some embodiments of the present application, the processing module 802 is specifically used to: calculate the coordinate information of the third center point of the fusion plane based on the coordinate information of the first center point and the coordinate information of the second center point; determine the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane and the coordinate information of the third center point.

[0214] In some embodiments of the present application, the processing module 802 is specifically used to: map the contour points of the target plane to the fusion plane equation based on the coordinate information and posture information of the contour points of the target plane to obtain second plane contour information, and map the contour points of the second plane to the fusion plane equation based on the coordinate information and posture information of the contour points of the second plane to obtain third plane contour information; determine the contour information of the fusion plane based on the second plane contour information, the third plane contour information and the normal vector of the fusion plane; wherein the second plane contour information includes the coordinate information of at least one first contour point of the fusion plane, and each first contour point corresponds to a contour point of the target plane; the third plane contour information includes the coordinate information of at least one second contour point of the fusion plane, and each second contour point corresponds to a contour point of the second plane.

[0215] In some embodiments of the present application, the processing module 802 is specifically used to: determine multiple second straight line equations based on the coordinate information and posture information of the contour points of the target plane, each second straight line equation corresponds to a contour point of the target plane; map the multiple second straight line equations to the fused plane equation to obtain second plane contour information; determine multiple third straight line equations based on the coordinate information and posture information of the contour points of the second plane, each third straight line equation corresponds to a contour point of the second plane; map the multiple third straight line equations to the fused plane equation to obtain third plane contour information.

[0216] In some embodiments of the present application, the processing module 802 is specifically used to: perform matrix transformation processing on the normal vector of the fusion plane to obtain a rotation matrix; based on the rotation matrix and the coordinate information of at least one first contour point, rotate at least one first contour point to the two-dimensional plane, and based on the rotation matrix and the coordinate information of the at least one second contour point, rotate at least one second contour point to the two-dimensional plane; perform convex hull detection on at least one first contour point in the two-dimensional plane and at least one second contour point in the two-dimensional plane to obtain a convex hull, the convex hull represents the maximum contour of the contour point on the two-dimensional plane; based on the inverse matrix of the rotation matrix and the coordinate information of the contour points on the convex hull, inversely rotate the contour points on the convex hull to the fusion plane to obtain contour information of the fusion plane, and the contour information of the fusion plane includes the coordinate information of the contour points of the fusion plane.

[0217] In the plane display device provided by the embodiment of the present application, the plane display device acquires the target depth image and the target position information associated with the target depth image, and the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the target depth image acquired by the acquisition module is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, at least one first plane information is determined, and each first plane information represents the first plane where at least one depth image block is located in three-dimensional space; based on the target posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed. In this way, in the present application, only the pixel information of the central pixel points of the N depth image blocks is needed to determine the N normal vectors, and then the N normal vectors and the target posture information are used to determine at least one first plane information, thereby greatly reducing the amount of data calculation in the plane detection process, thereby improving the efficiency of plane detection.

[0218] The flat display device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0219] The flat display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0220] The flat display device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned flat display method, and will not be described again here to avoid repetition.

[0221] Alternatively, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores a program or instruction that can be executed on the processor 901, and when the program or instruction is executed by the processor 901, each step of the above-mentioned planar display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0222] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0223] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0224] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0225] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0226] Among them, the sensor 1005 is used to obtain a target depth image and target posture information associated with the target depth image, the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the processor 1010 is used to evenly divide the target depth image into N depth image blocks, and determine N normal vectors based on the pixel information of the central pixel points of the N depth image blocks, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, determine at least one first plane information, each first plane information represents the first plane where at least one depth image block is located in the three-dimensional space; the display unit 1006 is used to project the first plane onto a two-dimensional plane based on the posture information, obtain the projection plane, and display the projection plane.

[0227] In some embodiments of the present application, the pixel information includes the depth value and coordinate information of the center pixel point; the processor 1010 is specifically used to: obtain the depth value of the target center pixel point of the target depth image block and the first coordinate information of the target center pixel point in the target depth image; based on the depth value, the first coordinate information and the posture information of the target center pixel point, determine the second coordinate information of the target center pixel point in the three-dimensional space; based on the second coordinate information and the first coordinate information, determine the normal vector corresponding to the target depth image block; wherein the target depth image block is any depth image block among the N depth image blocks.

[0228] In some embodiments of the present application, the first plane information includes a plane equation and plane contour information; the processor 1010 is specifically used to: cluster the N normal vectors based on the first angle value of every two normal vectors in the N normal vectors to obtain at least one cluster cluster; determine at least one depth image block corresponding to the target cluster cluster based on at least one normal vector in the target cluster cluster, and the target cluster cluster is any cluster cluster in the at least one cluster cluster; determine the target plane equation based on the depth value of the central pixel point corresponding to the target cluster cluster and at least one normal vector in the target cluster cluster, and the target plane equation is the plane equation of the first plane of the depth image block corresponding to the target cluster cluster in the three-dimensional space; mark and contour detect the at least one depth image block corresponding to the target cluster cluster to obtain a target plane label image corresponding to the target cluster cluster, and the target plane label image includes the pixel contour points of at least one depth image block corresponding to the target cluster cluster; determine the first plane contour information based on the target plane label image and the target posture information, and the first plane contour information is the contour information of the depth image block corresponding to the target cluster cluster in the first plane in the three-dimensional space.

[0229] In some embodiments of the present application, the processor 1010 is specifically used to: simplify the pixel contour points in the target plane label image to obtain simplified pixel contour points; based on the target posture information, project the simplified pixel contour points into the three-dimensional space to obtain the three-dimensional contour points of the depth image block corresponding to the target cluster cluster; based on the coordinate information of the three-dimensional contour points in the three-dimensional space, the target posture information and the target plane equation, determine the first plane contour information.

[0230] In some embodiments of the present application, the processor 1010 is specifically used to: determine multiple first straight line equations based on coordinate information of three-dimensional contour points in three-dimensional space and target posture information, each first straight line equation corresponding to a contour point in the three-dimensional contour points; map the multiple first straight line equations to the target plane equation to obtain first plane contour information, the first plane contour information including coordinate information of contour points of the depth image block corresponding to the target cluster in the first plane in the three-dimensional space.

[0231] In some embodiments of the present application, each first plane information includes a target plane equation of the first plane and coordinate information of the contour points of the first plane; the processor 1010 is further used to: after determining at least one first plane information based on N normal vectors and target posture information, calculate the second angle value between the normal vector of the target plane and the normal vector of the second plane, where the target plane is any first plane; calculate the first distance between the first center point and the second center point based on the coordinate information of the first center point of the target plane and the coordinate information of the second center point of the second plane, where the second plane is any plane in at least one plane in which the previous frame depth image of the target depth image is located in the three-dimensional space; when the first distance is less than or equal to the second threshold value and the second angle value is less than or equal to the third threshold value, determine the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point and the coordinate information of the second center point, where the fusion plane equation is the plane equation of the fusion plane corresponding to the target plane and the second plane; determine the contour information of the fusion plane based on the fusion plane equation, the coordinate information of the contour points of the target plane and the coordinate information of the contour points of the second plane.

[0232] In some embodiments of the present application, the processor 1010 is specifically used to: calculate the coordinate information of the third center point of the fusion plane based on the coordinate information of the first center point and the coordinate information of the second center point; determine the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane and the coordinate information of the third center point.

[0233] In some embodiments of the present application, the processor 1010 is specifically used to: map the contour points of the target plane to the fusion plane equation based on the coordinate information and posture information of the contour points of the target plane to obtain second plane contour information, and map the contour points of the second plane to the fusion plane equation based on the coordinate information and posture information of the contour points of the second plane to obtain third plane contour information; determine the contour information of the fusion plane based on the second plane contour information, the third plane contour information and the normal vector of the fusion plane; wherein the second plane contour information includes the coordinate information of at least one first contour point of the fusion plane, and each first contour point corresponds to a contour point of the target plane; and the third plane contour information includes the coordinate information of at least one second contour point of the fusion plane, and each second contour point corresponds to a contour point of the second plane.

[0234] In some embodiments of the present application, the processor 1010 is specifically used to: determine multiple second straight line equations based on the coordinate information of the contour points of the target plane and the target posture information, each second straight line equation corresponds to a contour point of the target plane; map the multiple second straight line equations to the fused plane equation to obtain second plane contour information; determine multiple third straight line equations based on the coordinate information and posture information of the contour points of the second plane, each third straight line equation corresponds to a contour point of the second plane; map the multiple third straight line equations to the fused plane equation to obtain third plane contour information.

[0235] In some embodiments of the present application, the processor 1010 is specifically used to: perform matrix transformation processing on the normal vector of the fusion plane to obtain a rotation matrix; based on the rotation matrix and the coordinate information of the at least one first contour point, rotate at least one first contour point to the two-dimensional plane, and based on the rotation matrix and the coordinate information of the at least one second contour point, rotate at least one second contour point to the two-dimensional plane; perform convex hull detection on at least one first contour point in the two-dimensional plane and at least one second contour point in the two-dimensional plane to obtain a convex hull, the convex hull represents the maximum contour of the contour point on the two-dimensional plane; based on the inverse matrix of the rotation matrix and the coordinate information of the contour points on the convex hull, inversely rotate the contour points on the convex hull to the fusion plane to obtain contour information of the fusion plane, and the contour information of the fusion plane includes the coordinate information of the contour points of the fusion plane.

[0236] In the electronic device provided in the present application, by acquiring the target depth image and the target posture information associated with the target depth image, the target posture information represents the posture of the image acquisition component when acquiring the target depth image; the target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; based on the N normal vectors and the target posture information, at least one first plane information is determined, and each first plane information represents the first plane where at least one depth image block is located in three-dimensional space; based on the posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed. In this way, in the present application, only the pixel information of the central pixel points of the N depth image blocks is needed to determine the N normal vectors, and then the N normal vectors and the target posture information are used to determine at least one first plane information, thereby greatly reducing the amount of data calculation in the plane detection process, thereby improving the efficiency of plane detection.

[0237] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0238] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0239] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0240] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned planar display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0241] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0242] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned planar display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0243] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0244] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned planar display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0245] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0246] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A planar display method, characterized in that: Applied to an image acquisition component, the method comprises: Acquire a target depth image and target posture information associated with the target depth image, wherein the target posture information represents the posture of the image acquisition component when acquiring the target depth image; The target depth image is evenly divided into N depth image blocks, and based on the pixel information of the central pixel points of the N depth image blocks, N normal vectors are determined, each depth image block corresponds to a normal vector, and N is a positive integer; Based on the N normal vectors and the target pose information, determine at least one first plane information, each first plane information represents a first plane where at least one depth image block is located in the three-dimensional space; Based on the target posture information, the first plane is projected onto a two-dimensional plane to obtain a projection plane, and the projection plane is displayed.

2. The method according to claim 1, characterized in that: The pixel information includes the depth value and coordinate information of the central pixel; The determining N normal vectors based on the pixel information of the central pixel points of the N depth image blocks includes: Obtain a depth value of a target central pixel of a target depth image block and first coordinate information of the target central pixel in the target depth image; Determine second coordinate information of the target center pixel in three-dimensional space based on the depth value of the target center pixel, the first coordinate information, and the target pose information; Determine a normal vector corresponding to the target depth image block based on the second coordinate information and the first coordinate information; The target depth image block is any depth image block among the N depth image blocks.

3. The method according to claim 1, characterized in that: The first plane information includes a plane equation and plane contour information; The determining, based on the N normal vectors and the target pose information, at least one first plane information comprises: Based on a first angle value between every two normal vectors in the N normal vectors, clustering the N normal vectors to obtain at least one cluster cluster; Based on at least one normal vector in a target cluster, determining at least one depth image block corresponding to the target cluster, wherein the target cluster is any one of the at least one cluster; Determine a target plane equation based on a depth value of a central pixel corresponding to the target cluster and at least one normal vector in the target cluster, where the target plane equation is a plane equation of a first plane of a depth image block corresponding to the target cluster in a three-dimensional space; Marking and contour detection are performed on at least one depth image block corresponding to the target cluster to obtain a target plane label image corresponding to the target cluster, wherein the target plane label image includes pixel contour points of at least one depth image block corresponding to the target cluster; Based on the target plane label image and the target pose information, first plane contour information is determined, where the first plane contour information is contour information of a first plane of a depth image block corresponding to the target cluster in a three-dimensional space.

4. The method according to claim 3, characterized in that The determining of first plane contour information based on the target plane label image and the target pose information includes: Simplifying the pixel contour points in the target plane label image to obtain simplified pixel contour points; Based on the target pose information, the simplified pixel contour points are projected into a three-dimensional space to obtain three-dimensional contour points of the depth image block corresponding to the target cluster; The first plane contour information is determined based on the coordinate information of the three-dimensional contour points in the three-dimensional space, the target posture information and the target plane equation.

5. The method according to claim 4, characterized in that The determining the first plane contour information based on the coordinate information of the three-dimensional contour points in the three-dimensional space, the target posture information and the target plane equation includes: Based on the coordinate information of the three-dimensional contour points in the three-dimensional space and the target posture information, a plurality of first straight line equations are determined, each of the first straight line equations corresponding to a contour point among the three-dimensional contour points; The multiple first straight line equations are mapped to the target plane equation to obtain the first plane contour information, where the first plane contour information includes coordinate information of contour points of the depth image block corresponding to the target cluster in the first plane in the three-dimensional space.

6. The method according to any one of claims 1 to 5, characterized in that Each of the first plane information includes a target plane equation of the first plane and coordinate information of contour points of the first plane; After determining at least one first plane information based on the N normal vectors and the target pose information, the method further includes: Calculating a second angle value between a normal vector of a target plane and a normal vector of a second plane, wherein the target plane is any first plane; Calculate a first distance between the first center point and the second center point based on the coordinate information of the first center point of the target plane and the coordinate information of the second center point of the second plane, where the second plane is any plane of at least one plane where a previous frame of the depth image of the target depth image is located in the three-dimensional space; When the first distance is less than or equal to a second threshold value, and the second angle value is less than or equal to a third threshold value, determining a fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point, and the coordinate information of the second center point, the fusion plane equation being a plane equation of a fusion plane corresponding to the target plane and the second plane; Based on the fusion plane equation, the coordinate information of the contour points of the target plane and the coordinate information of the contour points of the second plane, the contour information of the fusion plane is determined.

7. The method according to claim 6, characterized in that The determining of the fusion plane equation based on the normal vector of the target plane, the normal vector of the second plane, the coordinate information of the first center point, and the coordinate information of the second center point includes: Calculating the coordinate information of the third center point of the fusion plane based on the coordinate information of the first center point and the coordinate information of the second center point; The fusion plane equation is determined based on the normal vector of the target plane, the normal vector of the second plane and the coordinate information of the third center point.

8. The method according to claim 6, characterized in that The determining the contour information of the fused plane based on the fused plane equation, the coordinate information of the contour points of the target plane and the coordinate information of the contour points of the second plane comprises: Based on the coordinate information of the contour points of the target plane and the target pose information, the contour points of the target plane are mapped to the fused plane equation to obtain second plane contour information, and based on the coordinate information of the contour points of the second plane and the target pose information, the contour points of the second plane are mapped to the fused plane equation to obtain third plane contour information; Determine contour information of the fused plane based on the second plane contour information, the third plane contour information and the normal vector of the fused plane; Among them, the second plane contour signal includes coordinate information of at least one first contour point of the fused plane, and each first contour point corresponds to a contour point of the target plane; the third plane contour signal includes coordinate information of at least one second contour point of the fused plane, and each second contour point corresponds to a contour point of the second plane.

9. The method according to claim 8, characterized in that The step of mapping the contour points of the target plane to the fusion plane equation based on the coordinate information of the contour points of the target plane and the target pose information to obtain the second plane contour information includes: Based on the coordinate information of the contour point of the target plane and the target pose information, a plurality of second straight line equations are determined, each second straight line equation corresponding to a contour point of the target plane; Mapping the plurality of second straight line equations to the fused plane equation to obtain the second plane contour information; The step of mapping the contour points of the second plane to the fusion plane equation based on the coordinate information of the contour points of the second plane and the target pose information to obtain the contour information of the third plane includes: Based on the coordinate information of the contour point of the second plane and the target pose information, determine a plurality of third straight line equations, each of which corresponds to a contour point of the second plane; The plurality of third straight line equations are mapped to the fused plane equation to obtain the third plane contour information.

10. The method according to claim 8, characterized in that The determining the contour information of the fused plane based on the second plane contour information, the third plane contour information and the normal vector of the fused plane includes: Performing matrix transformation processing on the normal vector of the fusion plane to obtain a rotation matrix; Based on the rotation matrix and the coordinate information of the at least one first contour point, the at least one first contour point is rotated onto a two-dimensional plane, and based on the rotation matrix and the coordinate information of the at least one second contour point, the at least one second contour point is rotated onto the two-dimensional plane; Performing convex hull detection on the at least one first contour point in the two-dimensional plane and the at least one second contour point in the two-dimensional plane to obtain a convex hull, wherein the convex hull represents a maximum contour where the contour point is located on the two-dimensional plane; Based on the inverse matrix of the rotation matrix and the coordinate information of the contour points on the convex hull, the contour points on the convex hull are inversely rotated to the fusion plane to obtain the contour information of the fusion plane, and the contour information of the fusion plane includes the coordinate information of the contour points of the fusion plane.

11. A flat display device, characterized in that: Applied to image acquisition components, the device comprises: an acquisition module, a processing module and a display module; wherein, The acquisition module is used to acquire a target depth image and target position information associated with the target depth image, wherein the target posture information represents the posture of the image acquisition component when acquiring the target depth image; The processing module is used to evenly divide the target depth image acquired by the acquisition module into N depth image blocks, and determine N normal vectors based on pixel information of central pixels of the N depth image blocks, each depth image block corresponds to a normal vector, and N is a positive integer; The processing module is further used to determine at least one first plane information based on the N normal vectors and the target pose information, each first plane information represents a first plane where at least one depth image block is located in the three-dimensional space; The display module is used to project the first plane onto a two-dimensional plane based on the target posture information to obtain a projection plane, and display the projection plane.

12. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the flat panel display method according to any one of claims 1 to 11 are implemented.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the flat panel display method according to any one of claims 1 to 11 are implemented.

Images