Plane detection method, device, medium and program product

Abstract

The purpose of this application is to provide a plane detection method, device, medium and program product, specifically comprising: obtaining point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information; dividing the current image frame into a central area and a non-central area, and dividing the point cloud information into central point set information and non-central point set information of the current image frame based on the central area and the non-central area; determining one or more corresponding sub-plane information based on multiple 3D point information in the central point set information; determining the central plane information corresponding to the central point set information based on the one or more sub-plane information, and determining the image plane information of the current image frame based on the central plane information and the non-central point set information. This application improves the calculation rate and accuracy of plane detection and reduces the consumption of computing resources.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a technology for plane detection. Background Art

[0002] In recent years, with the increasing application of autonomous driving, augmented reality, virtual reality, robotics, and other fields, the demand for analyzing and understanding three-dimensional data has become increasingly urgent. Point cloud information, as a crucial form of three-dimensional data representation, is the cornerstone of many technological research projects. The ability to recover the geometric structure of visible surfaces is crucial for scene understanding. Among them, the detection of two-dimensional planes in three-dimensional point clouds is an important prerequisite for virtual object interaction. While the analysis and research of dense point clouds is a long-standing tradition, there are few fast plane detection methods for sparse point clouds, which are more widely used on mobile devices. Summary of the Invention

[0003] One object of the present application is to provide a plane detection method, device, medium and program product.

[0004] According to one aspect of the present application, a plane detection method is provided, wherein the method includes:

[0005] Obtaining point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information;

[0006] Dividing the current image frame into a central area and a non-central area, and dividing the point cloud information into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame;

[0007] Determining one or more corresponding sub-plane information based on the multiple 3D point information in the central point set information, wherein the sub-plane information includes a plane formed by at least one 3D point information in the multiple 3D point information and adjacent point information of the 3D point information;

[0008] Central plane information corresponding to the central point set information is determined according to the one or more sub-plane information, and image plane information of the current image frame is determined according to the central plane information and the non-central point set information.

[0009] According to one aspect of the present application, a plane detection device is provided, wherein the device includes:

[0010] A module for obtaining point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of a plurality of 3D point information;

[0011] Module 12 is configured to divide the current image frame into a central area and a non-central area, and divide the point cloud information into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame;

[0012] a module 13, configured to determine one or more corresponding sub-plane information based on the multiple 3D point information in the central point set information, wherein the sub-plane information includes a plane formed by at least one 3D point information in the multiple 3D point information and adjacent point information of the 3D point information;

[0013] A fourth module is used to determine the central plane information corresponding to the central point set information based on the one or more sub-plane information, and to determine the image plane information of the current image frame based on the central plane information and the non-central point set information.

[0014] According to one aspect of the present application, a computer device is provided, wherein the device includes:

[0015] processor; and

[0016] A memory arranged to store computer executable instructions, which when executed cause the processor to perform the steps of any of the methods described above.

[0017] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed, the system performs the steps of any of the methods described above.

[0018] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the steps of any of the above methods when executed by a processor.

[0019] Compared to existing technologies, this application provides plane detection for sparse point clouds. Due to the large distances between 3D points in sparse point clouds, this solution utilizes a neighboring point information strategy, successfully applying agglomerative hierarchical clustering to sparse point clouds. Furthermore, this solution performs clustered plane detection only on the central region of the image, using the fitting error of points in the non-central region as a screening criterion. This improves the algorithm's computational speed and accuracy, while reducing computational resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 A flow chart of a plane detection method according to an embodiment of the present application is shown;

[0022] Figure 2 Showing functional modules of a computer device according to another embodiment of the present application;

[0023] Figure 3 An exemplary system is shown that can be used to implement the various embodiments described in this application.

[0024] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0025] The present application is described in further detail below with reference to the accompanying drawings.

[0026] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0027] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

[0028] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0029] The devices referred to in this application include but are not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic product that can interact with the user, such as smartphones, tablet computers, etc. The mobile electronic products can use any operating system, such as Android operating system, iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc. The network devices include but are not limited to computers, network hosts, single network servers, multiple network server sets, or a cloud consisting of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless self-organizing network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the network device and the touch terminal, or the network device and the touch terminal via a network.

[0030] Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices that are applicable to this application should also be included in the scope of protection of this application and are included here by reference.

[0031] In the description of the present application, “plurality” means two or more, unless otherwise clearly defined.

[0032] Figure 1A plane detection method according to one aspect of the present application is shown, which is applied to a computer device. The method includes step S101, step S102, step S103 and step S104. In step S101, point cloud information of a current image frame in a video stream is obtained, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information; in step S102, the current image frame is divided into a central area and a non-central area, and the point cloud information is divided into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame; in step S103, one or more corresponding sub-plane information is determined according to multiple 3D point information in the central point set information, wherein the sub-plane information includes a plane composed of at least one 3D point information in the multiple 3D point information and adjacent point information of the 3D point information; in step S104, central plane information corresponding to the central point set information is determined according to the one or more sub-plane information, and image plane information of the current image frame is determined according to the central plane information and the non-central point set information. Among them, the computer devices include but are not limited to user devices, network devices and combined devices of user devices and network devices, etc. The user devices include but are not limited to any mobile electronic products that can interact with users, such as smart phones, tablets, smart glasses, etc.; the network devices include but are not limited to computers, network hosts, single network servers, multiple network server sets or clouds composed of multiple servers.

[0033] This solution addresses the issue of user interaction with point cloud information on low-cost computing platforms, fulfilling the technical need for rapid analysis and understanding of point cloud data. Traditional man-made environments (home, office, and factory) contain numerous planes, and understanding and analyzing planes in point clouds facilitates the establishment of many user interactions. Low-cost computing platforms often utilize sparse point clouds, which require less computation. This solution addresses the problem of rapid plane detection in sparse point clouds, proposing an efficient and fast plane detection technology.

[0034] Specifically, in step S101, point cloud information of the current image frame in the video stream is obtained, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information. For example, the point cloud includes a point data set composed of multiple 3D point information of the product appearance surface obtained by measuring instruments or software algorithms. The 3D point information in the sparse point cloud is generally derived from feature points, and feature points include some point information in the current image frame with obvious features that are easy to detect and match. The 3D point information in the sparse point cloud is relatively discretely distributed, retaining only the approximate shape. Feature points refer to points where the image grayscale value changes dramatically or points with large curvature on the image edge (i.e., the intersection of two edges). Image feature points can reflect the essential characteristics of the image and can identify target objects in the image. The video stream includes a transmission stream of video data, which is usually processed as a stable, continuous stream through the network. The video stream contains multiple image frames. This solution can realize real-time image frame processing interaction during video stream transmission, for example, obtaining point cloud information of the current image frame in the video stream in real time and performing plane detection based on this point cloud information. The sparse point cloud referred to in this solution can be composed of 3D point information determined by the SLAM algorithm, or it can be a sparse point cloud obtained by filtering the dense point cloud collected by devices such as lidar and RGB-D. Here, the filtering algorithm includes but is not limited to bilateral filtering, Gaussian filtering, conditional filtering, pass filtering, voxel filtering, etc.

[0035] The computer device obtains point cloud information of the current image frame corresponding to the video stream. For example, the computer device includes a corresponding camera device, and the camera device collects relevant information of the current scene to form a video stream. The computer device intercepts the currently collected current image frame from the video stream. Alternatively, the computer device establishes a communication connection with another device (such as an external camera device, another user device, or another network device). Based on the communication connection, the computer device receives the video stream transmitted by the other device and obtains the current image frame transmitted in real time by the video stream.

[0036] In step S102, the current image frame is divided into a central region and a non-central region. Based on the central region and the non-central region, the point cloud information is divided into central point set information and non-central point set information for the current image frame. The central region includes a pixel region within a certain range centered on the image center of the current image frame. For example, points at different locations within the same image may be processed differently and have different priorities. 3D point information at the image center provides more accurate information and better reflects relevant features of the current image frame than 3D point information at the image edge. Here, we can determine the points close to the center of the image in the current image frame as the center points, and the points close to the edge of the image as the non-center points, etc. For example, first divide the corresponding center area and non-center area in the image coordinate system of the current image frame, such as determining the expression of the corresponding boundary range of the center area / non-center area or the coordinate set of pixel coordinates, etc.; then judge whether each 3D point information is in the center area or in the non-center area according to the pixel / image coordinates of each 3D point information in the image coordinate system, so that the 3D point information in the center area is composed of the corresponding center point set information, and the 3D point information in the non-center area is determined as the non-center point set information.

[0037] The central area includes a pixel area within a certain range centered on the image center of the current image frame, for example, a rectangular, square, or circular area centered on the image center, etc., without limitation. The non-central area includes part or all of the area outside the central area of the current image frame, etc. For example, the image area outside the central area of the current image frame is determined as the non-central area, or a part of the area outside the central area is determined as the corresponding non-central area, and the image area outside the non-central area and the central area is determined as the edge area, and the 3D point information of the edge area will be discarded and not involved in the calculation.

[0038] In step S103, one or more corresponding sub-plane information is determined based on the multiple 3D point information in the central point set information, wherein the sub-plane information includes a plane composed of at least one 3D point information in the multiple 3D point information and the adjacent point information of the 3D point information. For example, the computer device determines the corresponding central point set information and non-central point set information based on the point cloud information of the current image frame, and determines one or more sub-planes based on the more reliable 3D point information in the central point set information, each sub-plane consisting of the corresponding 3D point information and part or all of the adjacent point information of the 3D point information. For example, the computer device divides the central area into one or more sub-areas, and selects a 3D point in each sub-area as a seed point, and performs plane fitting on the seed point of the corresponding sub-area and the adjacent point information of the seed point, thereby determining the sub-plane information of each sub-area. Alternatively, the computer device first selects (for example, randomly selects or selects based on weight, etc.) a 3D point in the central area as a seed point, determines a sub-plane information based on the seed point and part or all of the adjacent point information of the seed point, and continues to select the next seed point after determining the sub-plane information, determines the next sub-plane information based on the next seed point and part or all of the adjacent point information of the next seed point, repeats the above steps until all 3D point information in the central point set information is traversed until no sub-plane information can be obtained, thereby obtaining one or more corresponding sub-plane information, etc.

[0039] Each sub-plane information in the one or more sub-plane information is expressed by a corresponding plane center point / seed point and a normal vector. The adjacent point information of the corresponding plane center point / seed point is calculated by the corresponding search algorithm. For example, the nearest neighbor search is used to obtain the first k 3D points closest to each 3D point as the adjacent point information, and the normal vector of the plane formed by each 3D point and its adjacent point information is calculated as the initialization normal vector of the 3D point. k is generally 2-7. If a 3D point does not find K matching adjacent point information, the point will not be subjected to subsequent operations. Among them, the search algorithm includes but is not limited to kd-tree (k-dimensional tree), random projection forest, locality sensitive hashing algorithm (LSH, Locality Sentitive Hashing), etc.

[0040] In step S104, the central plane information corresponding to the central point set information is determined based on the one or more sub-plane information, and the image plane information of the current image frame is determined based on the central plane information and the non-central point set information. For example, after the computer device determines the one or more sub-plane information of the central point set information, the computer device can determine the central plane information corresponding to the central point set information based on the one or more sub-plane information. The central plane information is used to indicate the plane information corresponding to the central area range. If there is only one sub-plane information, the computer device can directly determine the one sub-plane information as the central plane information of the central point set information; if there are multiple sub-plane information, the computer device performs plane fitting on the multiple sub-plane information according to preset rules to obtain the corresponding central plane information. In some cases, the computer device can randomly select a sub-plane information, use the sub-plane information as the starting plane, and traverse the other sub-plane information in sequence, thereby performing plane fitting on the other sub-planes that meet the fitting conditions to obtain the central plane information. In some cases, the computer device determines a corresponding starting plane from multiple sub-plane information according to a preset rule based on the plane weight information of each sub-plane information, thereby performing plane fitting on other sub-planes that meet the fitting conditions based on the starting plane to obtain the center plane information. In some cases, the computer device determines a corresponding starting plane from multiple sub-plane information according to a preset rule based on the plane weight information of each sub-plane information, performs plane fitting on at least one adjacent sub-plane that meets the fitting conditions based on the starting plane to obtain the corresponding fitting plane, and then updates the remaining sub-plane information (including other sub-plane information in one or more sub-plane information except the starting plane and the at least one adjacent sub-plane, and fitting plane information, etc.) and the plane weight information of the fitting plane information, and according to the plane weight information of the remaining sub-plane, determines the next starting plane from the remaining sub-plane information again according to the preset rule, and performs plane fitting on at least one adjacent sub-plane that meets the fitting conditions based on the next starting plane, and updates the remaining sub-plane information and the plane weight information of the fitting plane, etc., and repeats the above process until the plane fitting of the center area is completed to obtain the corresponding center plane information, etc.

[0041] The image plane information is used to indicate the plane information corresponding to the valid area in the current image frame (for example, the point set area participating in the calculation composed of the central area and the non-central area, etc.). After the computer device determines the corresponding central plane information, it can incorporate the 3D point information of the non-central area that meets certain requirements into the corresponding central plane information, thereby obtaining the corresponding image plane information. Alternatively, the computer device can also determine the sub-plane information of the non-central point set information based on the acquisition process of one or more sub-plane information in the aforementioned central point set information, and incorporate the sub-plane information of the non-central point set information into the central plane information based on different weights, thereby obtaining the corresponding image plane information, etc. The expression of the image plane information includes the corresponding center point and the normal vector. In some cases, the expression of the image plane information also includes the minimum circumscribed rectangle formed by the projection points of the 3D point information contained in the image plane information on the image plane information, etc.

[0042] In some embodiments, the method further includes step S105 (not shown), in which plane merging is performed based on the image plane information and other image plane information determined by the previous image frame in the video stream to determine the corresponding result plane information. For example, in addition to performing plane detection on the current image frame, this solution can also perform plane merging on other image plane information determined by one or more other previous image frames in the video, thereby unifying the positions of the planes in the previous and next image frames, achieving the purpose of plane tracking in the video stream, and further achieving the detection and tracking of one or more planes in space in a continuous video sequence. The result plane information includes plane information obtained by plane merging the image plane information of the current image frame with other image plane information determined by one or more previous image frames. Here, the other image plane information can include corresponding image plane information obtained by one or more image frames in the previous image frame, or can include previous result plane information obtained by merging image plane information of multiple image frames in the previous video, etc.

[0043] In some embodiments, dividing the current image frame into a central region and a non - central region includes: determining a corresponding central region according to the resolution of the current image frame and the first ratio information, where the central region includes a first pixel region within a scaled pixel range determined by the first ratio centered on the image center of the current image frame; determining a corresponding non - central region according to the resolution of the current image frame, the first ratio information, and the second ratio information, where the non - central region includes a second pixel region outside the first pixel region and within the scaled pixel range determined by the second ratio information, and the second ratio information is less than the first ratio information. For example, to ensure that the final result is as accurate as possible, when dividing the central region and the non - central region, we try to make the central region and the non - central region fit the aspect ratio of the current image frame, etc. The computer device sets corresponding first ratio information and second ratio information for determining corresponding scaled ranges according to this ratio information and determining the central region and the non - central region based on different scaled ranges, etc., where the first ratio information is greater than the second ratio information. For example, the current image frame has a corresponding resolution of (w, h), where w and h respectively represent the number of pixels corresponding to the width and height of the current image frame. Since both the corresponding central region and the non - central region are centered on the image center, if the corresponding first ratio information is r1 and the second ratio information is r2, then the pixel region satisfying r1*w < x < w - r1*w and r1*h < y < w - r1*h is divided into the central region, and the pixel region satisfying r2*w < x < w - r2*w and r2*h < y < w - r2*h and excluding the central region is divided into the non - central region, and the points in the remaining regions are discarded, where x and y represent the pixel coordinates of the corresponding pixels, etc. In some cases, r1 generally takes values from 0.3 to 0.5, and r2 generally takes values from 0.1 to 0.25.

[0044] In some embodiments, step S103 includes sub-steps S1031 (not shown) and S1032 (not shown). In step S1031, at least one initial sub-plane information corresponding to the plurality of 3D point information in the center point set information is determined, wherein the initial sub-plane information includes a plane consisting of an initial seed point of the initial sub-plane and N adjacent point information of the initial seed point, where N is a positive integer greater than or equal to 3. In step S1032, one or more sub-plane information is determined based on the at least one initial sub-plane information and the center point set information, wherein the sub-plane information includes a plane consisting of one of the at least one initial sub-plane information and adjacent point information of the at least one initial sub-plane information. For example, the computer device determines the initial sub-plane information of the central area based on the center point set information, and then forms corresponding sub-plane information based on the initial sub-plane information and the adjacent point information of the initial sub-plane. The corresponding initial sub-plane information consists of the corresponding initial seed point and N adjacent point information of the initial seed point, where N is a positive integer greater than or equal to 3. We can assign a certain weight to each point in the central point set information and determine the corresponding initial seed point from the central point set information based on the weight. Of course, the initial seed point can also be selected from the central point set information according to certain rules, such as randomly selecting or taking the point closest to the center of the image frame as the initial seed point. The corresponding N adjacent point information includes part or all of the adjacent point information of the initial seed point. For example, if N is set to the total number of adjacent points of the initial seed point, the computer device first performs plane fitting based on the initial seed point and all corresponding adjacent point information to determine the corresponding initial sub-plane information. For another example, if N is set to a fixed value (such as N = 3), the computer device selects three adjacent points from the adjacent point information of the initial seed point to fit the corresponding initial sub-plane information. The selection of these three adjacent points can be random, selected according to weight sorting, or selected in sequence. The adjacent points that are fitted with the initial seed point to form the corresponding initial sub-plane information meet certain conditions, such as the similarity between the normal vectors of the initial seed point information and the adjacent point information is greater than or equal to a similarity threshold.

[0045] In some cases, when at least one neighboring point of the initial seed point meets certain conditions with the initial seed point (for example, the normal vector similarity between the initial seed point information and the at least one neighboring point information is greater than or equal to a similarity threshold, etc.), the at least one neighboring point of the initial seed point is first included in the initial seed plane information of the initial seed point, and the neighboring point information of the at least one neighboring point information is updated to the neighboring point information of the initial seed point, until the number of neighboring points included in the initial seed plane information is greater than or equal to N, thereby determining the initial sub-plane information based on the initial seed plane information, etc.

[0046] The computer device can obtain the corresponding initial sub-plane information through plane fitting based on the corresponding initial seed point and the N adjacent point information of the initial seed point. Subsequently, based on the initial sub-plane information, a sub-plane information is obtained by incorporating the adjacent points. If the update process of the sub-plane has traversed the adjacent points of the sub-plane information in the center point set information, the update process of the sub-plane is completed; if after the update process of the sub-plane is completed, there is still remaining 3D point information in the center point set information, then other initial sub-plane information is determined based on the remaining 3D point information, and then the adjacent point information is incorporated to form other sub-plane information to obtain multiple sub-plane information, and finally the corresponding center plane information is determined based on the multiple sub-plane information.

[0047] In some embodiments, the point weight information of the initial seed point is the smallest in the point weight set of the center point set information, and the point weight set includes the point weight information of each 3D point information in the center point set information, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the center of the image. For example, the computer device can assign corresponding weight information to each 3D point information in the center point set information to determine the corresponding initial seed point, and the point weight information of each 3D point information is positively correlated with the distance of the point from the center of the image. The closer to the center, the lower the point weight information, and the farther from the center, the higher the point weight information. For example, the corresponding weight information is directly assigned according to the corresponding distance, or the corresponding weight information is calculated using the following formula:

[0048]

[0049] Where α represents the weight information of the 3D point information, x and y are the coordinates of the 3D point information in the image, and w and h are the width and height of the image. The computer device sorts the 3D point information in the center point set information to determine the 3D point information with the smallest point weight information as the corresponding initial seed point. Accordingly, if the update process of a sub-plane information determined based on the initial sub-plane information is completed and it is necessary to determine another initial sub-plane, the 3D point information with the smallest point weight is determined from the remaining 3D point information in the center point set information except for the points included in the sub-plane information as the initial seed point for the next initial sub-plane information, etc.

[0050] In some embodiments, in step S1031, the corresponding vector similarities are determined based on the initial normal vector of the initial seed point in the central point set information and the initial adjacent normal vectors of the adjacent point information of the initial seed point; if the vector similarity of a certain adjacent point information is greater than or equal to a vector similarity threshold, the adjacent point information is determined as the target adjacent point information of the initial seed point; and the initial sub-plane information corresponding to the initial seed point is determined by taking N target adjacent point information and the initial seed point, where N is a positive integer greater than or equal to 3. For example, the computer device determines the corresponding initial plane information based on each point in the central point set information and the M adjacent point information corresponding to the point, calculates the normal vector of the initial plane information, and uses the normal vector of the initial plane information as the initial normal vector of the point, where the M adjacent point information corresponding to the point can be all the adjacent point information of the point or a preset number of adjacent point information, etc. After the computer device determines the adjacent point information of each 3D point information, for the initial seed point, a neighboring point information is selected from the neighboring point information of the initial seed point. The similarity of the two vectors can be determined based on the initial normal vector of the initial seed point and the initial adjacent normal vector of one of the neighboring point information of the initial seed point (for example, the normal vector corresponding to the initial plane information determined by the one adjacent point information and the M adjacent point information corresponding to the point, etc.), such as calculating the corresponding cosine similarity based on the two vectors. Here, the vector similarity also includes but is not limited to Euclidean distance, Manhattan distance, Chebyshev distance, standardized Euclidean distance, etc.; Here, the normal vectors referred to in this application are all unit vectors. The computer device is preset with a corresponding vector similarity threshold (for example, 0.6 or 0.8, etc.). If the vector similarity of one of the neighboring point information of the initial seed point information is greater than or equal to the vector similarity threshold, the one neighboring point information is determined as the target neighboring point information. Furthermore, the target neighboring point is included in the initial seed plane information of the initial seed point. Among them, the selection of the adjacent point information can be random selection, selection based on weight sorting, or selection in sequence, etc. Then the next adjacent point information is selected from the remaining adjacent point information of the initial seed point. If the vector similarity of the next adjacent point information of the initial seed point information is greater than or equal to the vector similarity threshold, the next adjacent point information is determined as the target adjacent point information. Similarly, the target adjacent point is included in the initial seed plane information of the initial seed point. And so on. When the number of the target adjacent point information is greater than or equal to N, the computer device can perform plane fitting based on the initial seed point and the N target adjacent point information to determine the corresponding initial sub-plane information. Among them, the initial seed plane information included in the initial seed point includes recording that the target adjacent point belongs to the initial seed plane information. When the target adjacent point information is recorded in the initial seed plane information, the 3D point information corresponding to the adjacent point information no longer participates in other calculations, and all adjacent point information of the target adjacent point information is updated to the adjacent point of the initial seed point.

[0051] In some embodiments, the method further includes step S106 (not shown). In step S106, for each 3D point information in the central point set information, a nearest neighbor search algorithm is used to select M adjacent point information closest to the 3D point information from multiple 3D point information in the central point set information as the M adjacent point information of the 3D point information. Initial plane information for each 3D point information is determined based on each 3D point information and the corresponding M adjacent point information. A corresponding initial normal vector is determined based on the initial plane information of each 3D point information, where M is a positive integer greater than or equal to 2. For example, the computer device uses a nearest neighbor search algorithm based on each 3D point information in the central point set information in the central region to determine adjacent point information for each 3D point information, calculate the distance between each adjacent point information and the 3D point information, and select the first M adjacent point information closest to the 3D point information as the adjacent point information corresponding to the 3D point information, thereby ensuring calculation accuracy while avoiding excessive waste of computing resources. Since the sorting is based on the first M points closest to each point, when a point is the adjacent point information of another 3D point information, the other 3D point information is not necessarily the adjacent point information of the point. In order to ensure that the corresponding initial plane information can be successfully fitted, M is set to a positive integer greater than or equal to 2, and the value of M is usually 2-7. In order to avoid excessive computational overhead caused by too much adjacent point information, etc., the computer device determines each 3D point information and the corresponding adjacent point information, and determines the initial plane information corresponding to each 3D point information and the corresponding adjacent point information, and uses the normal vector of the corresponding initial plane information as the initial normal vector of the 3D point information.

[0052] In some embodiments, the method further includes step S107 (not shown). In step S107, if the number of neighboring points of a certain 3D point information in the central point set information is less than M, the 3D point information is directly ignored. For example, if the number of neighboring point information of a certain 3D point information in the central point set information is less than a preset M, it is determined that the 3D point information cannot be plane-fitted to determine the corresponding initial plane, and thus cannot participate in subsequent calculations. The 3D point information is ignored in the plane detection process for the current image frame, and the calculation process disregards the 3D point information, such as the 3D point information is not included in the calculation of the initial sub-plane information, sub-plane information, and central plane information.

[0053] In some embodiments, in step S1032, corresponding fitting sub-plane information is determined based on the initial sub-plane information and at least one adjacent point information of the initial sub-plane information; if the fitting sub-plane information of the at least one adjacent point information meets a preset condition, the at least one adjacent point information is incorporated into the initial sub-plane information, wherein incorporating the at least one adjacent point information into the initial sub-plane information includes updating the adjacent point information of the initial sub-plane information and updating the initial sub-plane information based on the fitting sub-plane information; the above-mentioned updating process of the initial sub-plane information and the adjacent point information of the initial sub-plane information is repeated until the adjacent point information of the latest updated initial sub-plane information is traversed, thereby determining the latest updated initial sub-plane information as the corresponding sub-plane information. For example, after the computer device determines the corresponding initial sub-plane information using N target adjacent point information and the initial seed point, the adjacent point information of the initial sub-plane information includes the adjacent points of the initial seed point corresponding to the initial sub-plane information. For example, the remaining adjacent point information of the initial seed point, excluding the adjacent point information already included in the initial seed plane information (for example, the adjacent point information of the initial seed point that has already participated in the fitting process to generate the initial sub-plane information or other sub-plane information and no longer participates in the subsequent plane fitting process, etc.) is determined as the adjacent point information of the initial sub-plane information. The at least one adjacent point information may be one of the adjacent point information of the initial sub-plane information, or may be part or all of the adjacent point information of the initial sub-plane information. The computer device performs plane fitting based on the initial sub-plane information and the at least one adjacent point information of the initial sub-plane to determine the corresponding fitted sub-plane information. The fitted sub-plane information includes the 3D information points in the initial sub-plane information and the at least one adjacent point information. After determining the corresponding fitting sub-plane information, the computer device determines whether the fitting sub-plane information meets the preset conditions. If so, the at least one adjacent point information is included in the initial sub-plane information.

[0054] The preset conditions are, but are not limited to: the mean square error of the fitted sub-plane information is less than or equal to a mean square error threshold (e.g., 600 or 640); the absolute value of the mean square error difference between the fitted sub-plane information and the initial sub-plane information is less than or equal to a mean square error difference threshold (e.g., 200 or 250). The computer device determines whether the mean square error calculated for the fitted sub-plane information is less than or equal to the mean square error threshold; the computer device also calculates the mean square error of the initial sub-plane information, thereby determining the absolute value of the mean square error difference between the fitted sub-plane information and the initial sub-plane information, and determining whether the absolute value is less than or equal to the mean square error difference threshold, wherein the plane mean square error calculation formula is as follows:

[0055]

[0056] Among them, mse is used to identify the mean square error of the plane, dis i is the distance from the i-th point to the fitting plane, is the average distance from N points to the fitting plane. The preset condition includes at least one of the above conditions. If the fitting sub-plane information meets the preset condition, the at least one adjacent point information is included in the initial sub-plane information. For example, the at least one adjacent point information is recorded to the initial sub-plane. After the at least one adjacent point information is recorded to a certain sub-plane, the at least one adjacent point no longer participates in the subsequent calculation of the plane inclusion. The computer device updates the adjacent point information of the at least one adjacent point information to the new adjacent point information of the fitting sub-plane information based on the fitting sub-plane information, and uses the fitting sub-plane information as the updated initial sub-plane information. The adjacent points of the updated initial sub-plane information include the adjacent point information in the initial sub-plane information that does not participate in the fitting sub-plane information calculation and the adjacent point information of the at least one adjacent point information, etc.

[0057] After the initial sub-plane information is updated, the computer device repeats the above fitting process based on the updated initial sub-plane information and the adjacent points of the updated initial sub-plane information, and continues to update the corresponding initial sub-plane information and the adjacent point information of the initial sub-plane information until the adjacent point information of the newly updated initial sub-plane information is traversed, and the adjacent point information of the initial sub-plane information can no longer perform the above steps. For example, the fitting sub-plane information determined by all the adjacent point information of the updated initial sub-plane information cannot meet the preset conditions, etc., thereby determining the newly updated initial sub-plane information as the corresponding sub-plane information. In some embodiments, based on this solution, the computer device can include 3D point information in the center point set information through the initial sub-plane information. If the initial sub-plane information includes 3D point information in most or all areas of the center point set information, the initial sub-plane information updated to the end of the iteration can be directly determined as the sub-plane information, and the center plane information of the center point set information can be determined based on the sub-plane information, such as directly determining the sub-plane information as the center plane information.

[0058] In some embodiments, the step S1032 also includes: step f, taking other points in the 3D point information except the sub-plane information as other seed points, determining the corresponding other initial sub-plane information based on the other seed points and the adjacent point information of the other seed points, and repeating the updating process of the other initial sub-plane information and the corresponding adjacent point information based on the other initial sub-plane information to update the other initial sub-plane information until the latest updated adjacent point information of the other initial sub-plane information is traversed, thereby determining the latest updated other initial sub-plane information as the corresponding sub-plane information, wherein there is no overlapping 3D point information between the other initial sub-plane information and the sub-plane information; repeating the above step f to obtain multiple sub-plane information until the remaining 3D point information in the 3D point information cannot execute the above step f to fit into a plane. For example, after the computer device determines the corresponding sub-plane information based on the initial sub-plane information, it can continue to perform plane fitting based on the remaining 3D point information in the center point set information, thereby determining the subsequent other sub-plane information, until all adjacent points of the other sub-plane information do not meet the preset conditions, and then repeat the above process to determine another other sub-plane information to obtain the corresponding one or more other sub-plane information. After the computer device determines the corresponding sub-plane information and one or more other sub-plane information, it uses the one sub-plane information and the one or more other sub-plane information as multiple sub-plane information of the center point set information. Here, after the 3D point information in each sub-plane information is incorporated into the sub-plane information, it no longer participates in the subsequent sub-plane update process, thereby ensuring that there is no overlapping 3D point information between each sub-plane.

[0059] In some embodiments, the point weight information of the other seed points is the smallest in the other point weight sets among the other points, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the center of the image. For example, after the computer device determines that the corresponding iterative update has completed the sub-plane information or one or more other sub-plane information, when obtaining subsequent other sub-plane information, the computer device first takes the 3D point information with the smallest point weight information in the other point weight set as the other seed point of the subsequent other sub-plane information based on the other point weight set of the remaining 3D point information other than the sub-plane information and the completed one or more other sub-plane information in the current center point set information, and then determines the corresponding subsequent other initial sub-plane information based on the other seed point and the adjacent point information of the other seed point, and repeats the updating process of the subsequent other initial sub-plane information and the corresponding adjacent point information based on the subsequent other initial sub-plane information to update the subsequent other initial sub-plane information until the newly updated adjacent point information of the subsequent other initial sub-plane information is traversed, thereby determining the newly updated subsequent other initial sub-plane information as the corresponding subsequent other sub-plane information.

[0060] In some cases, there are some 3D point information in the central point set information, and all of its adjacent point information cannot be fitted to form a sub-plane. In this case, these 3D point information are discarded during the sub-plane fitting process. These 3D point information do not participate in the plane fitting process, which reduces the computational cost while improving the accuracy and speed of the plane fitting process.

[0061] In some embodiments, determining the central plane information corresponding to the central point set information based on the one or more sub-plane information includes: based on the plane weight information of the multiple sub-plane information, selecting the sub-plane with the smallest plane weight information as the corresponding initial central plane information, wherein the plane weight information is determined by the mean square error of the corresponding sub-plane information; and determining the central plane information corresponding to the central point set information based on the initial central plane information and the remaining sub-plane information in the multiple sub-plane information. For example, the computer device calculates the plane weight information of each sub-plane information, where the plane weight information is related to the mean square error of each sub-plane information, for example, using the mean square error value as the corresponding plane weight information, etc. The mean square error calculation method of the sub-plane information is the same or similar to Formula 2 and is not further described here. The computer device determines the sub-plane information with the smallest plane weight information from the multiple sub-plane information as the initial central plane information, performs plane fitting on the initial central plane information, incorporates the adjacent sub-plane information of the initial central plane information, and so on, thereby forming the corresponding central plane information. When the adjacent point information corresponding to the 3D point information in one sub-plane information is incorporated into the information of another sub-plane information, the two sub-plane information are adjacent sub-planes.

[0062] In some embodiments, determining the central plane information corresponding to the central point set information based on the initial central plane information and the remaining sub-plane information in the multiple sub-plane information includes: g (not shown) determining corresponding merged central plane information based on the initial central plane information and at least one adjacent sub-plane information of the initial central plane information; if the merged central plane information of the at least one adjacent sub-plane information meets a preset condition, incorporating the at least one adjacent sub-plane information into the initial central plane information, and updating the initial central plane information based on the merged central plane information; repeating the above step g, traversing the multiple sub-planes until the remaining sub-plane information of the multiple sub-plane information cannot be merged into a single plane by performing the above step g, thereby determining the most recently updated initial central plane information as the central plane information corresponding to the central point set information. For example, after determining the initial central plane information, the computer device can determine one or more adjacent sub-plane information of the initial central plane information based on each 3D point information in the initial central plane information, wherein the at least one adjacent sub-plane information includes all or part of the one or more adjacent sub-plane information; and the computer device performs a planar merge of the at least one adjacent sub-plane information with the initial central plane information to obtain the corresponding merged central plane information. The computer device may determine whether the merged central plane information and the initial central plane information meet a preset condition, and if so, update the corresponding initial central plane information based on the merged central plane information.

[0063] The preset conditions include, but are not limited to: the mean square error of the merged central plane information is less than or equal to a mean square error threshold (e.g., 600 or 640); the absolute value of the difference between the mean square errors of the merged central plane information and the initial central plane information is less than or equal to a mean square error difference threshold (e.g., 200 or 250). The computer device determines whether the calculated mean square error of the merged central plane information is less than or equal to the mean square error threshold; the computer device also calculates the mean square error of the initial central plane information to determine the absolute value of the difference between the mean square errors of the merged central plane information and the initial central plane information, and determines whether the absolute value is less than or equal to the mean square error difference threshold. The preset conditions include at least one of the aforementioned conditions. If the merged center plane information meets the preset conditions, the initial center plane information is updated according to the merged center plane information. For example, the at least one adjacent sub-plane information no longer participates in the calculation of subsequent plane merging. The computer device updates the adjacent sub-plane information of the at least one adjacent sub-plane information to the new adjacent sub-plane information of the merged center plane information, and uses the merged center plane information as the updated initial center plane information. The adjacent sub-plane information of the updated initial center plane information includes the adjacent sub-plane information in the initial center plane information that does not participate in the calculation of the merged center plane information and the adjacent sub-plane information of the at least one adjacent sub-plane information, etc., wherein the adjacent sub-plane information of the at least one adjacent sub-plane information does not include the sub-plane information that has been incorporated into the initial center plane information, etc.

[0064] The computer device can continue to execute the above step g based on the updated initial center plane information and the adjacent sub-plane information of the updated initial center plane information, and continue to update the corresponding initial center plane information and the adjacent sub-plane information of the initial center plane information until the sub-plane information of the current center point set information is traversed, and the adjacent sub-plane information of the latest updated initial center plane information can no longer execute step g, for example, all the adjacent sub-plane information of the latest updated initial center plane information does not meet the preset conditions or there is no remaining adjacent sub-plane information, etc., thereby obtaining the latest updated initial center plane information at the end of the iteration, and determining the initial center plane information as the center plane information of the center point set information.

[0065] In some embodiments, step g further includes: updating the plane weight information of the initial center plane information based on the merged center plane information; taking the sub-plane information with the smallest plane weight information from the remaining sub-plane information as the new initial center plane information, wherein the remaining sub-plane information includes the updated initial center plane information and other sub-plane information except for the at least one adjacent sub-plane information. For example, after the initial center plane information is updated once, the computer device updates the plane weight information of the initial center plane information; the computer device uses the initial center plane information as the sub-plane information to participate in the subsequent plane merging process, and re-determines the sub-plane information with the smallest plane weight information from the remaining sub-plane information as the new initial center plane information. In other words, each time a plane merging process is performed, the corresponding initial center plane information is re-determined for plane merging, etc. The computer device repeats the above-mentioned plane merging and initial center plane information re-determination process, uses the updated initial center plane information as new sub-plane information, participates in the plane merging process of the remaining sub-plane information, and continues to update the newly determined initial center plane information and the adjacent sub-plane information of the initial center plane information until the sub-plane information of the current center point set information is traversed, and the adjacent sub-plane information of the latest determined initial center plane information can no longer perform the above steps, for example, the merged sub-plane information determined by all adjacent sub-planes of the latest determined initial center plane information cannot meet the preset conditions or there is no remaining adjacent sub-plane information, etc.

[0066] In some embodiments, determining the image plane information of the current image frame based on the central plane information and the non-central point set information includes: determining point-to-plane distance information between each non-central 3D point information in the non-central point set information and the central plane information; if the point-to-plane distance information of a certain non-central 3D point information is less than or equal to a preset distance threshold, determining the non-central 3D point information as target non-central 3D point information to obtain at least one target non-central 3D point information; incorporating the at least one target non-central 3D point information into the central plane information, and updating the central plane information to determine the image plane information of the current image frame. For example, for the non-central 3D point information included in the non-central point set information, based on the central plane information of the central point set information, determining the point-to-plane distance information of each non-central 3D point information to the central plane information; if there is any non-central 3D point information whose point-to-plane distance information is less than or equal to a preset distance threshold (e.g., 10-30), determining that the non-central 3D point information meets the requirements and is determined as the target non-central 3D point information. The computer device determines at least one target non-center 3D point information from the non-center point set information, and incorporates the at least one non-center 3D point information into the center plane information. For example, plane fitting is performed based on the at least one non-center 3D point information and the center 3D point information contained in the center plane information to determine the corresponding image plane information, etc. The expression of the image plane information includes a center point and a normal vector. Furthermore, the expression of the image plane information also includes the minimum circumscribed rectangle formed by the projection points of the 3D point information contained in the image plane information on the image plane information, etc.

[0067] In some embodiments, step S105 includes sub-step S1051 (not shown) and step S1052 (not shown). In step S1051, target other image plane information that meets preset plane merging conditions is determined from one or more other image plane information determined from the preceding image frame in the video stream based on the image plane information; in step S1052, the image plane information and the target other image plane information are plane-merged to determine corresponding result plane information. For example, plane detection is also performed on the preceding image frame in the video stream to obtain corresponding image plane information. Here, we refer to the image plane information determined by the preceding image frame as other image plane information. The other image plane information may include corresponding image plane information obtained from one or more preceding image frames, or may include preceding result plane information obtained by merging image plane information of multiple image frames in the preceding video. The result plane information includes plane information determined by plane merging the image plane information of the current image frame with one of the aforementioned other image plane information. This solution can obtain one or more other image plane information corresponding to the previous image frame, determine the target other image plane information that meets the preset plane merging conditions, and perform plane merging based on the image center plane information and the target other image plane information, unify the plane positions of the previous and next image frames, and determine the result plane information, thereby achieving fast and accurate plane tracking.

[0068] In some embodiments, the preset plane merging condition includes but is not limited to: the vector similarity between the normal vector of a certain other image plane information among the one or more other image plane information and the normal vector of the image plane information is greater than or equal to a preset vector similarity threshold; the projection overlap ratio between the plane projection of a certain other image plane information among the one or more other image plane information and the plane projection of the image plane information is greater than or equal to a preset projection overlap ratio threshold; the absolute value of the difference between the distance from the center point of a certain other image plane information among the one or more other image plane information to the preset plane and the distance from the center point of the image plane information to the preset plane is less than or equal to a preset plane relative distance threshold. For example, the computer device compares the normal vector of the image plane information with the normal vectors of the other image plane information. If the vector similarity is greater than or equal to the similarity threshold (e.g., a value of 0.65-0.95, etc.), the condition is met to determine that the other image plane information is the target image plane information. For example, the computer device projects the image plane information and other image plane information along a preset plane direction (e.g., horizontal and / or vertical direction). If the overlap ratio of the two projected planes is greater than or equal to a preset projection overlap ratio threshold (e.g., 0.4-0.7), then the condition is met to determine the other image plane information as the target image plane information. For another example, the computer device determines, based on the plane normal vector, which of the two planes is closer to the xy plane, yz plane, or xz plane, and uses this to calculate the distance difference between the two planes in their orientation, with the closest resulting plane being recorded as the preset plane. The computer device calculates the distance from the center points of the two planes to the preset plane, subtracts the two distances, and takes the absolute value to determine the absolute value of the relative distance difference between the two planes. If the absolute value of the relative distance difference between the two planes is less than or equal to a preset relative distance difference threshold (e.g., 30-60), then the condition is met to determine the other image plane information as the target image plane information. The preset plane merging conditions may include one or more combinations of the aforementioned conditions, without limitation. If there is other image plane information that does not meet the preset merging condition, the other image plane information is stored as a historical plane detection result and participates in the image plane merging of subsequent image frames.

[0069] In some embodiments, in step S1052, the image plane information and the target other image plane information are plane-merged to determine the corresponding candidate result plane information; if the fitting mean square error of the candidate result plane information is less than or equal to a preset fitting error threshold, the candidate result plane information is determined as the corresponding result plane information. For example, the computer device merges the image plane information of the current image frame and the target other image plane information, for example, the points of the image plane information and the points of the target other image plane information are fitted into a plane by the least squares method, and recorded as the candidate result plane information. The computer device calculates the fitting mean square error of the merged candidate result plane information. If the fitting error is less than or equal to a preset fitting error threshold (for example, a value of 6800-9000, etc.), it is determined that the candidate result plane information meets the conditions, is determined as the corresponding result plane information, and the result is output. The output result can participate in the image plane merging process of subsequent image frames, etc.

[0070] In some embodiments, the method further includes step S108 (not shown). In step S108, if the image plane information does not determine target other image plane information that meets the preset plane merging condition from one or more other image plane information determined from the previous image frame in the video stream, or if the fitting mean square error of the candidate result plane information is greater than a preset fitting error threshold, the image plane information is retained for plane merging of subsequent image frames. For example, if the image plane information fails to determine target other image plane information that meets the condition from other image plane information of the previous image frame based on the preset plane merging condition, or if the fitting mean square error of the candidate result plane information determined by merging the target other image plane information does not meet the preset fitting error threshold, it is determined that the image plane information of the current image frame cannot complete the merging of the result plane information, the image plane merging process is not performed, and the image plane information is retained as a historical plane detection result to participate in the image plane merging process of the subsequent image frame.

[0071] In some embodiments, the method further includes step S109 (not shown) before step S105. In step S109, a plane determination is performed on the image plane information to determine whether the image plane information satisfies a preset plane determination condition. If not, the image plane information is removed. For example, before performing image plane merging, the computer device may further perform a plane determination on the image plane information of the current image frame to determine whether the image plane information satisfies the preset plane determination condition. If so, the computer device performs a subsequent image plane merging process based on the image plane information. If the image plane merging condition is not satisfied, the computer device retains the image plane information as a historical plane detection result to participate in the image plane merging process of subsequent image frames. If the image plane information does not satisfy the preset plane determination condition, the computer device directly removes the image plane information without performing the image plane merging calculation for the image plane information, and the image plane information does not participate in the image plane merging process of subsequent image frames.

[0072] In some embodiments, the preset plane judgment condition includes but is not limited to: the unit vector similarity between the normal vector of the image plane information and the corresponding vertical unit vector is greater than the unit vector similarity threshold; the unit vector inner product between the normal vector of the image plane information and the corresponding vertical unit vector is less than the unit vector inner product threshold. For example, the vertical unit vector is expressed as (0,0,1). The computer device performs normal vector detection on the image plane information. If the vertical or horizontal requirements are not met, it will not participate in the image plane merging, and then perform plane detection on the next image frame until the vertical or horizontal requirements are met after the plane detection of a certain image frame. Among them, the horizontal condition of the normal vector includes but is not limited to: the similarity between the normal vector of the image plane information and the (0,0,1) unit vector is greater than or equal to the unit vector similarity threshold (for example, 0.95, etc.); the vertical condition of the normal vector of the image plane information includes but is not limited to: the inner product between the normal vector of the image plane information and the (0,0,1) unit vector is less than or equal to the unit vector inner product threshold (for example, 0.05, etc.).

[0073] The above mainly introduces the various embodiments of the plane detection method of the present application. In addition, we also provide specific devices that can implement the above embodiments. Figure 2 Make an introduction.

[0074] Figure 2A computer device 100 for plane detection according to one aspect of the present application is shown. The device includes a first module 101 , a second module 102 , a third module 103 and a fourth module 104 . A first module 101 is used to obtain point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information; a second module 102 is used to divide the current image frame into a central area and a non-central area, and divide the point cloud information into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame; a third module 103 is used to determine one or more corresponding sub-plane information based on multiple 3D point information in the central point set information, wherein the sub-plane information includes a plane composed of at least one 3D point information among the multiple 3D point information and adjacent point information of the 3D point information; a fourth module 104 is used to determine central plane information corresponding to the central point set information based on the one or more sub-plane information, and determine the image plane information of the current image frame based on the central plane information and the non-central point set information.

[0075] Here, the Figure 2 The specific implementations of the modules 101, 102, 103 and 104 are the same as those described above. Figure 1 The illustrated embodiments of step S101 , step S102 , step S103 and step S104 are the same or similar, and thus are not described in detail and are incorporated herein by reference.

[0076] In some embodiments, the device further includes a module (not shown) configured to perform plane merging based on the image plane information and other image plane information determined by a previous image frame in the video stream, and determine corresponding result plane information. In some embodiments, dividing the current image frame into a central area and a non-central area includes: determining the corresponding central area based on the resolution of the current image frame and first ratio information, wherein the central area includes a first pixel area centered on the image center of the current image frame and having a scaled pixel range determined by the first ratio; and determining the corresponding non-central area based on the resolution of the current image frame, the first ratio information, and the second ratio information, wherein the non-central area includes a second pixel area outside the first pixel area and within the scaled pixel range determined by the second ratio information, and the second ratio information is smaller than the first ratio information.

[0077] In some embodiments, the one-three module 103 includes a one-three-one unit (not shown) and a one-three-two unit (not shown); the one-three-one unit is used to determine at least one initial sub-plane information corresponding to multiple 3D point information in the center point set information, wherein the initial sub-plane information includes a plane composed of an initial seed point of the initial sub-plane and N adjacent point information of the initial seed point, where N is a positive integer greater than or equal to 3; the one-three-two unit is used to determine one or more sub-plane information based on the at least one initial sub-plane information and the center point set information, wherein the sub-plane information includes a plane composed of one of the at least one initial sub-plane information and the adjacent point information of the at least one initial sub-plane information.

[0078] In some embodiments, the point weight information of the initial seed point is the smallest in the point weight set of the center point set information, and the point weight set includes the point weight information of each 3D point information in the center point set information, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the center of the image.

[0079] In some embodiments, a 131 unit is used to determine the corresponding vector similarity based on the initial normal vector of the initial seed point in the center point set information and the initial adjacent normal vector of the adjacent point information of the initial seed point; if the vector similarity of a certain adjacent point information is greater than or equal to the vector similarity threshold, the adjacent point information is determined as the target adjacent point information of the initial seed point; N target adjacent point information and the initial seed point are taken to determine the corresponding initial sub-plane information, where N is a positive integer greater than or equal to 3.

[0080] In some embodiments, the device further includes a six-module (not shown) for using a nearest neighbor search algorithm for each 3D point information in the center point set information to obtain the M adjacent point information closest to the 3D point information from multiple 3D point information in the center point set information as the M adjacent point information of the 3D point information, determine the initial plane information of each 3D point information based on each 3D point information and the corresponding M adjacent point information, and determine the corresponding initial normal vector based on the initial plane information of each 3D point information, where M is a positive integer greater than or equal to 2.

[0081] In some embodiments, the device further includes a module (not shown) configured to directly ignore a certain 3D point information if the number of neighboring points of the 3D point information in the central point set information is less than M.

[0082] In some embodiments, unit 132 is used to determine the corresponding fitting sub-plane information based on the initial sub-plane information and at least one adjacent point information of the initial sub-plane information; if the fitting sub-plane information of the at least one adjacent point information meets a preset condition, the at least one adjacent point information is incorporated into the initial sub-plane information, wherein the incorporation of the at least one adjacent point information into the initial sub-plane information includes updating the adjacent point information of the initial sub-plane information and updating the initial sub-plane information based on the fitting sub-plane information; repeat the above-mentioned updating process of the initial sub-plane information and the adjacent point information of the initial sub-plane information until the adjacent point information of the latest updated initial sub-plane information is traversed, thereby determining the latest updated initial sub-plane information as the corresponding sub-plane information.

[0083] In some embodiments, the one-three-two unit is also used to: f, take other points in the 3D point information except the sub-plane information as other seed points, determine the corresponding other initial sub-plane information based on the other seed points and the adjacent point information of the other seed points, and repeat the updating process of the other initial sub-plane information and the corresponding adjacent point information based on the other initial sub-plane information to update the other initial sub-plane information until the latest updated adjacent point information of the other initial sub-plane information is traversed, thereby determining the latest updated other initial sub-plane information as the corresponding sub-plane information, wherein there is no overlapping 3D point information between the other initial sub-plane information and the sub-plane information; repeat the above step f to obtain multiple sub-plane information until the remaining 3D point information in the 3D point information cannot execute the above step f to fit into a plane.

[0084] In some embodiments, the point weight information of the other seed points is the smallest in a set of other point weights among the other points, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the image center.

[0085] In some embodiments, determining the center plane information corresponding to the center point set information based on the one or more sub-plane information includes: based on the plane weight information of the multiple sub-plane information, taking the sub-plane with the smallest plane weight information as the corresponding initial center plane information, wherein the plane weight information is determined by the mean square error of the corresponding sub-plane information; and determining the center plane information corresponding to the center point set information based on the initial center plane information and the remaining sub-plane information in the multiple sub-plane information.

[0086] In some embodiments, determining the center plane information corresponding to the center point set information based on the initial center plane information and the remaining sub-plane information in the multiple sub-plane information includes: g (not shown) determining the corresponding merged center plane information based on the initial center plane information and at least one adjacent sub-plane information of the initial center plane information; if the merged center plane information of the at least one adjacent sub-plane information meets a preset condition, incorporating the at least one adjacent sub-plane information into the initial center plane information, and updating the initial center plane information based on the merged center plane information; repeating the above step g, traversing the multiple sub-planes until the remaining sub-plane information of the multiple sub-plane information cannot execute the above step g to merge into one plane, thereby determining the latest updated initial center plane information as the center plane information corresponding to the center point set information.

[0087] In some embodiments, step g also includes: updating the plane weight information of the initial center plane information based on the merged center plane information; taking the sub-plane information with the smallest plane weight information from the remaining sub-plane information as the new initial center plane information, wherein the remaining sub-plane information includes the updated initial center plane information and other sub-plane information except the at least one adjacent sub-plane information.

[0088] In some embodiments, the 15 module includes a 151 unit (not shown) and a 152 unit (not shown). The 151 unit is configured to determine, based on the image plane information, target other image plane information that meets a preset plane merging condition from one or more other image plane information determined in a preceding image frame in the video stream; and the 152 unit is configured to perform plane merging on the image plane information and the target other image plane information to determine corresponding result plane information.

[0089] In some embodiments, the preset plane merging conditions include but are not limited to: the vector similarity between the normal vector of a certain other image plane information in the one or more other image plane information and the normal vector of the image plane information is greater than or equal to a preset vector similarity threshold; the projection overlap ratio of the plane projection of a certain other image plane information in the one or more other image plane information and the plane projection of the image plane information is greater than or equal to a preset projection overlap ratio threshold; the absolute value of the difference between the distance from the center point of a certain other image plane information in the one or more other image plane information to the preset plane and the distance from the center point of the image plane information to the preset plane is less than or equal to a preset plane relative distance threshold.

[0090] In some embodiments, unit 152 is used to perform a planar merger on the image plane information and other target image plane information to determine the corresponding candidate result plane information; if the fitting mean square error of the candidate result plane information is less than or equal to a preset fitting error threshold, the candidate result plane information is determined as the corresponding result plane information.

[0091] In some embodiments, the device also includes an eight-module (not shown) for retaining the image plane information for plane merging of subsequent image frames if the image plane information does not determine target other image plane information that meets the preset plane merging conditions from one or more other image plane information determined from the previous image frame in the video stream, or if the fitting mean square error of the candidate result plane information is greater than a preset fitting error threshold.

[0092] In some embodiments, the device further includes a module 19 (not shown) before module 15, configured to perform plane determination on the image plane information to determine whether the image plane information meets a preset plane determination condition, and if not, remove the image plane information. In some embodiments, the preset plane determination condition includes, but is not limited to: a unit vector similarity between the normal vector of the image plane information and the corresponding vertical unit vector is greater than a unit vector similarity threshold; and a unit vector inner product between the normal vector of the image plane information and the corresponding vertical unit vector is less than a unit vector inner product threshold.

[0093] Here, the specific implementations corresponding to the modules 15 to 19 are the same as or similar to the embodiments of the aforementioned steps S105 to S109, and thus are not described in detail and are included herein by reference.

[0094] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the above items is executed.

[0095] The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any one of the preceding items is executed.

[0096] The present application also provides a computer device, comprising:

[0097] one or more processors;

[0098] a memory for storing one or more computer programs;

[0099] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any one of the preceding items.

[0100] Figure 3 shows an exemplary system that can be used to implement the various embodiments described in this application;

[0101] like Figure 3 In some embodiments, the system 300 can function as any of the aforementioned devices in the various embodiments described. In some embodiments, the system 300 may include one or more computer-readable media (e.g., system memory or NVM / storage device 320) having instructions and one or more processors (e.g., processor(s) 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement the modules and thereby perform the actions described herein.

[0102] For one embodiment, system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 305 and / or any suitable device or component in communication with system control module 310 .

[0103] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0104] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. For one embodiment, system memory 315 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, system memory 315 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0105] For one embodiment, system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to NVM / storage device 320 and communication interface(s) 325 .

[0106] For example, NVM / storage 320 may be used to store data and / or instructions. NVM / storage 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0107] NVM / storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 320 may be accessed over a network via communication interface(s) 325.

[0108] Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and / or with any other suitable devices. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0109] For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 to form a system-on-chip (SoC).

[0110] In various embodiments, system 300 may be, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, system 300 may have more or fewer components and / or a different architecture. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0111] It should be noted that the application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the application can be executed by a processor to realize the steps or functions described above. Similarly, the software program of the application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0112] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes but is not limited to a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0113] Communication media include media by which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media may include guided transmission media such as cables and wires (e.g., fiber optic, coaxial, etc.) and wireless (unguided transmission) media capable of propagating energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data may be embodied as, for example, a modulated data signal in a wireless medium such as a carrier wave or similar mechanism such as that embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal that has one or more of its characteristics changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0114] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information / data for use by a computer system.

[0115] Here, according to one embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the methods and / or technical solutions based on the aforementioned multiple embodiments of the present application.

[0116] It is obvious to those skilled in the art that the present application is not limited to the details of the above-mentioned exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A plane detection method, wherein: The method includes: Obtaining point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of multiple 3D point information; Dividing the current image frame into a central area and a non-central area, and dividing the point cloud information into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame; Determine the corresponding vector similarity based on the initial normal vector of the initial seed point in the central point set information and the initial adjacent normal vector of the adjacent point information of the initial seed point, wherein the normal vector is the normal vector of the plane information determined by each point and the M adjacent points of the point; if the vector similarity of a certain adjacent point information is greater than or equal to the vector similarity threshold, then determine the adjacent point information as the target adjacent point information of the initial seed point; take N target adjacent point information and the initial seed point to determine the corresponding initial sub-plane information, wherein N is a positive integer greater than or equal to 3, and the initial sub-plane information is obtained by plane fitting the corresponding initial seed point and the N target adjacent point information; determining one or more sub-plane information according to the at least one initial sub-plane information and the central point set information, wherein the sub-plane information includes a plane formed by one of the at least one initial sub-plane information and adjacent point information of the at least one initial sub-plane information; Determine the center plane information corresponding to the center point set information based on the one or more sub-plane information, and determine the image plane information of the current image frame based on the center plane information and the non-center point set information, wherein the center plane information is used to indicate the plane information corresponding to the center area range.

2. The method according to claim 1, wherein The method further comprises: Plane merging is performed according to the image plane information and other image plane information determined by a preceding image frame in the video stream to determine corresponding result plane information.

3. The method according to claim 1, wherein The dividing the current image frame into a central area and a non-central area includes: Determining a corresponding central area according to the resolution of the current image frame and first ratio information, wherein the central area includes a first pixel area having an image center of the current image frame as the center and a scaled pixel range determined by the first ratio; A corresponding non-central area is determined based on the resolution of the current image frame, the first ratio information, and the second ratio information, wherein the non-central area includes a second pixel area outside the first pixel area and within a scaled pixel range determined by the second ratio information, and the second ratio information is smaller than the first ratio information.

4. The method according to claim 1, wherein The method further comprises: For each 3D point information in the central point set information, a nearest neighbor search algorithm is used to select M adjacent point information closest to the 3D point information from multiple 3D point information in the central point set information as the M adjacent point information of the 3D point information, and initial plane information of each 3D point information is determined according to each 3D point information and the corresponding M adjacent point information. Based on the initial plane information of each 3D point information, a corresponding initial normal vector is determined, where M is a positive integer greater than or equal to 2.

5. The method according to claim 4, wherein The method further comprises: If the number of neighboring points of a certain 3D point information in the central point set information is less than M, the 3D point information is directly ignored.

6. The method according to claim 1, wherein The determining one or more sub-plane information according to the at least one initial sub-plane information and the center point set information includes: Determining corresponding fitted sub-plane information based on the initial sub-plane information and at least one adjacent point information of the initial sub-plane information; if the fitted sub-plane information of the at least one adjacent point information satisfies a preset condition, incorporating the adjacent point information into the initial sub-plane information, wherein incorporating the adjacent point information into the initial sub-plane information includes updating the adjacent point information of the initial sub-plane information and updating the initial sub-plane information based on the fitted sub-plane information; The updating process of the initial sub-plane information and the adjacent point information of the initial sub-plane information is repeated until the adjacent point information of the latest updated initial sub-plane information is traversed, thereby determining the latest updated initial sub-plane information as the corresponding sub-plane information.

7. The method according to claim 6, wherein: The determining one or more sub-plane information according to the at least one initial sub-plane information and the center point set information further includes: f. Taking other points in the 3D point information except the sub-plane information as other seed points, determining corresponding other initial sub-plane information according to the other seed points and the adjacent point information of the other seed points, and repeating the updating process of the other initial sub-plane information and the corresponding adjacent point information based on the other initial sub-plane information to update the other initial sub-plane information until the newly updated adjacent point information of the other initial sub-plane information is traversed, thereby determining the newly updated other initial sub-plane information as the corresponding sub-plane information, wherein the other initial sub-plane information does not have overlapping 3D point information with the sub-plane information; Repeat step f to obtain multiple sub-plane information until the remaining 3D point information in the 3D point information cannot be fitted into a plane by performing step f.

8. The method according to any one of claims 1 or 4 to 7, wherein The point weight information of the initial seed point is the smallest in the point weight set of the center point set information, and the point weight set includes the point weight information of each 3D point information in the center point set information, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the image center.

9. The method according to claim 7, wherein: The point weight information of the other seed points is the smallest in the other point weight sets of the other points, and the point weight information of each 3D point information is positively correlated with the distance between the 3D point information and the image center.

10. The method according to claim 1, wherein The determining, based on the one or more sub-plane information, the central plane information corresponding to the central point set information includes: According to the plane weight information of the plurality of sub-plane information, the sub-plane with the smallest plane weight information is used as the corresponding initial central plane information, wherein the plane weight information is determined by the mean square error of the corresponding sub-plane information; The central plane information corresponding to the central point set information is determined according to the initial central plane information and the remaining sub-plane information in the plurality of sub-plane information.

11. The method according to claim 10, wherein: The determining, according to the initial central plane information and the remaining sub-plane information in the plurality of sub-plane information, the central plane information corresponding to the central point set information includes: g. determining corresponding merged central plane information based on the initial central plane information and at least one adjacent sub-plane information of the initial central plane information; if the merged central plane information of the at least one adjacent sub-plane information meets a preset condition, incorporating the at least one adjacent sub-plane information into the initial central plane information, and updating the initial central plane information based on the merged central plane information; Repeat step g above, traversing the multiple sub-planes until the remaining sub-plane information of the multiple sub-plane information cannot perform step g above to merge into one plane, thereby determining the most recently updated initial central plane information as the central plane information corresponding to the central point set information.

12. The method according to claim 11, wherein The step g further comprises: updating the plane weight information of the initial center plane information based on the merged center plane information; The sub-plane information with the smallest plane weight information is selected from the remaining sub-plane information as the new initial central plane information, wherein the remaining sub-plane information includes the updated initial central plane information and other sub-plane information except the at least one adjacent sub-plane information.

13. The method according to claim 1, wherein The determining the image plane information of the current image frame according to the central plane information and the non-central point set information includes: Determine the point-to-plane distance information between each non-center 3D point information in the non-center point set information and the center plane information; If the point-to-surface distance information of a certain non-central 3D point information is less than or equal to a preset distance threshold, the non-central 3D point information is determined as the target non-central 3D point information to obtain at least one target non-central 3D point information; The at least one target non-central 3D point information is incorporated into the central plane information, and the central plane information is updated to determine the image plane information of the current image frame.

14. The method according to claim 2, wherein: The performing plane merging according to the image plane information and other image plane information determined by a preceding image frame in the video stream to determine corresponding result plane information includes: determining target other image plane information that meets a preset plane merging condition from one or more other image plane information determined by a preceding image frame in the video stream according to the image plane information; The image plane information and the target other image plane information are plane-merged to determine corresponding result plane information.

15. The method according to claim 14, wherein The preset plane merging condition includes at least one of the following: A vector similarity between a normal vector of one of the one or more other image plane information and a normal vector of the image plane information is greater than or equal to a preset vector similarity threshold; A projection overlap ratio between a plane projection of one of the one or more other image plane information and the plane projection of the image plane information is greater than or equal to a preset projection overlap ratio threshold; An absolute value of a difference between a distance from a center point of one of the one or more other image plane information to a preset plane and a distance from a center point of the image plane information to the preset plane is less than or equal to a preset plane relative distance difference threshold.

16. The method according to claim 14, wherein The performing plane merging of the image plane information and the target other image plane information to determine corresponding result plane information includes: Performing a planar merging of the image plane information and the target other image plane information to determine corresponding candidate result plane information; If the fitting mean square error of the candidate result plane information is less than or equal to a preset fitting error threshold, the candidate result plane information is determined as the corresponding result plane information.

17. The method according to claim 16, wherein The method further comprises: If the image plane information does not determine the target other image plane information that meets the preset plane merging conditions from one or more other image plane information determined from the previous image frame in the video stream, or if the fitting mean square error of the candidate result plane information is greater than the preset fitting error threshold, the image plane information is retained for plane merging of subsequent image frames.

18. The method according to any one of claims 14 to 17, wherein The method further includes, before performing plane merging based on the image plane information and other image plane information determined according to a preceding image frame in the video stream to determine corresponding result plane information: Perform plane judgment on the image plane information to determine whether the image plane information meets a preset plane judgment condition; if not, remove the image plane information.

19. The method according to claim 18, wherein The preset plane judgment condition includes any one of the following: The unit vector similarity between the normal vector of the image plane information and the corresponding vertical unit vector is greater than a unit vector similarity threshold; A unit vector inner product of a normal vector of the image plane information and a corresponding vertical unit vector is less than a unit vector inner product threshold.

20. A plane detection device, wherein: The device includes: A module for obtaining point cloud information of a current image frame in a video stream, wherein the point cloud information includes a sparse point cloud composed of a plurality of 3D point information; Module 12 is configured to divide the current image frame into a central area and a non-central area, and divide the point cloud information into central point set information and non-central point set information of the current image frame according to the central area and the non-central area, wherein the central area includes a pixel area within a certain range centered on the image center of the current image frame; A three-module is configured to determine the corresponding vector similarity based on the initial normal vector of the initial seed point in the point cloud information and the initial adjacent normal vector of the adjacent point information of the initial seed point, wherein the normal vector is the normal vector of the plane information determined by each point and the M adjacent points of the point; if the vector similarity of a certain adjacent point information is greater than or equal to the vector similarity threshold, the adjacent point information is determined as the target adjacent point information of the initial seed point; N target adjacent point information and the initial seed point are used to determine the corresponding initial sub-plane information, wherein N is a positive integer greater than or equal to 3, and the initial sub-plane information is obtained by plane fitting the corresponding initial seed point and the N target adjacent point information; one or more sub-plane information is determined based on the at least one initial sub-plane information and the center point set information, wherein the sub-plane information includes a plane composed of one of the at least one initial sub-plane information and the adjacent point information of the at least one initial sub-plane information, and is determined based on the at least one initial sub-plane information and the center point set information; A fourth module is used to determine the center plane information corresponding to the center point set information based on the one or more sub-plane information, and to determine the image plane information of the current image frame based on the center plane information and the non-center point set information, wherein the center plane information is used to indicate the plane information corresponding to the center area range.

21. A computer device, wherein: The device includes: processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 19.

22. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: The computer program / instructions, when executed, cause the system to perform the steps of the method as claimed in any one of claims 1 to 19.

23. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

Images