Processing method and device
By acquiring point features and line features in environmental images in augmented reality AR applications, and determining three-dimensional spatial position information in combination with inertial measurement units, the problems of high cost and poor universality in detecting physical worlds in the prior art are solved, and the multi-dimensional space is achieved Accurate and efficient plane detection on equipment.
Patent Information
- Application Number
- CN202110339210.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-03-30
AI Technical Summary
In existing augmented reality AR applications, the method of detecting physical world planes is expensive and not very versatile, especially on devices such as mobile phones that do not contain depth cameras, and the RGB camera-based solution detects in areas with poor textures. Not accurate enough.
By acquiring point features and line features in the environment image, a monocular or multi-eye camera is used to determine the three-dimensional spatial position information of feature points and lines in combination with an inertial measurement unit, and then the plane information in the environment is determined to realize plane detection.
No depth camera is required, and the plane information can be accurately detected in environments with less texture, improving the accuracy and robustness of detection, and reducing processing complexity, and suitable for a variety of devices.
Smart Images
Figure CN113052977B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of image processing, and in particular to a processing method and device. Background Art
[0002] In augmented reality (AR) applications, a typical application scenario is to detect planes in the physical world, such as horizontal desktops and floors, vertical walls, etc., for displaying virtual models, such as virtual dinosaurs or airplane models, etc. Therefore, how to detect and output plane information on mobile devices such as mobile phones and AR glasses in real time and accurately is an urgent problem to be solved in augmented reality (AR) applications.
[0003] In the prior art, a monocular or binocular camera and a depth camera are generally used to detect planes, wherein the monocular or binocular camera is used to run the SLAM algorithm to solve the 6dof posture data, and the depth camera is used to obtain the depth information of the surrounding environment to obtain the three-dimensional dense point cloud information of the surrounding environment, thereby detecting the geometric plane of the surrounding environment. Since a depth camera is required, the cost is high, and it cannot be applied to devices such as mobile phones that do not contain a depth camera, and its versatility is not strong. Another plane detection solution based on RGB cameras, such as Google's ARcore, does not require the use of a depth camera, but its detection is not accurate enough, especially in places with sparse textures, it can only detect part of the plane, such as the plane at the edge of the desktop cannot be detected. In addition, this solution is not easy to expand to AR glasses based on binocular fisheye cameras. Summary of the invention
[0004] According to one of the solutions of the present disclosure, a processing method is provided, comprising:
[0005] Obtaining an environment image of the environment in which the electronic device is currently located, and acquiring point features and / or line features on the environment image;
[0006] Determine the three-dimensional spatial position information of the point feature and / or line feature;
[0007] Plane information within the environment where the electronic device is currently located is determined based at least on the three-dimensional space position information.
[0008] In some embodiments, the method further comprises:
[0009] The content to be outputted by the electronic device is outputted to a target plane in the environment at least based on the plane information.
[0010] In some embodiments, obtaining point features and / or line features on the environment image includes:
[0011] Extracting corner points in the environment image whose pixel changes are greater than a first preset threshold, taking the corner points as feature points, and obtaining the point features; and / or,
[0012] Pixels whose gradient changes are greater than a second preset threshold value are extracted from the environment image, and the pixels are used as feature lines to obtain the line features.
[0013] In some embodiments, determining the three-dimensional spatial position information of the point feature and / or line feature includes:
[0014] Solving the point features and / or line features to obtain relative spatial coordinates of feature points representing the point features and / or feature lines representing the line features, wherein the point features and / or line features are extracted from the environment image captured by a monocular camera;
[0015] The absolute spatial coordinates of the characteristic points and / or characteristic lines are determined using an inertial measurement unit.
[0016] In some embodiments, determining the three-dimensional spatial position information of the point feature and / or line feature includes:
[0017] The point features and / or line features are solved to obtain the absolute spatial coordinates of feature points representing the point features and / or feature lines representing the line features, wherein the point features and / or line features are extracted from the environment image collected by a binocular camera.
[0018] In some embodiments, determining plane information in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes:
[0019] Selecting three first feature points from the plurality of feature points based at least on three-dimensional spatial position information of the plurality of feature points;
[0020] triangulate the three first feature points to form a first plane;
[0021] Based on the number of second feature points belonging to the first plane, determine the effective plane in the current environment of the electronic device, wherein the second feature point is a feature point different from the first feature point among the multiple feature points, and the effective plane is the plane containing the most second feature points.
[0022] In some embodiments, determining plane information in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes:
[0023] Selecting two first feature lines from the plurality of feature lines based at least on the three-dimensional spatial position information of the plurality of feature lines;
[0024] forming a second plane using the two first characteristic lines;
[0025] Detecting whether the second plane is valid based on a second characteristic line among the plurality of characteristic lines, wherein the second characteristic line is different from the first characteristic line;
[0026] The valid second plane is determined as a valid plane in the environment where the electronic device is currently located.
[0027] In some embodiments, wherein determining the plane information within the environment where the electronic device is currently located based at least on the three-dimensional spatial position information comprises: determining a third plane within the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of a plurality of feature points;
[0028] Determining a fourth plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines;
[0029] determining whether the third plane and the fourth plane are the same plane;
[0030] If so, a fifth plane formed by merging the third plane and the fourth plane is determined as a valid plane in the environment where the electronic device is currently located.
[0031] In some embodiments, determining plane information in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes:
[0032] Determine a first valid plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points;
[0033] Determine edge information of the first effective plane based on three-dimensional spatial position information of at least one feature line in the first effective plane; or,
[0034] Determine a second valid plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines;
[0035] The second effective plane is expanded based on the three-dimensional spatial position information of a plurality of feature points in the second effective plane to determine edge information of the second effective plane.
[0036] According to one of the solutions of the present disclosure, there is also provided a processing device, comprising:
[0037] An acquisition module, used to obtain an environment image of the environment in which the electronic device is currently located, and to obtain point features and / or line features on the environment image;
[0038] A first determination module, used to determine the three-dimensional spatial position information of the point feature and / or line feature;
[0039] The second determination module is used to determine the plane information in the environment where the electronic device is currently located based at least on the three-dimensional space position information.
[0040] According to one of the solutions of the present disclosure, there is also provided an electronic device, including a processor and a memory, wherein the memory is used to store computer-executable instructions, and the processor implements the above-mentioned processing method when executing the computer-executable instructions.
[0041] According to one of the solutions of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the above-mentioned processing method is implemented.
[0042] The processing methods and devices provided by various embodiments of the present disclosure use a general monocular or multi-camera to collect an environmental image of the environment in which the electronic device is currently located, and extract point features and / or line features therefrom, and determine the plane information in the environment in which the electronic device is currently located based on the extracted point features and / or line features. The operation is convenient, no depth camera is required, and the versatility is strong. At the same time, the plane information in the environment in which the electronic device is currently located can be accurately and comprehensively detected, and the plane information can also be accurately determined in an environment with less texture, and the robustness is good. In addition, the processing method provided by the embodiments of the present disclosure only needs to extract sparse point features and line features, does not involve dense feature extraction, can determine plane information in real time, and improves processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart showing a processing method of an embodiment of the present disclosure;
[0044] Figure 2 A flow chart showing a method for determining plane information using point features according to an embodiment of the present disclosure;
[0045] Figure 3 A flow chart showing a method for determining plane information using line features according to an embodiment of the present disclosure;
[0046] Figure 4 A flow chart showing a method for determining plane information using point features and line features according to an embodiment of the present disclosure;
[0047] Figure 5 A flowchart showing another method for determining plane information using point features and line features according to an embodiment of the present disclosure;
[0048] Figure 6 A flowchart showing another method for determining plane information using point features and line features according to an embodiment of the present disclosure;
[0049] FIG. 7( a ) and FIG. 7( b ) are schematic diagrams showing how to determine plane information using point features according to an embodiment of the present disclosure;
[0050] Figure 8(a) to Figure 8(c)Another schematic diagram showing the use of point features to determine plane information according to an embodiment of the present disclosure;
[0051] Fig. 9 A schematic diagram showing the use of line features to determine plane information according to an embodiment of the present disclosure;
[0052] FIG. 10( a ) and FIG. 10( b ) are schematic diagrams showing how to determine plane information using point features and line features according to an embodiment of the present disclosure;
[0053] Fig.11 Another schematic diagram showing the use of point features and line features to determine plane information according to an embodiment of the present disclosure;
[0054] Fig.12 Another schematic diagram showing the use of point features and line features to determine plane information according to an embodiment of the present disclosure;
[0055] Fig.13 A schematic diagram showing the structure of a processing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0056] Various aspects and features of the present disclosure are described herein with reference to the drawings.
[0057] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.
[0058] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.
[0059] These and other characteristics of the present disclosure will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the attached drawings.
[0060] It should also be understood that although the disclosure has been described with reference to certain specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the disclosure.
[0061] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.
[0062] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments claimed are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.
[0063] Figure 1 A flow chart of the processing method of the embodiment of the present disclosure is shown. Figure 1 As shown, the embodiment of the present disclosure provides a processing method, including:
[0064] S101: Obtain an environment image of the environment in which the electronic device is currently located, and obtain point features and / or line features on the environment image.
[0065] Specifically, continuous image frames of the current environment of the electronic device can be obtained through an image acquisition device, and point features and / or line features can be extracted from the continuous image frames; or only the current image frame of the current environment of the electronic device can be obtained, and point features and / or line features can be extracted from the current frame image.
[0066] The image acquisition device can be a monocular camera (such as a monocular RGB camera) or a multi-eye camera (such as a binocular fisheye camera on AR glasses), which is not specifically limited in the embodiments of the present disclosure.
[0067] In some embodiments, in step S101, obtaining point features and / or line features on the environment image includes:
[0068] Extracting corner points in the environment image whose pixel changes are greater than a first preset threshold, taking the corner points as feature points, and obtaining the point features; and / or,
[0069] Pixels whose gradient changes are greater than a second preset threshold value are extracted from the environment image, and the pixels are used as feature lines to obtain the line features.
[0070] Specifically, when acquiring point features on the environment image, point features can be extracted by finding corner points in the environment image where pixel changes are significant (greater than a first preset threshold) based on feature extraction algorithms such as SIFT, SURF and ORB.
[0071] When acquiring line features on the environment image, line features can be extracted by searching for pixels with large gradient changes (greater than a second preset threshold) in the environment image based on the LSD (Line Segment Detector) line feature extraction algorithm. The gradient of a pixel refers to the degree of change in the pixel value. The feature line can be a feature straight line or a feature line segment. In a specific implementation, line features can also be extracted based on line feature extraction algorithms such as FLD (Fast Line Detector) and EDline.
[0072] It should be noted that the point features and / or line features extracted above are all sparse point features and / or line features, and there is no need to perform dense feature extraction, which can improve processing efficiency.
[0073] S102: Determine the three-dimensional spatial position information of the point feature and / or line feature.
[0074] After extracting the point features and / or line features on the environment image, the three-dimensional spatial position information of the point features and / or line features is calculated.
[0075] In some embodiments, in step S102, determining the three-dimensional spatial position information of the point feature and / or line feature includes:
[0076] Solving the point features and / or line features to obtain relative spatial coordinates of feature points representing the point features and / or feature lines representing the line features, wherein the point features and / or line features are extracted from the environment image captured by a monocular camera;
[0077] The absolute spatial coordinates of the characteristic points and / or characteristic lines are determined using an inertial measurement unit.
[0078] Specifically, when the environment image is a monocular image captured by a monocular camera, the SLAM (Simultaneous Localization And Mapping) algorithm is first used to extract point features in the monocular image, and then the 6dof posture data of the monocular camera is solved, and then the three-dimensional relative spatial coordinates of the feature points representing the point features in the environment image and / or the feature lines representing the line features in the environment image (the feature lines can be obtained based on two feature points) are obtained.
[0079] For example, when the same feature point is observed in two image frames captured by a monocular camera, the 3D spatial coordinates of the feature point can be calculated using the 6dof posture data of the two image frames. Since the monocular camera lacks depth information, the 3D spatial coordinates calculated by the above SLAM algorithm are relative spatial coordinates.
[0080] Furthermore, in order to obtain the absolute spatial coordinates of the feature points and / or feature lines, an inertial measurement unit (IMU) may be used to restore the scale information of the spatial coordinates, thereby obtaining the absolute spatial coordinates.
[0081] The above-mentioned monocular camera and inertial measurement unit (IMU) are common settings of mobile phones and other devices, and therefore, are easy to operate.
[0082] In some other embodiments, in step S102, determining the three-dimensional spatial position information of the point feature and / or line feature includes:
[0083] The point features and / or line features are solved to obtain the absolute spatial coordinates of the feature points representing the point features and / or the feature lines representing the line features in the environment image, wherein the point features and / or line features are extracted from the environment image collected by a binocular camera.
[0084] When the environment image is a multi-eye image collected by a multi-eye camera, the SLAM algorithm is first used to extract the point features and / or line features in the multi-eye image, and then the 6dof posture data of the multi-eye camera is solved, and then the three-dimensional absolute spatial coordinates of the feature points representing the point features and / or the feature lines representing the line features are obtained. The multi-eye camera can use stereoscopic vision (multi-eye parallax) to restore depth information, so the SLAM algorithm can be used to directly obtain the three-dimensional absolute spatial coordinates of the feature points and / or feature lines.
[0085] S103: Determine plane information in the environment where the electronic device is currently located based at least on the three-dimensional space position information.
[0086] After determining the three-dimensional absolute spatial coordinates of the feature points and / or feature lines, the plane information in the environment image can be determined, and then the plane information in the environment where the electronic device is currently located can be determined to provide more accurate and rich plane information. In addition, the absolute spatial coordinates are used as the three-dimensional spatial position information, and the accuracy of the plane information determination can be guaranteed due to the stability and accuracy of the absolute spatial coordinates.
[0087] The processing method provided by the embodiment of the present disclosure uses a general monocular or multi-camera to collect an environmental image of the environment in which the electronic device is currently located, and extracts point features and / or line features therefrom, and determines the plane information in the environment in which the electronic device is currently located based on the extracted point features and / or line features. The operation is convenient, no depth camera is required, and the versatility is strong. At the same time, the plane information in the environment in which the electronic device is currently located can be accurately and comprehensively detected, and the plane information can also be accurately determined in an environment with less texture, and the robustness is good. In addition, the processing method provided by the embodiment of the present disclosure only needs to extract sparse point features and line features, does not involve dense feature extraction, can determine the plane information in real time, and improves processing efficiency.
[0088] Figure 2 A flow chart showing a method for determining plane information using point features according to an embodiment of the present disclosure is shown;
[0089] FIG. 7( a ) and FIG. 7( b ) are schematic diagrams showing how to determine plane information using point features according to an embodiment of the present disclosure. Figure 2 7(a) and 7(b), in some embodiments, in step S103, determining the plane information in the environment where the electronic device is currently located based at least on the three-dimensional space position information includes:
[0090] S201: selecting three first feature points A1 from the plurality of feature points based at least on three-dimensional spatial position information of the plurality of feature points;
[0091] S202: triangulate the three first feature points A1 to form a first plane;
[0092] S203: Determine a valid plane in the current environment of the electronic device based on the number of second feature points A2 belonging to the first plane, wherein the second feature point A2 is a feature point among the multiple feature points that is different from the first feature point A1, and the valid plane is a plane containing the most second feature points A2.
[0093] Specifically, after detecting that there are multiple feature points in the three-dimensional space of the environment in which the electronic device is currently located, the plane of the environment image is extracted using the Ransac algorithm, as shown in FIG7(a), firstly, three first feature points A1 are selected from the extracted multiple feature points to construct a first plane, and then, as shown in FIG7(b), the number of second feature points A2 belonging to the first plane is counted, and the plane with the largest number of second feature points A2 is determined as the effective plane in the environment image. After obtaining the effective plane, the plane information such as the boundary information and normal vector of the plane can be obtained and stored.
[0094] It can be understood that, since the three-dimensional spatial position information of the above point feature is an absolute spatial coordinate, the second feature point A2 belonging to the first plane is a 3D point.
[0095] Furthermore, other valid planes in the environment where the electronic device is currently located may be determined in a similar manner to obtain all valid planes in the environment where the electronic device is currently located.
[0096] In the above embodiment, only the three-dimensional spatial position information of the point feature is used to determine the plane information in the environment where the electronic device is currently located.
[0097] Figure 8(a) to Figure 8(c) Another schematic diagram of using point features to determine plane information in an embodiment of the present disclosure is shown. Figure 8(a) to Figure 8(c)As shown, when it is determined that there are at least two valid planes in the environment where the electronic device is currently located, and at least two of the valid planes have the same feature point, the method further includes:
[0098] Acquire the same third feature points A3 in the at least two valid planes, wherein a line connecting the same third feature points A3 is not a straight line;
[0099] At least two of the valid planes are merged based on the same third feature point A3.
[0100] Specifically, the first plane shown in FIG8(a) contains eight second feature points A2, and the other first plane shown in FIG8(b) also contains eight second feature points A2. The number of second feature points A2 in these two first planes is the largest, so it can be determined that these two first planes are both valid planes. At the same time, the first plane shown in FIG8(a) and the other first plane shown in FIG8(b) contain the same third feature points A3, and the line connecting these same third feature points A3 is not a straight line, then it is determined that the two valid planes overlap, therefore, the two first planes can be fused to obtain a larger plane as shown in FIG8(c), that is, the two first planes are actually one plane, and the plane information such as the plane boundary information and normal vector after the fusion is recorded and saved, so that the boundary of the plane detection can be expanded.
[0101] If the line connecting these identical third feature points A3 is a straight line, the straight line can be determined as the boundary of the two first planes. Furthermore, the plane information such as the normal vectors of the two first planes can be used to determine information such as the angle between the two first planes, so as to obtain the relationship between different planes.
[0102] Figure 3 A flow chart showing a method for determining plane information using line features according to an embodiment of the present disclosure is shown;
[0103] Fig. 9 FIG. 2 is a schematic diagram showing the use of line features to determine plane information in an embodiment of the present disclosure. Figure 3 and Fig. 9 As shown, in some other embodiments, in step S103, determining the plane information in the environment where the electronic device is currently located based at least on the three-dimensional space position information includes:
[0104] S301: selecting two first feature lines from the plurality of feature lines based at least on the three-dimensional spatial position information of the plurality of feature lines;
[0105] S302: forming a second plane using the two first characteristic lines L1;
[0106] S303: Detecting whether the second plane is valid based on a second characteristic line L2 among the plurality of characteristic lines, wherein the second characteristic line is different from the first characteristic line;
[0107] S304: Determine the valid second plane as a valid plane in the environment where the electronic device is currently located.
[0108] Specifically, after detecting that there are multiple characteristic lines in the three-dimensional space of the current environment of the electronic device, such as Fig. 9 As shown, two first characteristic lines L1 can be used to determine a second plane, and then other characteristic lines among the plurality of characteristic lines, such as the second characteristic line L2, can be used to detect whether the second plane is valid, that is, the third characteristic line can be used to detect whether the second plane is valid, thereby extracting the valid plane in the current environment of the electronic device. Fig. 9 The plane shown can be partially detected using two first characteristic lines L1 on the edges of the desktop, and then the second characteristic line L2 in the middle is used to determine that the second plane is valid.
[0109] In the above embodiment, only the three-dimensional spatial position information of the line feature is used to determine the plane information in the environment where the electronic device is currently located.
[0110] Figure 4 A flow chart of a method for determining plane information using point features and line features in an embodiment of the present disclosure is shown; FIG10(a) and FIG10(b) are schematic diagrams of determining plane information using point features and line features in an embodiment of the present disclosure. In some other embodiments, in step S103, determining plane information in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes:
[0111] S401: determining a third plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of a plurality of feature points;
[0112] S402: Determine a fourth plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature lines;
[0113] S403: Determine whether the third plane and the fourth plane are the same plane;
[0114] S404: If yes, determine a fifth plane formed by merging the third plane and the fourth plane as a valid plane in the current environment of the electronic device.
[0115] Specifically, step S401 can determine the effective plane in the current environment of the electronic device using the extracted point features as in steps S201-S203, as shown in FIG10(a). Step S402 can determine the effective plane in the current environment of the electronic device using the extracted line features as in steps S301-S304, and then determine whether the effective plane determined using the point features and the effective plane determined using the line features are the same plane through step S403. If so, the two effective planes are merged to determine the final effective plane as shown in FIG10(b).
[0116] The above method of jointly determining the effective plane using point features and line features is particularly suitable for smooth planes, such as a clean desktop, which has relatively few texture features and relatively few feature points detected, especially in the edge area. Therefore, only using the three-dimensional spatial position information of the point features to detect the effective plane in the current environment of the electronic device may easily result in missing edges. At the same time, for line features, the edges of the planes are easily blocked, resulting in incomplete edge detection. Therefore, only using line features to detect the effective planes in the current environment of the electronic device may easily result in partial failure to detect the planes or incomplete plane detection.
[0117] Figure 5 and Figure 6 A flowchart showing another method for determining plane information using point features and line features according to an embodiment of the present disclosure; Fig.11 and Fig.12 Another schematic diagram of using point features and line features to determine plane information in an embodiment of the present disclosure is shown. In some other embodiments, in step S103, determining the plane information in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes:
[0118] S501: Determine a first valid plane in an environment where the electronic device is currently located based at least on three-dimensional spatial position information of a plurality of feature points;
[0119] S502: Determine edge information of the first effective plane based on three-dimensional spatial position information of at least one feature line in the first effective plane; or,
[0120] S601: Determine a second valid plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of a plurality of feature lines;
[0121] S602: Expand the second effective plane based on the three-dimensional spatial position information of multiple feature points in the second effective plane to determine edge information of the second effective plane.
[0122] In the above, in step S401 and step S402, the effective plane in the current environment of the electronic device is detected by using point features and line features respectively, and then the two effective planes are compared to obtain the final effective plane. When the number of the third plane and the fourth plane determined by step S401 and step S402 is large, the third plane and the fourth plane need to be compared one by one, which has the problem of large data processing volume. Therefore, in this embodiment, the effective plane in the current environment of the electronic device can be determined by using point features or line features, and then the effective plane is optimized by using line features corresponding to the point features or point features corresponding to the line features, which can greatly reduce the data processing volume and improve the data processing efficiency.
[0123] Steps S501 and S502 first use the three-dimensional spatial position information of the extracted point feature (A) to determine the first effective plane S1, and then use the three-dimensional spatial position information of the line feature (L) to improve and optimize the first effective plane S1, so as to obtain the first effective plane S1 containing the plane edge information (such as the desktop edge); Steps S601 and S602 first use the three-dimensional spatial position information of the extracted line feature (L) to determine the second effective plane S2, and then use the three-dimensional spatial position information of the point feature (A) to improve and optimize the second effective plane S2, and effectively expand the second effective plane S2 ( Fig.12 The dotted line portion in the middle) makes the plane edge of the second effective plane S2 richer.
[0124] As described above, the processing method of the disclosed embodiment can not only use point features to determine the plane information in the environment where the electronic device is currently located, but also use line features to determine the plane information in the environment where the electronic device is currently located. In particular, it can effectively integrate point features and line features. On the one hand, it can quickly detect planes that are difficult to detect, such as the edges of the desktop, to ensure the comprehensiveness and accuracy of plane detection; on the other hand, in areas with relatively few point features, it can stably and effectively determine the effective planes, making the plane detection processing more stable and accurate.
[0125] In some embodiments, after determining the plane information in the environment where the electronic device is currently located in step S103, the method further includes:
[0126] S104: Outputting the to-be-output content of the electronic device to a target plane in the environment at least based on the plane information.
[0127] The plane information is information contained in a valid plane, such as plane edge information, plane normal vector information, plane center information, and the like.
[0128] The content to be output of the electronic device can be a virtual object in an augmented reality (AR) application. After determining the effective plane in the current environment of the electronic device, the effective plane can be selected as the target plane, and the content to be output of the electronic device can be placed on the target plane for presentation, so as to present an augmented reality scene to users wearing augmented reality devices such as AR glasses, thereby improving the user's AR immersion experience.
[0129] Fig.13 FIG. 2 shows a schematic diagram of the structure of the processing device according to an embodiment of the present disclosure. Fig.13 As shown, the present disclosure provides a processing device, including:
[0130] An acquisition module 1301 is used to obtain an environment image of the environment in which the electronic device is currently located, and obtain point features and / or line features on the environment image;
[0131] A first determining module 1302 is used to determine the three-dimensional spatial position information of the point feature and / or line feature;
[0132] The second determining module 1303 is configured to determine plane information in the environment where the electronic device is currently located based at least on the three-dimensional space position information.
[0133] Specifically, the acquisition module 1301 can use the image acquisition device to obtain continuous image frames of the environment in which the electronic device is currently located, and extract point features and / or line features from the continuous image frames; or it can only obtain the current image frame of the environment in which the electronic device is currently located, and extract point features and / or line features from the current frame image.
[0134] The image acquisition device can be a monocular camera (such as a monocular RGB camera) or a multi-eye camera (such as a binocular fisheye camera on AR glasses), which is not specifically limited in the embodiments of the present disclosure.
[0135] After the point features and / or line features on the environmental image are extracted by the acquisition module 1301, the three-dimensional spatial position information of the above-mentioned point features and / or line features is calculated by the first determination module 1302, and then the plane information in the environmental image is determined by the second determination module 1303, thereby determining the plane information in the environment where the electronic device is currently located.
[0136] The processing device provided by the embodiment of the present disclosure uses a general monocular or multi-camera to collect an environmental image of the environment in which the electronic device is currently located, and extracts point features and / or line features therefrom, and determines the plane information in the environment in which the electronic device is currently located based on the extracted point features and / or line features. The operation is convenient, no depth camera is required, and the versatility is strong. At the same time, the plane information in the environment in which the electronic device is currently located can be accurately and comprehensively detected, and the plane information can also be accurately determined in an environment with less texture, and the robustness is good. In addition, the processing device provided by the embodiment of the present disclosure only needs to extract sparse point features and line features, and does not involve dense feature extraction, and can determine the plane information in real time, thereby improving processing efficiency.
[0137] The three-dimensional spatial position information of the point features and / or line features determined by the first determination module 1302 is the three-dimensional absolute spatial coordinates of the feature points and / or feature lines. Due to the stability and accuracy of the absolute spatial coordinates, the accuracy of the plane information determination can be guaranteed.
[0138] In some embodiments, the acquisition module 1301 includes a first extraction unit and / or a second extraction unit, the first extraction unit is used to extract corner points in the environmental image whose pixel changes are greater than a first preset threshold, and use the corner points as feature points to obtain the point features; the second extraction unit is used to extract pixels in the environmental image whose gradient changes are greater than a second preset threshold, and use the pixels as feature lines to obtain the line features.
[0139] In some embodiments, when the environment image of the current environment of the electronic device obtained by the acquisition module 1301 is a monocular image captured by a monocular camera, the first determination module 1302 is specifically used to:
[0140] Solving the point features and / or line features to obtain relative spatial coordinates of feature points representing the point features and / or feature lines representing the line features, wherein the point features and / or line features are extracted from the environment image captured by a monocular camera;
[0141] The absolute spatial coordinates of the characteristic points and / or characteristic lines are determined using an inertial measurement unit.
[0142] In other embodiments, when the environment image of the current environment of the electronic device obtained by the acquisition module 1301 is a multi-eye image captured by a multi-eye camera, the first determination module 1302 is specifically used to:
[0143] The point features and / or line features are solved to obtain the absolute spatial coordinates of feature points representing the point features and / or feature lines representing the line features, wherein the point features and / or line features are extracted from the environment image collected by a binocular camera.
[0144] In some embodiments, the second determination module 1303 includes a first determination subunit, which is used to determine the plane information in the environment where the electronic device is currently located using the three-dimensional spatial position information of the point feature determined by the first determination module 1302, specifically for:
[0145] Selecting three first feature points A1 from the plurality of feature points based at least on the three-dimensional spatial position information of the plurality of feature points;
[0146] Triangulating the three first feature points A1 to form a first plane;
[0147] Based on the number of second feature points A2 belonging to the first plane, determine the effective plane in the current environment of the electronic device, wherein the second feature point A2 is a feature point different from the first feature point A1 among the multiple feature points, and the effective plane is the plane containing the most second feature points A2.
[0148] In some embodiments, when the first determining subunit determines that there are at least two valid planes in the environment where the electronic device is currently located, and at least two of the valid planes have the same feature point, the first determining subunit is further used to:
[0149] Acquire the same third feature points A3 in the at least two valid planes, wherein a line connecting the same third feature points A3 is not a straight line;
[0150] At least two of the valid planes are merged based on the same third feature point A3.
[0151] In some embodiments, the second determination module 1303 includes a second determination subunit, which is used to determine the plane information in the environment where the electronic device is currently located using the three-dimensional spatial position information of the line feature determined by the first determination module 1302, specifically for:
[0152] Selecting two first feature lines from the plurality of feature lines based at least on the three-dimensional spatial position information of the plurality of feature lines;
[0153] Using the two first characteristic lines L1 to form a second plane;
[0154] Detecting whether the second plane is valid based on a second characteristic line L2 among the plurality of characteristic lines, wherein the second characteristic line is different from the first characteristic line;
[0155] The valid second plane is determined as a valid plane in the environment where the electronic device is currently located.
[0156] In some embodiments, the second determination module 1303 includes a third determination subunit, which is used to jointly determine the plane information in the current environment of the electronic device using the three-dimensional spatial position information of the point features and the line features determined by the first determination module 1302, specifically for:
[0157] Determine a third plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points;
[0158] Determining a fourth plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines;
[0159] determining whether the third plane and the fourth plane are the same plane;
[0160] If so, a fifth plane formed by merging the third plane and the fourth plane is determined as a valid plane in the environment where the electronic device is currently located.
[0161] In some other embodiments, the third determining subunit is used to:
[0162] Determine a first valid plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points;
[0163] Determine edge information of the first effective plane based on three-dimensional spatial position information of at least one feature line in the first effective plane; or,
[0164] Determine a second valid plane in the environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines;
[0165] The second effective plane is expanded based on the three-dimensional spatial position information of a plurality of feature points in the second effective plane to determine edge information of the second effective plane.
[0166] In some embodiments, the processing device further comprises:
[0167] An output module is used to output the to-be-output content of the electronic device to a target plane in the environment based at least on the plane information.
[0168] The processing device provided in the embodiment of the present disclosure corresponds to the processing method in the above-mentioned embodiment. Based on the above-mentioned processing method, those skilled in the art can understand the specific implementation manner of the processing device in the embodiment of the present disclosure and its various variations. Any optional items in the processing method embodiment are also applicable to the processing device, which will not be repeated here.
[0169] An embodiment of the present disclosure further provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store computer-executable instructions, and the processor implements the above-mentioned processing method when executing the computer-executable instructions.
[0170] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0171] The memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0172] The embodiment of the present disclosure further provides a computer-readable storage medium on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the above-mentioned processing method is implemented.
[0173] The above embodiments are only exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. The protection scope of the present disclosure is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present disclosure within the essence and protection scope of the present disclosure, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present disclosure.
Claims
1. A processing method comprising: Obtaining an environmental image of the environment in which the electronic device is currently located, and acquiring point features on the environmental image; Determining the three-dimensional spatial position information of the point feature; Determining plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes: selecting three first feature points from the plurality of feature points based at least on three-dimensional spatial position information of the plurality of feature points; triangulate the three first feature points to form a first plane; Based on the number of second feature points belonging to the first plane, determine the effective plane in the environment where the electronic device is currently located, wherein the second feature point is a feature point different from the first feature point among the multiple feature points, and the effective plane is the plane containing the most second feature points.
2. The method according to claim 1, further comprising: Outputting the to-be-output content of the electronic device to a target plane within the environment based at least on the plane information.
3. The method according to claim 1, wherein Acquiring point features on the environment image includes: Corner points whose pixel changes are greater than a first preset threshold value in the environment image are extracted, and the corner points are used as feature points to obtain the point features.
4. The method according to claim 1, wherein Determining the three-dimensional spatial position information of the point feature includes: Solving the point features to obtain relative spatial coordinates of feature points representing the point features, wherein the point features are extracted from the environment image captured by a monocular camera; The absolute spatial coordinates of the feature points are determined using an inertial measurement unit.
5. The method according to claim 1, wherein Determining the three-dimensional spatial position information of the point feature includes: The point features are solved to obtain absolute spatial coordinates of feature points representing the point features, wherein the point features are extracted from the environment image collected by a binocular camera.
6. A treatment method comprising: Obtaining an environmental image of the environment in which the electronic device is currently located, and acquiring line features on the environmental image; Determining three-dimensional spatial position information of the line feature; Determining plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes: selecting two first feature lines from the plurality of feature lines based at least on three-dimensional spatial position information of the plurality of feature lines; forming a second plane using the two first characteristic lines; detecting whether the second plane is valid based on a second characteristic line among the plurality of characteristic lines, wherein the second characteristic line is different from the first characteristic line; The valid second plane is determined as a valid plane in the environment where the electronic device is currently located.
7. A treatment method comprising: Obtaining an environmental image of the environment in which the electronic device is currently located, and acquiring point features and line features on the environmental image; Determining three-dimensional spatial position information of the point feature and the line feature; Determining plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes: determining a third plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points; determining a fourth plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines; determining whether the third plane and the fourth plane are the same plane; If so, a fifth plane formed by fusing the third plane and the fourth plane is determined as a valid plane in the environment where the electronic device is currently located.
8. A treatment method comprising: Obtaining an environmental image of the environment in which the electronic device is currently located, and acquiring point features and line features on the environmental image; Determining three-dimensional spatial position information of the point feature and the line feature; Determining plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information includes: Determining a first valid plane within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points; Determine edge information of the first effective plane based on three-dimensional spatial position information of at least one characteristic line within the first effective plane; or, determining a second valid plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines; The second effective plane is expanded based on the three-dimensional spatial position information of a plurality of feature points in the second effective plane to determine edge information of the second effective plane.
9. A processing device comprising: An acquisition module, configured to obtain an environmental image of the environment in which the electronic device is currently located, and obtain point features on the environmental image; A first determining module, configured to determine the three-dimensional spatial position information of the point feature; a second determining module, configured to determine plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information; The second determining module includes a first determining subunit, and the first determining subunit is configured to: selecting three first feature points from the plurality of feature points based at least on three-dimensional spatial position information of the plurality of feature points; triangulate the three first feature points to form a first plane; Based on the number of second feature points belonging to the first plane, determine the effective plane in the environment where the electronic device is currently located, wherein the second feature point is a feature point different from the first feature point among the multiple feature points, and the effective plane is the plane containing the most second feature points.
10. A processing device comprising: An acquisition module, configured to obtain an environmental image of the environment in which the electronic device is currently located, and obtain line features on the environmental image; A first determining module, configured to determine the three-dimensional spatial position information of the line feature; a second determining module, configured to determine plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information; The second determining module includes a second determining subunit, and the second determining subunit is configured to: selecting two first feature lines from the plurality of feature lines based at least on three-dimensional spatial position information of the plurality of feature lines; forming a second plane using the two first characteristic lines; detecting whether the second plane is valid based on a second characteristic line among the plurality of characteristic lines, wherein the second characteristic line is different from the first characteristic line; The valid second plane is determined as a valid plane in the environment where the electronic device is currently located.
11. A processing device comprising: An acquisition module, configured to obtain an environmental image of the environment in which the electronic device is currently located, and obtain point features and line features on the environmental image; A first determining module, configured to determine the three-dimensional spatial position information of the point feature and the line feature; a second determining module, configured to determine plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information; The second determining module includes a third determining subunit, and the third determining subunit is configured to: determining a third plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points; determining a fourth plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines; determining whether the third plane and the fourth plane are the same plane; If so, a fifth plane formed by fusing the third plane and the fourth plane is determined as a valid plane in the environment where the electronic device is currently located.
12. A processing device comprising: An acquisition module, configured to obtain an environmental image of the environment in which the electronic device is currently located, and obtain point features and line features on the environmental image; A first determining module, configured to determine the three-dimensional spatial position information of the point feature and the line feature; a second determining module, configured to determine plane information within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information; The second determining module includes a third determining subunit, and the third determining subunit is further configured to: Determining a first valid plane within an environment where the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of feature points; determining edge information of the first effective plane based on three-dimensional spatial position information of at least one characteristic line within the first effective plane; or, determining a second valid plane within the environment in which the electronic device is currently located based at least on the three-dimensional spatial position information of the plurality of characteristic lines; The second effective plane is expanded based on the three-dimensional spatial position information of a plurality of feature points in the second effective plane to determine edge information of the second effective plane.