Plane detection method and device, electronic equipment and storage medium

Abstract

The invention provides a plane detection method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving a plane detection instruction, and carrying out the plane detection of a current detection region of the electronic equipment in response to the plane detection instruction, and obtaining plane detection data; determining whether plane detection succeeds or not according to the plane detection data; under the condition that the plane detection is not successful, judging a plane detection failure type according to the plane detection data; and detecting image data under the condition that a user executes a gesture operation in the current detection area, and executing a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain a target plane.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a plane detection method, device, electronic device and storage medium. Background Art

[0002] With the popularization of artificial intelligence, computer vision technology continues to develop and is widely used in AR (Augmented Reality) or VR (Virtual Reality) products, drones, robots and autonomous driving. Structured plane detection is a key technology in the above computer vision technology. The current structured plane detection method is based on SLAM (Simultaneous Localization And Mapping) to perceive environmental data and generate, add and update map points. The plane method detects and updates the plane based on the map points, but it is limited by the SLAM hardware solution and software accuracy, and there is a problem of plane detection failure. Summary of the invention

[0003] Embodiments of the present application provide a plane detection method, device, electronic device, and storage medium.

[0004] According to the first aspect of the present application, a plane detection method is provided, which includes: receiving a plane detection instruction, and performing plane detection on a current detection area of ​​an electronic device in response to the plane detection instruction to obtain plane detection data; determining whether the plane detection is successful based on the plane detection data; if the plane detection is unsuccessful, judging the type of plane detection failure based on the plane detection data; detecting image data when a user performs a gesture operation in the current detection area, and performing a plane construction operation corresponding to the type of plane detection failure based on the image data and the plane detection data to obtain a target plane.

[0005] According to one embodiment of the present application, the plane detection data includes map point data or includes map point data and plane data; accordingly, the plane detection is performed on the current detection area of ​​the electronic device to obtain the plane detection data, including: acquiring sensor data for the current detection area from at least one sensor of the electronic device, the sensor data including acceleration data, angular velocity data and image data; determining the posture data of the electronic device according to the acceleration data, the angular velocity data and the image data; acquiring feature points in the current frame according to the image data, and matching the feature points in the current frame with the feature points of the previous frame to obtain a feature point matching relationship; generating map point data according to the posture data and the feature point matching relationship; and generating plane data according to the map point data when the map point data meets the set conditions.

[0006] According to one embodiment of the present application, determining whether the plane detection is successful based on the plane detection data includes: determining that the plane detection has failed when the plane detection data does not include plane data; determining that the plane detection has failed when the plane detection data includes plane data and the plane shown by the plane data is not unique; determining that the plane detection has succeeded when the plane detection data includes plane data and the plane shown by the plane data is unique and satisfies a set plane condition.

[0007] According to an embodiment of the present application, the plane detection failure type includes at least one of the following: a first type, a second type and a third type; correspondingly, judging the plane detection failure type based on the plane detection data includes: when the plane detection data does not include plane data, detecting the number of map points in the map point data, and when the number of map points is less than the set number of map points, determining that the plane detection failure type is the first type, and when the number of map points is greater than the set number of map points, determining that the plane detection failure type is the second type; when the plane data includes map point data and plane data, detecting the number of planes in the plane data, and when the number of planes is not unique, determining that the plane detection failure type is the third type.

[0008] According to one embodiment of the present application, when the plane detection failure type is the first type or the second type, the plane construction operation corresponding to the plane detection failure type is performed based on the image data and the plane detection data to obtain the target plane, including: determining a gesture motion trajectory based on the image data; constructing an initial plane based on the gesture motion trajectory; and constructing the target plane based on the initial plane and the plane detection data.

[0009] According to an embodiment of the present application, when the plane detection failure type is the second type, constructing the target plane based on the initial plane and the plane detection data includes: associating the map point data with the initial plane to obtain a plane association result; adjusting the initial plane according to the plane association result to obtain the target plane.

[0010] According to an embodiment of the present application, when the plane detection failure type is the third type, the plane construction operation corresponding to the plane detection failure type is performed based on the image data and the plane detection data to obtain the target plane, including: performing interface element calibration on multiple planes in the plane data based on preset gesture interaction interface elements to obtain interface element calibration results; performing gesture recognition on the image data based on a preset gesture interaction type library to obtain gesture recognition results; and merging the planes in the plane data based on the gesture recognition results and the interface element calibration results to obtain the target plane.

[0011] According to a second aspect of the present application, a plane detection device is provided, which includes: a detection module, which is used to receive a plane detection instruction, and perform plane detection on a current detection area of ​​an electronic device in response to the plane detection instruction to obtain plane detection data; a determination module, which is used to determine whether the plane detection is successful based on the plane detection data; a judgment module, which is used to judge the type of plane detection failure based on the plane detection data when the plane detection is unsuccessful; and a construction module, which is used to detect image data when a user performs a gesture operation in the current detection area, and perform a plane construction operation corresponding to the type of plane detection failure based on the image data and the plane detection data to obtain a target plane.

[0012] According to a third aspect of the present application, an electronic device is provided, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present application.

[0016] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.

[0017] The method of the embodiment of the present application receives a plane detection instruction, and performs plane detection on the current detection area of ​​the electronic device in response to the plane detection instruction to obtain plane detection data; determines whether the plane detection is successful based on the plane detection data; if the plane detection is unsuccessful, determines the type of plane detection failure based on the plane detection data; detects image data when the user performs gesture operations in the current detection area, and performs a plane construction operation corresponding to the type of plane detection failure based on the image data and the plane detection data to obtain a target plane. The plane is constructed by combining plane detection and gesture operation detection to complete the plane detection, thereby improving the probability, completeness and speed of successful plane detection.

[0018] It should be understood that the teachings of the present application are not required to achieve all of the beneficial effects described above, but specific technical solutions can achieve specific technical effects, and other embodiments of the present application can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0020] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0021] Figure 1 A schematic diagram of the implementation process of the plane detection method provided in the embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of the implementation flow of the plane verification operation of the plane detection method provided in an embodiment of the present application is shown;

[0023] Figure 3 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 1 ;

[0024] Figure 4 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 2 ;

[0025] Figure 5 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 3 ;

[0026] Figure 6 A schematic diagram of the structure of a plane detection device provided in an embodiment of the present application is shown;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0029] Figure 1 A schematic diagram of the implementation flow of the plane detection method provided in an embodiment of the present application is shown.

[0030] refer to Figure 1 , an embodiment of the present application provides a plane detection method, the method comprising:

[0031] Operation 101 : receiving a plane detection instruction, and performing plane detection on a current detection area of ​​an electronic device in response to the plane detection instruction to obtain plane detection data.

[0032] The executor of this application is an electronic device, which can be an artificial intelligence device with a display function, preferably a smart wearable device, such as AR glasses. When using an artificial intelligence device, in order to achieve a more realistic scene interaction and object placement effect, plane detection is very important. When receiving a plane detection instruction, the electronic device starts to perform plane detection on the currently displayed area, also known as the current detection area.

[0033] In one embodiment of the present application, the plane detection instruction is received when the electronic device is started, the display area of ​​the electronic device changes, or a plane detection instruction is received from the user through a key, voice, etc. It should be noted that the present application does not specifically limit the method of receiving the plane detection instruction. As long as a plane detection is required, it can be regarded as receiving the plane detection instruction. For example, when the electronic device detects that a plane detection is required, it can also be regarded as receiving the plane detection instruction.

[0034] In one embodiment of the present application, plane detection is performed based on the SLAM plane method, and the plane detection data includes map point data or includes map point data and plane data. The specific plane detection process includes: obtaining sensor data for a current detection area from at least one sensor of the electronic device, the sensor data including acceleration data, angular velocity data and image data; determining the posture data of the electronic device based on the acceleration data, angular velocity data and image data; obtaining feature points in the current frame based on the image data, and matching the feature points in the current frame with the feature points of the previous frame to obtain a feature point matching relationship; generating map point data based on the posture data and the feature point matching relationship; and generating plane data based on the map point data when the map point data meets the set conditions.

[0035] In the process of plane detection based on SLAM method, it is necessary to generate map points first and then construct the plane. However, due to the influence of environment and motion constraints (such as insufficient motion of electronic equipment, etc.), there may be too few map points or no detection in the plane detection process, resulting in the failure to generate the plane. Therefore, the plane detection data may only include map point data or both map point data and plane data. Among them, map point data includes detected map points, and plane data includes detected planes.

[0036] Electronic devices usually have multiple sensors built in. Preferably, the multiple sensors may include a camera and an IMU (Inertial Measurement Unit). Image data can be obtained through the camera, and acceleration data and angular velocity data can be obtained based on the IMU. After that, the feature points of the current frame can be extracted from the image data based on the image processing algorithm, and the feature point matching relationship can be obtained based on the matching of the feature points of the current frame and the previous frame. The camera's posture (i.e., posture data) can be analyzed in real time based on the acceleration data, angular velocity data, and image data. Finally, triangulation can be performed based on the camera coordinate system according to the posture data and the feature point matching data. After triangulation, the position of the feature points in the current frame in the camera coordinate system is calculated, and the feature points in the current frame can be converted into map points to obtain map point data including map points. After obtaining the map point data, if the map point data meets the set conditions, a plane is constructed according to the map point data to obtain plane data including the plane, wherein the set conditions may be the construction conditions that meet the plane, such as the number of map points in the map point data that meet the conditions for constructing the plane, etc. The application does not specifically limit the construction conditions that meet the plane.

[0037] Operation 102 : determining whether the plane detection is successful according to the plane detection data.

[0038] In one embodiment of the present application, determining whether the plane detection is successful is based on the plane detection data, including: when the plane detection data does not include plane data, determining that the plane detection has failed; when the plane detection data includes plane data and the plane shown by the plane data is not unique, determining that the plane detection has failed; when the plane detection data includes plane data and the plane shown by the plane data is unique and satisfies the set plane conditions, determining that the plane detection is successful.

[0039] Affected by the environmental texture and device motion constraints, for example, when the environmental texture of the current detection area is weak or the device motion is insufficient, plane detection may fail. Plane detection failures may include situations where the plane is not generated, multiple planes are generated, etc. Therefore, situations where the plane detection data does not include plane data and situations where the plane data shows that the plane is not unique can be regarded as plane detection failures. When the plane is unique and meets the set plane conditions, the plane detection is deemed to be successful, where the set plane conditions can be configured according to the actual scene requirements, and this application does not make specific limitations.

[0040] Operation 103 : if the plane detection is unsuccessful, determine the type of plane detection failure according to the plane detection data.

[0041] In actual application scenarios, according to at least one situation of plane failure, such as weak regional texture, insufficient motion, etc., plane failure can be defined as at least one type to obtain at least one type of plane detection failure.

[0042] Operation 104 , detecting image data when the user performs a gesture operation in the current detection area, and performing a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain a target plane.

[0043] In the event of a plane detection failure, in order to ensure the complete operation of the electronic device, the plane is constructed by receiving user gesture operations to complete the plane detection operation. After determining that the plane detection has failed, the electronic device detects the user's gesture operation in the current detection area in real time, and acquires image data in real time when the user's gesture operation is detected. The image data includes all images of the user from the start of the gesture operation to the completion of the gesture operation. The image data can be video data or multi-frame image data distributed by frame, which is not specifically limited in this application.

[0044] After obtaining the image data, the plane construction operation corresponding to the type of plane detection failure is performed by analyzing the gesture operation in the image data and combining it with the plane detection data obtained during the initial detection, so as to obtain the target plane and complete the plane detection.

[0045] In this way, the embodiment of the present application performs initial plane detection through the SLAM method, and constructs the plane in combination with gesture operation detection according to different situations of plane failure, thereby improving the probability, completeness and speed of successful plane detection.

[0046] Figure 2 A schematic diagram of the implementation flow of the plane verification operation of the plane detection method provided in an embodiment of the present application is shown.

[0047] refer to Figure 2 In one embodiment of the present application, the plane detection failure type includes at least one of the first type, the second type and the third type. The above operation 103, when the plane detection is unsuccessful, determines the plane detection failure type according to the plane detection data, including:

[0048] Operation 201, when the plane detection data does not include plane data, detect the number of map points in the map point data, when the number of map points is less than the set number of map points, determine the plane detection failure type as the first type, when the number of map points is greater than the set number of map points, determine the plane detection failure type as the second type.

[0049] When the current detection area is a weak texture area, since the weak texture area lacks obvious visual features, such as corner points, edge points, etc., and these points are the source of map points in the SLAM plane method, therefore, when the current detection area is a weak texture area, there may be too few map points or no detection, and the plane cannot be generated. For the above-mentioned situation where plane detection fails due to encountering a weak texture area, there are very few map points. Therefore, a condition can be configured to measure whether the type of plane detection failure is encountering a weak texture area, that is, based on the comparison between the number of map points in the map point data and the set number of map points, it is determined whether it is encountering a weak texture area. When the number of map points is less than the set number of map points, the current plane detection failure type is judged to be encountering a weak texture area. Among them, the first type refers to encountering a weak texture area.

[0050] In addition to insufficient map points caused by environmental factors, there is also the situation where the number of map points is too small to complete plane detection due to insufficient movement of the device. When insufficient movement leads to too few map points and the plane cannot be generated, if the map points are reserved through continuous movement, the detection speed of the plane may be reduced, affecting the user experience. Therefore, when the electronic device does not move sufficiently, it is also regarded as a plane detection failure. Insufficient movement of the electronic device can be used as another type of plane detection failure, that is, the second type. In the case of insufficient movement of the electronic device, there are more map points than in the case of weak texture areas. Therefore, the situation where the number of map points is greater than the set number of map points can be configured as a standard for judging the plane detection failure type as the second type. When it is determined that the number of map points in the map data is greater than the set number of map points, the plane detection failure type is judged to be the second type.

[0051] In one embodiment of the present application, the number of map points can be set according to actual needs. As long as it can meet the distinction between the two situations of encountering weak texture areas and insufficient movement of electronic devices, the present application does not make any limitation. For example, the number of map points can be configured to 0, 1, 2, etc.

[0052] Operation 202 : in the case where the plane data includes map point data and plane data, detecting the number of planes in the plane data, and in the case where the number of planes is not unique, determining that the plane detection failure type is the third type.

[0053] During the movement of an electronic device, the motion trajectory may not completely cover the current detection area. In this case, multiple planes may be generated. Therefore, whether the plane in the plane data is unique can be used as a criterion for judging whether the type of plane detection failure is that the motion trajectory of the electronic device does not cover the entire current detection area. Among them, the motion trajectory of the electronic device does not cover the entire current detection area, which is the third type.

[0054] Figure 3 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 1 .

[0055] refer to Figure 3 In one embodiment of the present application, when the plane detection failure type is the first type or the second type, in the above operation 104, a plane construction operation corresponding to the plane detection failure type is performed according to the image data and the plane detection data to obtain a target plane, including:

[0056] Operation 301, determining a gesture motion trajectory according to the image data;

[0057] Operation 302, constructing an initial plane according to the gesture motion trajectory;

[0058] Operation 303 : constructing a target plane according to the initial plane and the plane detection data.

[0059] For the cases where the plane detection failure type is the first type and the second type, the plane of the current detection area is not detected, so it is necessary to construct the plane. First, the user's gesture motion trajectory is determined based on the image data containing the user's gesture operation, that is, the user's hand motion trajectory. The specific operation of determining the gesture motion trajectory may include: inputting multiple frames of images in the image data into a pre-trained model in sequence, and the model sequentially outputs the hand area and the hand posture in the camera of each frame of the image data, and the hand posture is combined by frame to obtain the hand motion trajectory. Among them, the training method of the pre-trained model can refer to the training method of the conventional gesture recognition model, which will not be repeated here.

[0060] The gesture motion trajectory includes multiple trajectory points. Since it is the posture under the camera, the coordinate system used is the camera's coordinate system, which is the same as the coordinate system of the map points used by SLAM. Therefore, the plane can be directly constructed based on multiple trajectory points to merge with the plane detection data to obtain the target plane.

[0061] In one embodiment of the present application, constructing the initial plane can be understood as calculating the plane equation of the initial plane. The plane equation of the initial plane is the expression of the initial plane. Specifically, the plane expression of the initial plane can be:

[0062] in, is the unit normal vector of the initial plane, and d is the distance from the origin to the initial plane.

[0063] In one embodiment of the present application, the plane equation of the initial plane may be calculated in the following manner:

[0064] 1) Calculate the distance z from each trajectory point in the gesture motion trajectory to the initial plane, where: p T The transpose of the vector corresponding to each trajectory point, σ is the plane observation noise, and the σ value is 0 in the ideal state;

[0065] 2) Construct the following least squares problem based on the trajectory points:

[0066] Among them, π * To find the optimal solution, use numerical methods (such as gradient descent, Newton's method, etc.) to optimize π to minimize the sum of the squares of the distances from all trajectory points to the plane to solve the least squares problem. The optimal π is obtained. * This is the plane equation of the initial plane.

[0067] After the initial plane is constructed, the target plane is also constructed based on the initial plane and the plane detection data.

[0068] In this embodiment of the present application, for the case where the plane detection failure type is the first type, there are almost no map points in the plane detection data and they can be ignored, and the initial plane is directly used as the final target plane. For the case where the plane detection failure type is the second type, there are a small number of map points in the plane detection data, and the initial plane and the plane detection data need to be fused to obtain the final target plane.

[0069] Figure 4 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 2 .

[0070] refer to Figure 4 In one embodiment of the present application, when the plane detection failure type is the second detection type, the above operation 303, constructing the target plane according to the initial plane and the plane detection data, includes:

[0071] Operation 401, associating the map point data with the initial plane to obtain a plane association result;

[0072] Operation 402 : adjusting the initial plane according to the plane association result to obtain the target plane.

[0073] The map points in the map point data are associated with the initial plane through common feature matching, spatial transformation, etc. to obtain the plane association result. Based on the plane association result, the position, direction, size and other parameters of the initial plane can be optimized and the initial plane can be fused with the map points to obtain the target plane.

[0074] Figure 5 The following is a schematic diagram showing the implementation process of the plane construction operation of the plane detection method provided in the embodiment of the present application. Figure 3 .

[0075] refer to Figure 5 In one embodiment of the present application, when the plane detection failure type is the third type, the above operation 105 performs a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain the target plane, including:

[0076] Operation 501, performing interface element calibration on multiple planes in the plane data based on preset gesture interaction interface elements to obtain interface element calibration results;

[0077] Operation 502, performing gesture recognition on the image data based on a preset gesture interaction type library to obtain a gesture recognition result;

[0078] Operation 503 , according to the gesture recognition result and the interface element calibration result, the planes in the plane data are merged to obtain a target plane.

[0079] The electronic device has a set of built-in interface elements for user gesture interaction. The interface elements can be buttons, sliders, arrows, markers or other interactive user interface components. They are used to visually represent planes, are associated with each plane, and allow users to interact with it through gestures. The electronic device also has a built-in gesture interaction type library, which defines all gesture types that the electronic device can recognize (such as sliding, merging, rotating, etc.) and their respective meanings and parameters.

[0080] After obtaining the image data, computer vision technology (such as machine learning models, template matching, etc.) is used to analyze the image data in real time to detect and recognize the user's gestures, and then the recognized gestures are matched with the gesture types in the gesture interaction type library to obtain the gesture recognition results. Finally, according to the gesture recognition results, the calibration results of the interface elements related to the gestures are found, the planes that need to be merged are determined and merged, and the construction of the target plane is completed.

[0081] In order to further illustrate the technical solution of the present application, a specific application example is provided below.

[0082] In this specific application example of the present application, a plane detection method of the present application is also called a SLAM plane method integrating gesture interaction. The electronic device is mainly AR glasses, and the AR glasses support gesture recognition and interaction. It is mainly aimed at the scenario where the user wears or holds AR glasses and moves in space. The plane detection failure type is encountered in a weak texture area, the electronic device does not move sufficiently, and the electronic device movement trajectory does not cover the entire current detection area. The method of constructing the target plane is described as follows:

[0083] 1) In the case of weak texture areas, initial plane detection is performed based on SLAM to obtain plane detection data with sparse map points or no map points. In this case, the user's gestures in the current detection area are detected in real time, the gesture trajectory is obtained by recognizing the user's gestures, and a plane (i.e., the target plane) is constructed based on the gesture trajectory in the real-time display interface of the current detection area.

[0084] 2) In the case of insufficient movement of electronic devices, initial plane detection is performed based on SLAM, and the number of map points generated is small, making it difficult to generate a plane. In this case, the user is prompted to draw a trajectory based on the map points, and the user's gestures in the current detection area are detected in real time. The gesture trajectory is obtained by recognizing the user's gestures, and the initial plane is constructed based on the gesture trajectory. SLAM optimizes the initial plane based on the map points to obtain the target plane.

[0085] 3) In case the device motion trajectory does not cover the entire current detection area, multiple fragmented planes are detected based on SLAM map points, and the target plane is obtained by merging multiple fragmented planes by detecting the user's gesture interaction operations.

[0086] Therefore, this specific application example of the embodiment of the present application, by combining SLAM with gesture recognition and interaction, improves the robustness of the SLAM-based plane detection method based on gesture recognition and interaction, solves the problem that SLAM cannot generate planes in weak texture areas, improves the generation speed of SLAM planes when the device movement is insufficient, and improves the integrity of the SLAM plane when the device motion trajectory is incomplete, thereby improving the user experience.

[0087] It should be noted that the description of this specific application example of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 5 The present invention can be understood by referring to the description of any one of the accompanying drawings.

[0088] Figure 6 A schematic diagram of the structure of a plane detection device provided in an embodiment of the present application is shown.

[0089] refer to Figure 6 Based on the above-mentioned plane detection method, an embodiment of the present application also provides a plane detection device, which includes:

[0090] In one embodiment of the present application, a detection module 601 is used to receive a plane detection instruction, and perform a plane detection on a current detection area of ​​the electronic device in response to the plane detection instruction to obtain plane detection data; a determination module 602 is used to determine whether the plane detection is successful based on the plane detection data; a judgment module 603 is used to judge the type of plane detection failure based on the plane detection data when the plane detection is unsuccessful; a construction module 604 is used to detect image data when a user performs a gesture operation in the current detection area, and perform a plane construction operation corresponding to the type of plane detection failure based on the image data and the plane detection data to obtain a target plane.

[0091] In one embodiment of the present application, the plane detection data includes map point data or includes map point data and plane data; accordingly, the detection module 601 includes: an acquisition submodule, used to acquire sensor data for a current detection area from at least one sensor of the electronic device, the sensor data including acceleration data, angular velocity data and image data; a first determination submodule, used to determine the posture data of the electronic device based on the acceleration data, angular velocity data and image data; a matching submodule, used to acquire feature points in the current frame based on the image data, and match the feature points in the current frame with the feature points of the previous frame to obtain a feature point matching relationship; a first generation submodule, used to generate map point data based on the posture data and the feature point matching relationship; a second generation submodule, used to generate plane data based on the map point data when the map point data meets the set conditions.

[0092] In one embodiment of the present application, the determination module 602 includes: a second determination submodule, used to determine that the plane detection fails when the plane detection data does not include plane data; a third determination submodule, used to determine that the plane detection fails when the plane detection data includes plane data and the plane shown by the plane data is not unique; a fourth determination submodule, used to determine that the plane detection is successful when the plane detection data includes plane data and the plane shown by the plane data is unique and satisfies the set plane conditions.

[0093] In one embodiment of the present application, the plane detection failure type includes at least one of the following: a first type, a second type and a third type; accordingly, the judgment module 603 includes: a fifth determination submodule, for detecting the number of map points in the map point data when the plane detection data does not include plane data, and determining that the plane detection failure type is the first type when the number of map points is less than the set number of map points, and determining that the plane detection failure type is the second type when the number of map points is greater than the set number of map points; a sixth determination submodule, for detecting the number of planes in the plane data when the plane data includes map point data and plane data, and determining that the plane detection failure type is the third type when the number of planes is not unique.

[0094] In one embodiment of the present application, when the plane detection failure type is the first type or the second type, the construction module 604 includes: a seventh determination submodule, used to determine the gesture motion trajectory based on the image data; a first construction submodule, used to construct an initial plane based on the gesture motion trajectory; and a second construction submodule, used to construct a target plane based on the initial plane and the plane detection data.

[0095] In one embodiment of the present application, when the plane detection failure type is the second type, the second construction submodule includes: an association unit, used to associate map point data with the initial plane to obtain a plane association result; an adjustment unit, used to adjust the initial plane according to the plane association result to obtain a target plane.

[0096] In one embodiment of the present application, when the plane detection failure type is the third type, the construction module 604 includes: a calibration submodule, which is used to perform interface element calibration on multiple planes in the plane data based on preset gesture interaction interface elements to obtain interface element calibration results; an identification submodule, which is used to perform gesture recognition on the image data based on a preset gesture interaction type library to obtain gesture recognition results; and a merging submodule, which is used to merge the planes in the plane data according to the gesture recognition results and the interface element calibration results to obtain the target plane.

[0097] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 5 The present invention can be understood by referring to the description of any one of the accompanying drawings.

[0098] According to an embodiment of the present application, the present application also provides an electronic device and a non-transitory computer-readable storage medium.

[0099] Figure 7 A schematic block diagram of an example electronic device 70 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0100] like Figure 7 As shown, the electronic device 70 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 70 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0101] A number of components in the electronic device 70 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 70 to exchange data / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a plane detection method. For example, in some embodiments, the plane detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 70 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the plane detection method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the plane detection method in any other appropriate manner (e.g., by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying data to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0110] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A plane detection method, characterized in that: The method comprises: receiving a plane detection instruction, and performing plane detection on a current detection area of ​​the electronic device in response to the plane detection instruction to obtain plane detection data; Determining whether the plane detection is successful according to the plane detection data; In the case where the plane detection is unsuccessful, determining the type of plane detection failure according to the plane detection data; Image data of a user performing a gesture operation in the current detection area is detected, and a plane construction operation corresponding to a plane detection failure type is performed according to the image data and the plane detection data to obtain a target plane.

2. The method according to claim 1, characterized in that The plane detection data includes map point data or map point data and plane data; accordingly, The performing plane detection on the current detection area of ​​the electronic device to obtain plane detection data includes: Acquire sensor data for a current detection area from at least one sensor of the electronic device, the sensor data including acceleration data, angular velocity data, and image data; Determining the position and posture data of the electronic device according to the acceleration data, the angular velocity data and the image data; Acquire feature points in a current frame according to the image data, and match the feature points in the current frame with feature points in a previous frame to obtain a feature point matching relationship; Generate map point data according to the matching relationship between the posture data and the feature points; When the map point data meets the set conditions, plane data is generated according to the map point data.

3. The method according to claim 2, characterized in that The step of determining whether the plane detection is successful according to the plane detection data includes: In a case where the plane detection data does not include plane data, determining that the plane detection fails; In a case where the plane detection data includes plane data and the plane shown by the plane data is not unique, determining that the plane detection fails; In a case where the plane detection data includes plane data and the plane shown by the plane data is unique and satisfies a set plane condition, it is determined that the plane detection is successful.

4. The method according to claim 2, characterized in that: The plane detection failure type includes at least one of the following: a first type, a second type, and a third type; accordingly, The determining the type of plane detection failure according to the plane detection data includes: In the case where the plane detection data does not include plane data, detecting the number of map points in the map point data, and in the case where the number of map points is less than the set number of map points, determining that the plane detection failure type is the first type, and in the case where the number of map points is greater than the set number of map points, determining that the plane detection failure type is the second type; In the case where the plane data includes map point data and plane data, the number of planes in the plane data is detected, and in the case where the number of planes is not unique, the plane detection failure type is determined to be the third type.

5. The method according to claim 4, characterized in that In the case where the plane detection failure type is the first type or the second type, performing a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain a target plane includes: Determining a gesture motion trajectory according to the image data; constructing an initial plane according to the gesture motion trajectory; The target plane is constructed according to the initial plane and the plane detection data.

6. The method according to claim 5, characterized in that In a case where the plane detection failure type is the second type, constructing the target plane according to the initial plane and the plane detection data includes: Associating the map point data with the initial plane to obtain a plane association result; The initial plane is adjusted according to the plane association result to obtain the target plane.

7. The method according to claim 4, characterized in that In the case where the plane detection failure type is the third type, performing a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain a target plane includes: Based on the preset gesture interaction interface elements, multiple planes in the plane data are calibrated with interface elements to obtain interface element calibration results; Based on a preset gesture interaction type library, gesture recognition is performed on the image data to obtain a gesture recognition result; According to the gesture recognition result and the interface element calibration result, the planes in the plane data are merged to obtain a target plane.

8. A plane detection device, characterized in that: The device comprises: A detection module, configured to receive a plane detection instruction, and perform plane detection on a current detection area of ​​the electronic device in response to the plane detection instruction to obtain plane detection data; A determination module, used to determine whether the plane detection is successful according to the plane detection data; A judgment module, used for judging the type of plane detection failure according to the plane detection data when the plane detection is unsuccessful; The construction module is used to detect image data when a user performs a gesture operation in the current detection area, and perform a plane construction operation corresponding to the plane detection failure type according to the image data and the plane detection data to obtain a target plane.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 7.

Images