Grounding detection method and device and mobile equipment

Abstract

The invention relates to a grounding detection method and device and mobile equipment, and the method comprises the steps: firstly obtaining the image information of a target obstacle, then carrying out the obstacle recognition and ground recognition of the image information, obtaining the detection frame information of the target obstacle and the ground contour information, and finally, combining the detection frame information with the ground contour information, and obtaining a grounding detection result. A contact edge portion between the target obstacle and the ground, i.e., grounding edge information, is determined. According to the scheme, through combination of obstacle identification and ground identification, extraction of the grounding edge information is realized, the grounding edge of the finally obtained target obstacle is ensured to be actually contacted with the ground, the extraction precision of the grounding edge can be effectively improved, and when the method is applied to obstacle avoidance operation of the mobile equipment, the obstacle avoidance reliability of the mobile equipment can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of obstacle detection, and in particular to a ground detection method, device, and mobile equipment. Background Art

[0002] With the rapid development of science and technology, mobile devices represented by sweepers and robots are increasingly used in daily production and life, bringing great convenience to people. During the movement of mobile devices, it is often necessary to use distance detectors to detect the distance to the target obstacle, so as to extract the ground edge of the target obstacle, so that the mobile device can achieve obstacle avoidance operation.

[0003] However, this method is easily affected by ambient lighting and the texture of target obstacles, resulting in deviations in ground edge extraction and poor obstacle avoidance reliability. Summary of the invention

[0004] Based on this, it is necessary to provide a ground detection method, device, and mobile device to improve the extraction accuracy of the ground edge and improve the obstacle avoidance reliability of the mobile device.

[0005] The present application provides a ground detection method, comprising: acquiring image information of a target obstacle; performing obstacle recognition according to the image information to obtain detection frame information including the target obstacle; performing ground recognition according to the image information to obtain ground contour information; and determining ground edge information of the target obstacle according to the detection frame information and the ground contour information.

[0006] The above-mentioned ground detection method first obtains the image information of the target obstacle, and then performs obstacle recognition and ground recognition on the image information respectively to obtain the detection frame information including the target obstacle and the ground contour information. Finally, the edge part of the contact between the target obstacle and the ground is determined by combining the detection frame information and the ground contour information, that is, the ground edge information. This solution combines obstacle recognition and ground recognition to realize the extraction of ground edge information, ensuring that the ground edge of the target obstacle obtained in the end is actually in contact with the ground, which can effectively improve the extraction accuracy of the ground edge. When it is applied to the obstacle avoidance operation of the mobile device, it can effectively improve the obstacle avoidance reliability of the mobile device.

[0007] The present application also provides a ground detection device, including: an image acquisition module, used to acquire image information of a target obstacle; an obstacle recognition module, used to perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle; a ground segmentation module, used to perform ground recognition based on the image information to obtain ground contour information; an edge detection module, used to determine the ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0008] The present application also provides a mobile device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned grounding detection method when executing the computer program.

[0009] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned grounding detection method are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned grounding detection method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of a grounding detection method flow in one embodiment of the present application;

[0012] Figure 2 This is a flow chart of a grounding detection method in another embodiment of the present application;

[0013] Figure 3 This is a schematic diagram of a grounding detection method flow in another embodiment of the present application;

[0014] Figure 4 This is a flow chart of a grounding detection method in another embodiment of the present application;

[0015] Figure 5 This is a flow chart of a grounding detection method in another embodiment of the present application;

[0016] Figure 6 This is a schematic diagram of a grounding detection method flow in another embodiment of the present application;

[0017] Figure 7 This is a structural schematic diagram of a grounding detection device in one embodiment of the present application;

[0018] Figure 8 This is a schematic diagram of the internal structure of a mobile device in one embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] The ground detection method provided in the embodiment of the present application is applied to a mobile device with obstacle detection requirements. This type of mobile device has an image recognition function and is provided with an image collector. During the operation of the mobile device, the image collector can be used to collect images in real time, and the ground edge information of the target obstacle in contact with the ground in the environment where the mobile device is located can be determined through image recognition. After that, obstacle avoidance and other controls are implemented in combination with the ground edge information to improve the operation reliability of the mobile device.

[0021] Specifically, in one embodiment, the image collector of the mobile device includes a monocular image collector, that is, the image collection and distance detection functions are realized through a single camera or other visual sensor. The specific type of mobile device is not limited, as long as it is a mobile device that needs to detect the ground edge of an obstacle, for example, in a detailed embodiment, the mobile device includes a sweeper, an AGV (Automated Guided Vehicle), etc.

[0022] See also Figure 1 The present application provides a ground detection method, which is applied to a mobile device, and includes steps 102, 104, 106 and 108.

[0023] Step 102: Acquire image information of the target obstacle.

[0024] Specifically, the image information is information related to the image obtained by capturing images of the environment in which the target obstacle is located. The target obstacle is an object in the environment in which the mobile device is located that will block the normal operation of the mobile device. It can be understood that the specific type of target obstacle is not unique. In one embodiment, it can be shoes, tables and chairs, or walls, etc., which are not specifically limited. During the operation of the mobile device, due to the complex and diverse operating environment, in order to avoid the collision between the mobile device and the obstacle, it is necessary to detect the obstacle in real time, and control the mobile device to avoid the obstacle according to the detection results to achieve obstacle avoidance. Therefore, during the operation of the mobile device, it is necessary to obtain the image information corresponding to the target obstacle in real time, so as to perform subsequent target obstacle detection operations.

[0025] Step 104: perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle.

[0026] Specifically, obstacle recognition is to use image processing and image recognition technology to identify information corresponding to the target obstacle from the captured image (that is, image information). Including the target obstacle in this embodiment means: obtaining a detection frame through image recognition so that the target obstacle can be partially or completely located in the detection frame. The detection frame information is the information corresponding to the detection frame that selects part or all of the target obstacles. That is, in the solution of this embodiment, after obtaining the image information containing the target obstacle, the mobile device processes the image information, further determines the detection frame that includes part or all of the target obstacles, and obtains the detection frame information.

[0027] Step 106, performing ground recognition based on the image information to obtain ground contour information.

[0028] Specifically, the ground contour is the boundary between the ground and other objects in the image corresponding to the collected image information. In the solution of this embodiment, the image information is not only input into the mobile device for obstacle recognition, but also the ground is recognized at the mobile device to obtain ground contour information.

[0029] It should be noted that the ground recognition method is not unique. In one embodiment, a ground segmentation algorithm may be used to segment and extract the ground contour to obtain ground contour information. In another embodiment, other model analysis methods may be used to extract the ground contour, which is not specifically limited.

[0030] Step 108: Determine ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0031] Specifically, the ground edge information is the information corresponding to the line formed by the contact interface between the target obstacle and the ground. In the actual processing process, the ground contour identified at the target obstacle can be equivalent to the line formed by the contact interface between the target obstacle and the ground. Therefore, the solution of this embodiment only needs to combine the detection frame information and the ground contour information to extract the part of the ground contour that represents the line formed by the contact interface between the target obstacle and the ground, and use it as the ground edge, and further use the information corresponding to this part of the ground contour as the ground edge information.

[0032] It should be noted that the specific forms of the detection frame information, ground contour information and ground edge information are not unique. In a more detailed embodiment, since the detection frame, ground contour and ground edge all include individual pixel points, the detection frame information, ground contour information and ground edge information can be represented in the form of pixel coordinate parameters in a pixel coordinate system. In another embodiment, the pixel coordinate parameters can be further converted to the coordinate system corresponding to the image collector (the fuselage coordinate system), and the detection frame information, ground contour information and ground edge information can be represented in the form of fuselage coordinate parameters. The selection can be combined with actual needs, and there is no specific limitation.

[0033] In one embodiment, when a monocular image collector performs image acquisition, the coordinates of each pixel point of the acquired image in the pixel coordinate system are first obtained. Then, the pixel coordinate system is converted to the image coordinate system to obtain the coordinates of each pixel point in the acquired image in the image coordinate system. Finally, the image coordinate system is converted to the camera coordinate system (body coordinate system), and each pixel point is converted to the coordinate parameter in the image coordinate system, that is, the image information is obtained.

[0034] It should be pointed out that the method of obtaining the ground edge information based on the detection frame information and the ground contour information is not the only one. In a more detailed embodiment, the detection frame information and the ground contour information can be compared, and the information corresponding to the ground contour in the detection frame surrounded by the detection frame information can be used as the ground edge information. The specific selection can be based on actual needs.

[0035] The above-mentioned ground detection method first obtains the image information of the target obstacle, and then performs obstacle recognition and ground recognition on the image information respectively to obtain the detection frame information including the target obstacle and the ground contour information. Finally, the edge part of the contact between the target obstacle and the ground is determined by combining the detection frame information and the ground contour information, that is, the ground edge information. This solution combines obstacle recognition and ground recognition to realize the extraction of ground edge information, ensuring that the ground edge of the target obstacle obtained in the end is actually in contact with the ground, which can effectively improve the extraction accuracy of the ground edge. When it is applied to the obstacle avoidance operation of the mobile device, it can effectively improve the obstacle avoidance reliability of the mobile device.

[0036] See also Figure 2 In one embodiment, step 102 includes step 202 .

[0037] Step 202: receiving image information of the target obstacle collected and sent by the monocular image collector.

[0038] Specifically, the specific type of image collector used for image information acquisition is not unique. In order to facilitate the subsequent extraction of the ground edge, the solution of this embodiment uses a monocular image collector with a distance detection function (such as a monocular camera, etc.) to acquire images, obtain image information of the target obstacle, and send it to a mobile device for further processing.

[0039] It is understandable that in other embodiments, other image collectors with image acquisition functions may be used to acquire image information, such as a binocular image collector with a binocular ranging function, etc., without specific limitation.

[0040] See also Figure 3 In one embodiment, step 104 includes step 302 .

[0041] Step 302: Obstacle recognition is performed based on the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

[0042] Specifically, the preset neural network detection model is a preset detection model obtained through deep learning training and used to realize object recognition. In the example scheme of this implementation, a pre-trained detection network, that is, a preset neural network detection model, is set in the mobile device. After receiving the image information collected and sent by the image collector, it is input into the preset neural network detection model for model analysis, and the corresponding detection frame information can be output.

[0043] It should be pointed out that the specific type of the preset neural network detection model is not unique. In a more detailed embodiment, the yolo (you only look once) model can be used for training. There is no specific limitation and the model can be selected or adjusted according to actual needs.

[0044] It can be understood that the yolo model is an algorithm for target detection using a convolutional neural network, which can identify the position of the target in an image and calibrate the position area of ​​the target with a rectangular or other shaped detection frame. The yolo model used in this application is not unique, and can be yolo v1, yolo v2 or yolo v5, etc., which is not specifically limited.

[0045] The above scheme uses a trained neural network detection model to identify and detect image information and determine the detection frame information including the target obstacle, which has high detection efficiency and detection accuracy.

[0046] See also Figure 4 , in one embodiment, step 302 includes step 402 .

[0047] Step 402 , analyzing the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

[0048] Specifically, the preset specific object detection model is a preset detection model obtained by deep learning training of specific objects, and is used to realize specific object recognition. The solution of this embodiment, combined with the use scenario of the mobile device, can perform model training on a specific type of object, and finally obtain and store a specific object detection model. When performing subsequent object recognition, this type of specific object can be recognized, so as to be applicable to a specific obstacle environment.

[0049] The solution of this embodiment, when performing obstacle recognition, can not only obtain the detection frame information corresponding to the target obstacle, but also obtain the type of the target obstacle. When performing obstacle avoidance subsequently, different obstacle avoidance operations can be performed in combination with the type of the target obstacle to improve the reliability of obstacle avoidance.

[0050] It should be noted that the types of specific objects may vary depending on the actual usage scenarios. For example, in one embodiment, the mobile device is used in a room scenario, and the specific objects include shoes, socks, tables and chairs, sofas, etc.

[0051] The above scheme can establish a preset specific object detection model for a specific object, and subsequently identify the specific object as a target obstacle to achieve detection of the specific object with high detection accuracy.

[0052] See also Figure 5 , in one embodiment, step 302 includes step 502 .

[0053] Step 502: Analyze the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

[0054] Specifically, the preset general object detection model is a preset detection model obtained by deep learning training of general objects, and is used to realize the recognition of all objects. The scheme of this embodiment, combined with the use scenario of mobile devices, can perform model training on all types of objects, and finally obtain and store the general object detection model. When performing object recognition later, all types of objects can be recognized. Therefore, in actual scenarios, the object detection method of this embodiment can be used to identify all objects other than the ground in the image information and obtain the corresponding detection frame information.

[0055] The above solution uses a preset general object detection model for obstacle recognition, which can realize the recognition of more types of obstacles in image information and has higher applicability.

[0056] See also Figure 6In one embodiment, step 108 includes step 602 and step 604 .

[0057] Step 602, checking whether the contour point corresponding to the ground contour information is located in the detection frame corresponding to the detection frame information.

[0058] Step 604: If there are contour points located in the detection frame, the ground edge information of the target obstacle is determined according to the contour points located in the detection frame.

[0059] Specifically, the ground contour and the lines corresponding to the detection frame are formed by connecting multiple pixel points. Therefore, in the actual detection scenario, the monocular image collector can obtain the pixel coordinate parameters of each pixel point in the pixel coordinate system, or convert each pixel point into a position point in the coordinate system of the monocular image collector itself (that is, the fuselage coordinate system), that is, expressed in the fuselage coordinate system parameters. Whether it is a detection frame including target obstacles or a ground contour obtained by ground segmentation, it can be represented by a series of position points. Therefore, in the actual detection process, the coordinate parameters of each point in the detection frame and the ground contour can be obtained, that is, the detection frame information and the ground contour information can be obtained.

[0060] Therefore, when analyzing the detection frame information and the ground contour information, we first need to determine whether the ground contour and the detection frame have an intersection, that is, whether the contour points in the ground contour are within the area defined by the detection frame. If there are contour points within the detection frame, it means that the target obstacle is actually detected to be in contact with the ground. At this time, the contour points within the detection frame will be combined to achieve further ground edge extraction and analysis.

[0061] It can be understood that when judging whether the ground contour and the detection frame have an intersection, it can be implemented in the pixel coordinate system. At this time, the contour points are represented in the form of pixel coordinate parameters, as well as the area framed by the detection frame. After that, it is only necessary to combine the various pixel coordinate parameters for analysis to determine whether there are contour points located within the detection frame.

[0062] In the above scheme, after obtaining the detection frame information and the ground contour information, it is first detected whether there are contour points located in the detection frame, and the ground edge analysis is performed based on this, so as to further improve the accuracy of obtaining the ground edge.

[0063] Please refer to Figure 6 In one embodiment, after step 602 , the method further includes step 606 .

[0064] Step 606: If there is no contour point within the detection frame, determine the ground edge information of the target obstacle according to the detection frame information.

[0065] Specifically, when the mobile device analyzes the contour points in the ground contour corresponding to the ground contour information, it may also be due to image acquisition errors or obstacle recognition errors, and there may be no contour points located in the detection frame. At this time, in order to ensure the smooth progress of the ground detection, the ground edge information of the target obstacle will be determined in combination with the detection frame information. For example, each point corresponding to a certain edge in the detection frame corresponding to the detection frame information is used as the ground contour point where the target obstacle contacts the ground, so as to obtain the ground edge information of the target obstacle.

[0066] In more detail, in one embodiment, taking a rectangular detection frame as an example, each point corresponding to the bottom edge of the rectangular detection frame can be used as a grounding contour point, so as to analyze and obtain the final grounding edge information.

[0067] In the above scheme, when there is no contour point located in the detection frame, the detection frame information is continued to be used to determine the grounding edge information of the target obstacle, thereby ensuring the smooth progress of the grounding detection and improving the reliability of the grounding detection operation.

[0068] In one embodiment, the ground edge information of the target obstacle is determined based on the contour points located in the detection frame, including: converting the pixel coordinate parameters corresponding to the contour points located in the detection frame to the body coordinates corresponding to the monocular image collector to obtain the body coordinate parameters of the contour points in the detection frame; and determining the ground edge information of the target obstacle based on the body coordinate parameters.

[0069] Specifically, as shown in the above embodiment, when the monocular image collector performs image acquisition, the coordinates of each pixel point of the acquired image in the pixel coordinate system are first obtained. When analyzing whether the contour point is located in the detection frame, it can be directly compared in the form of coordinate parameters in the pixel coordinate system. However, considering that the distance between the coordinate parameters of each pixel point in the pixel coordinate system is not obvious, the pixel coordinate system can be further converted to the image coordinate system, and the pixel coordinate parameters corresponding to the contour points in the detection frame are converted to the image coordinate system. Finally, the image coordinate system is converted to the parasitic coordinate system to obtain the fuselage coordinate parameters of the contour points in the detection frame, and the ground edge information of the target obstacle is determined by this.

[0070] In the above scheme, after obtaining the contour points located in the detection frame, the pixel coordinates of each contour point are converted to the fuselage coordinates, so as to obtain more accurate ground edge information.

[0071] In one embodiment, determining the ground edge information of the target obstacle according to the fuselage coordinate parameters includes: performing contour point filtering analysis according to the fuselage coordinate parameters to obtain precise contour points; and determining the ground edge information of the target obstacle according to the precise contour points.

[0072] Specifically, filtering analysis is performed based on the contour points located in the detection frame. Specifically, analysis is performed based on the information of the contour points located in the detection frame (that is, the coordinate parameters of the contour points in the fuselage coordinate system) to screen out contour points that can reasonably characterize the ground edge, that is, to screen out accurate contour points. Finally, the accurate contour points are combined into a set of position points where the target obstacle contacts the ground, and the coordinate parameters of each position point in the position point set are combined to obtain the ground edge information.

[0073] After obtaining each contour point within the detection frame, the above scheme will further filter the contour points according to actual needs, screen out accurate contour points that can reasonably characterize the grounding position, so as to determine the grounding edge information and effectively improve the accuracy of the grounding edge information.

[0074] It should be noted that the method of filtering each contour point is not unique. In one embodiment, the filtering analysis includes: based on the fuselage coordinate parameters of each contour point, extracting contour points whose distance from the monocular image collector is within a preset distance range as precise contour points.

[0075] Specifically, in the scheme of this embodiment, when filtering the contour points located in the detection frame, monocular ranging is performed through a monocular image collector to obtain the distance from each contour point to the monocular image collector, that is, the distance from the contour point to the origin of the fuselage coordinate system is calculated according to the fuselage coordinate parameters of the contour point in the fuselage coordinate system, and then the contour points with a distance within a preset distance range are filtered, and the contour points with a distance within the preset distance range are retained as precise contour points. Finally, the line formed by the precise contour points is used as the ground edge, and the set of coordinate parameters corresponding to each precise contour point is the ground edge information. It can be understood that the size within the preset distance range is not unique, and can be set in combination with the actual use scenario or use requirements, and is not specifically limited.

[0076] In one embodiment, the filtering analysis includes: based on the fuselage coordinate parameters of each contour point, extracting the contour point whose distance difference with the nearest contour point is less than or equal to a first preset distance threshold as the precise contour point; wherein the nearest contour point includes the contour point located in the detection frame and closest to the monocular image collector.

[0077] Specifically, when filtering the contour points located in the detection frame, a monocular distance measurement is performed through a monocular image collector. After obtaining the distance from each contour point to the monocular image collector, the distance is subtracted from the distance of the nearest contour point (that is, the minimum distance), and the contour points whose difference is greater than the first preset distance threshold are filtered, and the contour points whose difference is less than or equal to the first preset distance threshold are retained as precise contour points. Finally, the line formed by the precise contour points is used as the ground edge, and the set of coordinate parameters corresponding to each precise contour point is the ground edge information. It can be understood that the size of the first preset distance threshold is not unique, and it can be set in combination with the actual scene.

[0078] In one embodiment, the filtering analysis includes: based on the fuselage coordinate parameters of each contour point, extracting the contour points whose angles with adjacent contour points are greater than or equal to a preset angle threshold as precise contour points; wherein the angle includes the angles between the straight lines formed by two adjacent contour points and the origin of the fuselage coordinate system.

[0079] Specifically, the fuselage coordinate system is the coordinate system where the image collector is located, and the origin of the fuselage coordinate system is the origin of the fuselage coordinate system, which is generally the optical center of the monocular image collector. In this solution, when filtering the contour points located in the detection frame, the fuselage coordinate system is built according to the monocular image collector, and the coordinate parameters of each contour point are analyzed in combination with the coordinate origin, and the angle between adjacent contour points is finally determined, and the contour points with angles less than the preset angle threshold are filtered out, and the contour points with angles greater than or equal to the preset angle threshold are retained as precise contour points. Finally, the line formed by the precise contour points is used as the ground edge, and the set of coordinate parameters corresponding to each precise contour point is the ground edge information. It can be understood that the size of the preset angle threshold is not unique, and it can be set in combination with the actual scene.

[0080] In one embodiment, the filtering analysis includes: extracting, according to the fuselage coordinate parameters of each contour point, contour points whose distances to adjacent contour points are less than or equal to a second preset distance threshold as precise contour points.

[0081] In the above scheme, when filtering the contour points located in the detection frame, the monocular distance measurement is performed through the monocular image collector to obtain the distance between adjacent contour points, filter the contour points whose distance is greater than the second preset distance threshold, and retain the contour points whose distance is less than or equal to the second preset distance threshold as precise contour points. Finally, the line formed by the precise contour points is used as the ground edge, and the set of coordinate parameters corresponding to each precise contour point is the ground edge information. It can be understood that the size of the second preset distance threshold is not unique, and can be the same as or different from the first preset distance threshold, and can be set in combination with the actual scene.

[0082] In more detail, when filtering and analyzing the contour points within the detection frame, one or more of the above-mentioned filtering and analysis methods may be used, and the selection may be made in combination with real-time requirements, without any specific limitation.

[0083] In order to facilitate understanding of the technical solution of the present application, the present application is explained below in conjunction with more detailed embodiments.

[0084] First, during the operation of the mobile device, the image of the environment is collected in real time through a monocular camera to obtain image information corresponding to the target obstacle, and then obstacle recognition and ground segmentation are performed on the image information respectively. Specifically, a preset specific object detection model is stored in the mobile device, and the acquired image information is brought into the preset specific object detection model for analysis to obtain the type of target obstacle and the detection frame information including the target obstacle; the image information is brought into the corresponding ground segmentation model for analysis to obtain ground contour information.

[0085] In the actual detection process, image information, detection frame information, and ground contour information can all be regarded as a set of coordinate parameters of a series of pixel points. After obtaining the detection frame information and ground contour information, the corresponding pixel points are expressed in the form of pixel coordinates, and the contour points in the ground contour information are checked to determine whether each contour point is within the detection frame corresponding to the detection frame information, that is, whether the coordinate parameters of each contour point are within the area enclosed by the detection frame.

[0086] If there are contour points in the detection frame, the pixel coordinate parameters of each contour point in the detection frame are converted into fuselage coordinate parameters in combination with the monocular ranging algorithm, so as to filter and analyze the contour points in the detection frame to obtain precise contour points. Finally, the fuselage coordinate parameters corresponding to the precise contour points are used as the ground edge information, and the lines formed by each precise contour point are used as the ground edge.

[0087] If there is no contour point in the detection frame, the bottom edge of the detection frame is used to obtain the ground edge, and the pixel points of the bottom edge of the detection frame are filtered in the same way as the above contour point filtering. Finally, the coordinate parameters corresponding to the pixel points after filtering are used as the ground edge information, and the lines formed by each precise contour point are used as the ground edge.

[0088] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0089] Based on the same inventive concept, the embodiment of the present application also provides a grounding detection device for implementing the grounding detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more grounding detection device embodiments provided below can refer to the limitations of the grounding detection method above, and will not be repeated here.

[0090] In one embodiment, Figure 7 As shown, a ground detection device is provided, including: an image acquisition module 702 , an obstacle recognition module 704 , a ground segmentation module 706 and an edge detection module 708 .

[0091] The image acquisition module 702 is used to acquire image information of the target obstacle; the obstacle recognition module 704 is used to perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle; the ground segmentation module 706 is used to perform ground recognition based on the image information to obtain ground contour information; the edge detection module 708 is used to determine the ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0092] In one embodiment, the image acquisition module 702 is further configured to receive image information of the target obstacle collected and sent by the monocular image collector.

[0093] In one embodiment, the obstacle recognition module 704 is further configured to perform obstacle recognition based on the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

[0094] In one embodiment, the obstacle identification module 704 is further configured to perform analysis based on the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

[0095] In one embodiment, the obstacle recognition module 704 is further configured to perform analysis based on the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

[0096] In one embodiment, the edge detection module 708 is also used to verify whether the contour points corresponding to the ground contour information are located in the detection frame corresponding to the detection frame information; if there are contour points located in the detection frame, the ground edge information of the target obstacle is determined based on the contour points located in the detection frame.

[0097] In one embodiment, the edge detection module 708 is further configured to determine ground edge information of the target obstacle according to the detection frame information if there is no contour point located within the detection frame.

[0098] In one embodiment, the edge detection module 708 is further used to perform filtering analysis based on the contour points within the detection frame to obtain precise contour points; and determine the ground edge information of the target obstacle based on the precise contour points.

[0099] Each module in the above-mentioned ground detection device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0100] The above-mentioned ground detection device first obtains the image information of the target obstacle, and then performs obstacle recognition and ground recognition on the image information respectively, obtains the detection frame information including the target obstacle and the ground contour information, and finally, combines the detection frame information and the ground contour information to determine the edge portion of the target obstacle in contact with the ground, that is, the ground edge information. This solution combines obstacle recognition and ground recognition to extract the ground edge information, ensuring that the ground edge of the target obstacle obtained in the end is actually in contact with the ground, which can effectively improve the extraction accuracy of the ground edge. When it is applied to the obstacle avoidance operation of the mobile device, it can effectively improve the obstacle avoidance reliability of the mobile device.

[0101] In one embodiment, a mobile device is provided. The mobile device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The mobile device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the mobile device is used to provide computing and control capabilities. The memory of the mobile device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the mobile device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a ground detection method is implemented. The display screen of the mobile device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the mobile device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the mobile device, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the mobile device to which the solution of the present application is applied. The specific mobile device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0103] In one embodiment, a mobile device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0104] Obtain image information of the target obstacle; perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle; perform ground recognition based on the image information to obtain ground contour information; determine the ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0105] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: receiving image information of the target obstacle collected and sent by the monocular image collector.

[0106] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obstacle recognition is performed based on the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

[0107] In one embodiment, when the processor executes the computer program, the following steps are further implemented: analyzing the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

[0108] In one embodiment, when the processor executes the computer program, the following steps are further implemented: analyzing the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

[0109] In one embodiment, when the processor executes the computer program, the following steps are also implemented: checking whether the contour points corresponding to the ground contour information are located in the detection frame corresponding to the detection frame information; if there are contour points located in the detection frame, determining the ground edge information of the target obstacle based on the contour points located in the detection frame.

[0110] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if there is no contour point located within the detection frame, determining the ground edge information of the target obstacle according to the detection frame information.

[0111] In one embodiment, when the processor executes the computer program, the following steps are also implemented: converting the pixel coordinate parameters corresponding to the contour points within the detection frame into the fuselage coordinates corresponding to the monocular image collector to obtain the fuselage coordinate parameters of the contour points within the detection frame; and determining the ground edge information of the target obstacle based on the fuselage coordinate parameters.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing contour point filtering analysis according to the fuselage coordinate parameters to obtain precise contour points; and determining the ground edge information of the target obstacle according to the precise contour points.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0114] Obtain image information of the target obstacle; perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle; perform ground recognition based on the image information to obtain ground contour information; determine the ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0115] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving image information of the target obstacle collected and sent by the monocular image collector.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obstacle recognition is performed based on the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: analyzing the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: analyzing the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: checking whether the contour points corresponding to the ground contour information are located in the detection frame corresponding to the detection frame information; if there are contour points located in the detection frame, determining the ground edge information of the target obstacle based on the contour points located in the detection frame.

[0120] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is no contour point located within the detection frame, determining the ground edge information of the target obstacle based on the detection frame information.

[0121] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: converting the pixel coordinate parameters corresponding to the contour points within the detection frame into the fuselage coordinates corresponding to the monocular image collector to obtain the fuselage coordinate parameters of the contour points within the detection frame; and determining the ground edge information of the target obstacle based on the fuselage coordinate parameters.

[0122] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing contour point filtering analysis according to the fuselage coordinate parameters to obtain precise contour points; and determining the ground edge information of the target obstacle according to the precise contour points.

[0123] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0124] Obtain image information of the target obstacle; perform obstacle recognition based on the image information to obtain detection frame information including the target obstacle; perform ground recognition based on the image information to obtain ground contour information; determine the ground edge information of the target obstacle based on the detection frame information and the ground contour information.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving image information of the target obstacle collected and sent by the monocular image collector.

[0126] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obstacle recognition is performed based on the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

[0127] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: analyzing the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: analyzing the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: checking whether the contour points corresponding to the ground contour information are located in the detection frame corresponding to the detection frame information; if there are contour points located in the detection frame, determining the ground edge information of the target obstacle based on the contour points located in the detection frame.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is no contour point located within the detection frame, determining the ground edge information of the target obstacle based on the detection frame information.

[0131] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: converting the pixel coordinate parameters corresponding to the contour points within the detection frame into the fuselage coordinates corresponding to the monocular image collector to obtain the fuselage coordinate parameters of the contour points within the detection frame; and determining the ground edge information of the target obstacle based on the fuselage coordinate parameters.

[0132] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing contour point filtering analysis according to the fuselage coordinate parameters to obtain precise contour points; and determining the ground edge information of the target obstacle according to the precise contour points.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0134] The mobile device, storage medium and computer program product first obtain the image information of the target obstacle, then perform obstacle recognition and ground recognition on the image information respectively, obtain the detection frame information including the target obstacle and the ground contour information, and finally, combine the detection frame information and the ground contour information to determine the edge portion of the target obstacle in contact with the ground, that is, the ground edge information. This solution combines obstacle recognition and ground recognition to extract the ground edge information, ensure that the ground edge of the target obstacle obtained in the end is actually in contact with the ground, and can effectively improve the extraction accuracy of the ground edge. When it is applied to the obstacle avoidance operation of the mobile device, it can effectively improve the obstacle avoidance reliability of the mobile device.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A ground detection method, characterized in that: include: Obtain image information of target obstacles; Perform obstacle recognition according to the image information to obtain detection frame information including the target obstacle; Performing ground recognition according to the image information to obtain ground contour information; The ground edge information of the target obstacle is determined according to the detection frame information and the ground contour information.

2. The grounding detection method according to claim 1, characterized in that: The step of obtaining image information of the target obstacle includes: Receive the image information of the target obstacle collected and sent by the monocular image collector.

3. The grounding detection method according to claim 1, characterized in that: The performing obstacle recognition according to the image information to obtain detection frame information including the target obstacle includes: Obstacle recognition is performed according to the image information and a preset neural network detection model to obtain detection frame information including the target obstacle.

4. The grounding detection method according to claim 3, characterized in that: The performing obstacle recognition according to the image information and a preset neural network detection model to obtain detection frame information including the target obstacle includes: An analysis is performed based on the image information and a preset specific object detection model to obtain the type of the target obstacle and the detection frame information including the target obstacle.

5. The grounding detection method according to claim 3, characterized in that: The performing obstacle recognition according to the image information and a preset neural network detection model to obtain detection frame information including the target obstacle includes: Analyze the image information and a preset general object detection model to obtain detection frame information including the target obstacle.

6. The grounding detection method according to any one of claims 2 to 5, characterized in that: The determining, according to the detection frame information and the ground contour information, the ground edge information of the target obstacle includes: Verifying whether the contour point corresponding to the ground contour information is located within the detection frame corresponding to the detection frame information; If there are contour points located within the detection frame, ground edge information of the target obstacle is determined according to the contour points located within the detection frame.

7. The grounding detection method according to claim 6, characterized in that: After verifying whether the contour point corresponding to the ground contour information is located within the detection frame corresponding to the detection frame information, the method further includes: If there is no contour point within the detection frame, the ground edge information of the target obstacle is determined according to the detection frame information.

8. The grounding detection method according to claim 6, characterized in that: The step of determining the ground edge information of the target obstacle according to the contour points located in the detection frame includes: Convert pixel coordinate parameters corresponding to the contour points in the detection frame to fuselage coordinate parameters corresponding to the monocular image collector to obtain fuselage coordinate parameters of the contour points in the detection frame; The ground edge information of the target obstacle is determined according to the fuselage coordinate parameters.

9. The grounding detection method according to claim 8, characterized in that: The determining the ground edge information of the target obstacle according to the fuselage coordinate parameters includes: Perform contour point filtering analysis according to the fuselage coordinate parameters to obtain accurate contour points; The ground edge information of the target obstacle is determined according to the precise contour points.

10. The grounding detection method according to claim 9, characterized in that: The filtering analysis includes at least one of the following: Item 1: According to the fuselage coordinate parameters of each of the contour points, contour points whose distances from the monocular image collector are within a preset distance range are extracted as precise contour points; Item 2: According to the fuselage coordinate parameters of each of the contour points, a contour point whose distance difference with the nearest contour point is less than or equal to a first preset distance threshold is extracted as an accurate contour point; wherein the nearest contour point includes a contour point located in the detection frame and closest to the monocular image collector; Item 3: According to the fuselage coordinate parameters of each of the contour points, a contour point whose angle with an adjacent contour point is greater than or equal to a preset angle threshold is extracted as a precise contour point; wherein the angle includes the angle between the straight lines formed by two adjacent contour points and the origin of the fuselage coordinate system; Item 4: According to the fuselage coordinate parameters of each of the contour points, contour points whose distances to adjacent contour points are less than or equal to a second preset distance threshold are extracted as precise contour points.

11. A grounding detection device, characterized in that: include: An image acquisition module is used to acquire image information of target obstacles; An obstacle recognition module, used to perform obstacle recognition according to the image information to obtain detection frame information including the target obstacle; A ground segmentation module, used to perform ground recognition according to the image information to obtain ground contour information; The edge detection module is used to determine the ground edge information of the target obstacle according to the detection frame information and the ground contour information.

12. A mobile device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the ground detection method according to any one of claims 1 to 10 are implemented.

Images