Automatic obstacle avoidance method based on mobile device, electronic device and storage medium

Abstract

The invention provides an automatic obstacle avoidance method based on a mobile device, an electronic device and a storage medium. The mobile device comprises a central processing unit and data processing hardware. The automatic obstacle avoidance method comprises the following steps: acquiring an environment image of the movable equipment in a movement process; the environment image is input into data processing hardware, so that the data processing hardware extracts feature points used for representing obstacles in the environment image according to the environment image, and the central processing unit receives the feature points extracted by the data processing hardware; obtaining depth information of the environment image according to the feature points, wherein the depth information is used for representing the distance between the obstacle and the mobile device; and performing motion path planning on the mobile equipment according to the depth information. According to the invention, the method can reduce the calculation resource consumption of the mobile device for obstacle avoidance, and achieves the precise obstacle avoidance of the mobile device.

Description

Technical Field

[0001] The present application relates to the technical field of visual obstacle avoidance, and in particular to an automatic obstacle avoidance method based on a movable device, an electronic device, and a storage medium. Background Art

[0002] Obstacle avoidance technology is the key to ensuring the movement safety of mobile devices and achieving autonomous navigation. Taking mobile devices as an example, the mainstream obstacle avoidance solutions for mobile devices can be roughly divided into sensor-based solutions and vision-based solutions. Sensor-based solutions: including laser radar (LiDAR), sonar, infrared sensors, etc. These solutions can provide direct distance measurement and have high obstacle avoidance accuracy and reliability. Vision-based solutions: mainly use cameras to capture images and achieve obstacle avoidance through image processing technology. Compared with sensor solutions, vision solutions are low-cost, light-weight, and more suitable for small mobile devices.

[0003] However, high-precision sensors are expensive and increase the load on mobile devices, affecting flight time and flexibility. Vision solutions are sensitive to lighting conditions and environmental complexity, require complex algorithms to ensure accuracy, and have high demands on computing resources. Summary of the invention

[0004] In view of this, the present application provides an automatic obstacle avoidance method and related equipment based on a movable device, which can achieve accurate obstacle avoidance of the movable device while reducing the computing resource consumption of the movable device for obstacle avoidance.

[0005] An embodiment of the present application provides an automatic obstacle avoidance method based on a movable device, wherein the movable device includes a central processing unit and data processing hardware; the automatic obstacle avoidance method includes: collecting an environmental image of the movable device during movement; inputting the environmental image into the data processing hardware, so that the data processing hardware extracts feature points for characterizing obstacles in the environmental image based on the environmental image, wherein the central processing unit receives the feature points extracted by the data processing hardware; obtaining depth information of the environmental image based on the feature points, the depth information being used to characterize the distance between the obstacle and the movable device; and planning a motion path for the movable device based on the depth information.

[0006] Compared with the related art, the embodiments of the present application have at least the following advantages: the environmental image is input into the data processing hardware to obtain the feature points of the obstacles in the environmental image. In other words, the step of extracting the feature points in the environmental image is implemented in the data processing hardware, thereby reducing the computing resource consumption of the central processor of the mobile device when processing obstacle avoidance, and enabling accurate obstacle avoidance of the mobile device. In addition, during the movement of the mobile device, it is only necessary to capture the environmental image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the mobile device.

[0007] In some possible implementations, the data processing hardware includes at least a feature detection module, a direction allocation module and a descriptor generation module; the feature points include key points and feature descriptors, the key points are pixels with geometric features in the environment image, and the feature descriptors are used to describe the difference between the key points and the pixels around the key points; the environment image is input into the data processing hardware so that the data processing hardware calculates the feature points used to characterize obstacles in the environment image based on the environment image, including: inputting the environment image into the data processing hardware, wherein the feature detection module is used to calculate the key points in the environment image, the direction allocation module is used to calculate the main direction of the key points based on the calculation results of the feature detection module, and the descriptor generation module is used to calculate the feature descriptors based on the calculation results of the feature detection module.

[0008] By adopting the above technical solution, the accuracy of feature point extraction by data processing hardware can be improved, thereby ensuring the accuracy of subsequent depth information, and further improving the obstacle avoidance accuracy of the movable device.

[0009] In some possible implementations, the feature detection module includes a comparator and an integral image calculation circuit, the comparator is used to compare the grayscale values ​​of each pixel in the environment image with the surrounding pixels and determine the corner points from each pixel; the integral image calculation circuit is used to calculate the response value of the corner point; the direction allocation module includes a multiplier, an adder and a coordinate rotation digital calculation circuit, the multiplier and the adder are used to calculate the grayscale centroid of each pixel in the environment image, and the coordinate rotation digital calculation circuit is used to calculate the angle of the line between the grayscale centroid and the corner point; the descriptor generation module includes a pixel pair comparison circuit, the pixel pair comparison circuit is used to compare the grayscale values ​​of pixel pairs around the corner point, and generate a feature descriptor based on the comparison result.

[0010] By adopting the above technical solution, a feasible design solution for data processing hardware is provided, which can improve the accuracy of feature point extraction by data processing hardware, thereby ensuring the accuracy of subsequent depth information, and further improving the obstacle avoidance accuracy of movable equipment.

[0011] In some possible implementations, the movable device includes a drone; before inputting the environmental image into data processing hardware, it also includes: obtaining the flight speed and flight attitude data of the drone; determining, based on the flight speed and flight attitude data, a flight channel area under a target camera corresponding to the flight direction of the drone, wherein the flight channel area includes boundary areas in vertical and horizontal directions; performing target segmentation on the environmental image to segment the environmental image into multiple target segmentation areas, wherein each target segmentation area includes at least one object in the environmental image; determining an obstacle avoidance area of ​​the drone based on the multiple target segmentation areas and the flight channel area; and inputting the environmental image into data processing hardware, including: inputting the obstacle avoidance area into the data processing hardware.

[0012] By adopting the above technical solution, data processing can be performed only based on the obstacle avoidance area, which reduces the amount of data and thus the consumption of computing data, while improving data processing accuracy and achieving precise obstacle avoidance for the UAV.

[0013] In some possible implementations, target segmentation is performed on an environmental image to divide the environmental image into multiple target segmentation areas, including: inputting the environmental image into a pre-trained image segmentation network model, and determining multiple target segmentation areas based on an output result of the image segmentation network model; wherein the image segmentation network model is used to output annotation information of objects in each target segmentation area, and the annotation information includes at least size information of the object and position information in the environmental image.

[0014] By adopting the above technical solution, the accuracy of target segmentation of environmental images can be improved, ensuring that each target segmentation area includes the target object, thereby improving the accuracy of the subsequent obstacle avoidance area obtained based on the target segmentation area, and further improving the obstacle avoidance accuracy of the drone.

[0015] In some possible implementations, the image segmentation network model is trained according to the following steps: obtaining historical environment images of the drone during its historical flight; annotating historical objects in the historical environment images to obtain training images with real labels, wherein the real labels include at least the size of the historical objects and the position in the historical environment images; and training the initial image segmentation network model using the training images to obtain the image segmentation network model.

[0016] By adopting the above technical solution, the model accuracy of the image segmentation network model can be improved, thereby improving the accuracy of the image segmentation network model in performing target segmentation on environmental images.

[0017] In some possible implementations, an obstacle avoidance area of ​​a UAV is determined based on multiple target segmentation areas and a flight channel area, including: obtaining a priority of each target segmentation area, wherein a larger priority of a target segmentation area indicates a smaller shortest straight-line distance between the UAV and the target segmentation area; taking a target segmentation area whose priority satisfies a preset priority screening rule as a final segmentation area; determining obstacles in the flight channel area based on the final segmentation area; and taking an area with obstacles in the flight channel area as an obstacle avoidance area.

[0018] By adopting the above technical solution, it is possible to improve the accuracy of the obstacle avoidance area subsequently obtained based on the target object while reducing the flight path of the UAV in the obstacle avoidance area, thereby improving the flight efficiency of the UAV.

[0019] In some possible implementations, before planning the motion path of the movable device based on the depth information, it also includes: smoothing and denoising the depth information; planning the motion path of the movable device based on the depth information, including: planning the motion path of the movable device based on the final depth information after smoothing and denoising.

[0020] By adopting the above technical solution, the accuracy of the final depth information can be improved, thereby improving the reliability of the planned path obtained according to the final depth information after smoothing and denoising.

[0021] The second aspect of the present application discloses an electronic device, which includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the above-mentioned automatic obstacle avoidance method based on a movable device.

[0022] The third aspect of the present application discloses a storage medium including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned automatic obstacle avoidance method based on a movable device.

[0023] It can be understood that the electronic device of the second aspect and the storage medium of the third aspect provided above both correspond to the method of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of an automatic obstacle avoidance method based on a movable device according to an embodiment of the present invention.

[0025] Figure 2 The present invention is a flowchart of an automatic obstacle avoidance method based on a movable device according to an embodiment of the present invention.

[0026] Figure 3 The present invention is a flowchart of an automatic obstacle avoidance method based on a movable device according to an embodiment of the present invention.

[0027] Figure 4 The present invention is a flowchart of an automatic obstacle avoidance method based on a movable device according to an embodiment of the present invention.

[0028] Figure 5 Schematic diagram of functional modules of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0032] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0033] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0034] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0035] Please refer to Figure 1 , is a flow chart of an automatic obstacle avoidance method based on a mobile device provided in an embodiment of the present application. The mobile device includes a central processing unit and data processing hardware, such as Figure 1As shown, this embodiment is applied to a central processor of a mobile device, and includes the following steps: Step 101: Collect environmental images during the movement of the movable device.

[0036] In some embodiments, a photographing device is provided on the movable device, and the environmental image during the movement of the movable device is collected by the photographing device.

[0037] In some embodiments, the shooting device is preferably a monocular camera, which has the advantages of small size, light weight, low energy consumption, and wide field of view, and is more suitable for obstacle avoidance perception of small movable devices.

[0038] Step 102: Input the environment image into data processing hardware, so that the data processing hardware extracts feature points for characterizing obstacles in the environment image according to the environment image.

[0039] In some embodiments, feature points include key points and feature descriptors. Key points are points with significant geometric features in the point cloud corresponding to the environment image. These points are usually points with high curvature or large local curvature changes, and may be inflection points, edges, surface bumps, etc. in the point cloud. Feature descriptors are tools used to describe the geometric shape and structure of local areas in point clouds.

[0040] It should be noted that, from the above description, it can be seen that the key point is a point with a higher curvature or a larger change in local curvature, so the obstacle represented by the feature point is an obstacle with significant geometric features in the environment image.

[0041] In some embodiments, the data processing hardware includes at least a feature detection module, a direction allocation module and a descriptor generation module; the environmental image is input into the data processing hardware so that the data processing hardware calculates feature points for characterizing obstacles in the environmental image based on the environmental image, including: inputting the environmental image into the data processing hardware, wherein the feature detection module is used to calculate key points in the environmental image, the direction allocation module is used to calculate the main direction of the key points based on the calculation results of the feature detection module, and the descriptor generation module is used to calculate the feature descriptor based on the calculation results of the feature detection module.

[0042] Specifically, the feature detection module selects a pixel point P in the environment image and compares the pixel point P with the surrounding pixels. If the grayscale value of the pixel point P is different enough from the grayscale values ​​of the surrounding pixels, the pixel point P is considered to be a corner point.

[0043] In some embodiments, the difference between the grayscale value of the pixel point P and the grayscale values ​​of the surrounding pixels can be calculated by the following formula: ;in, is the gray value of pixel P, is the gray value of the surrounding pixels, is the difference between the grayscale value of pixel P and the grayscale values ​​of its surrounding pixels.

[0044] In some embodiments, the function of the feature detection module can be implemented by the following steps: 1. Using a comparator circuit to compare the grayscale values ​​of a pixel point with those of surrounding pixels; 2. Using an integral image to calculate the response value of a corner point. In other words, the feature detection module of this embodiment can be obtained by designing a comparator and an integral image calculation circuit.

[0045] Specifically, after determining that the pixel point P is a corner point, the direction assignment module calculates the grayscale centroid around the pixel point P and uses the direction of the centroid as the main direction of the feature point.

[0046] In some embodiments, the main direction of the corner point can be calculated by the following formula: ; The meaning of this formula is to traverse several pixels around the pixel point P coordinate (x, y), accumulate the grayscale value of each pixel, and obtain the coordinates of the grayscale centroid P_I (x_I, y_I), It is the angle between the grayscale centroid P_I and the pixel point P.

[0047] In some embodiments, the function of the direction allocation module can be implemented by the following steps: 1. Using a multiplier and an adder circuit to calculate the grayscale centroid; 2. Using a CORDIC (Coordinate Rotation Digital Computer) circuit to calculate the angle. In other words, the direction allocation module of this embodiment can be obtained by designing a multiplier, an adder, and a CORDIC circuit.

[0048] Specifically, after determining all corner points in the environment image, the descriptor generation module generates a feature descriptor for each corner point, selects a set of pixel pairs around each corner point, compares the grayscale values ​​of each pair of pixels, and generates a binary string, which is the feature descriptor of the corner point.

[0049] In some embodiments, the feature descriptor of the corner point can be calculated by the following formula: .

[0050] In some embodiments, the function of the descriptor generation module can be implemented by the following steps: using a comparator circuit to compare pixel pairs around the feature point to generate binary bits. In other words, the descriptor generation module of this embodiment can be obtained by designing a pixel pair comparison circuit.

[0051] It is worth noting that the data processing hardware in this embodiment may include other modules in addition to the feature detection module, the direction assignment module and the descriptor generation module, and this embodiment does not specifically limit the number and types of modules included in the data processing hardware. For example, the data processing hardware may also include an image preprocessing module, which is used to preprocess the environment image to remove noise data in the environment image.

[0052] The embodiments of the present application can reduce computing delays and improve data processing speed by introducing various hardware circuits to process image data, which is particularly important for real-time systems. It can reduce dependence on the central processing unit or graphics processing unit, thereby reducing the energy consumption and cost of the overall system.

[0053] Step 103: Obtain depth information of the environment image according to the feature points. The depth information is used to represent the distance between the obstacle and the movable device.

[0054] In some embodiments, the movable device includes a camera for capturing an environmental image; initial depth information of the environmental image is obtained based on feature points, including: matching feature points of environmental images of adjacent frames captured by the camera to obtain a pose transformation matrix of the camera; obtaining coordinates of the feature points in a world coordinate system based on the pose transformation matrix and an intrinsic parameter matrix of the camera; wherein the initial depth information includes the feature points and their coordinates in the world coordinate system.

[0055] In some embodiments, after obtaining the feature point, the pixel coordinates of the feature point in the environment image can be obtained. It is understandable that due to the optical structure and imaging system of the camera, any point in its field of view can be mapped to the pixel coordinate system of the image. The expression is as follows, indicating that the point in other coordinate systems (such as the camera coordinate system) is converted to the pixel coordinate system through the imaging model of the camera. The imaging model here is called the intrinsic parameter matrix K, which is obtained by camera calibration.

[0056] ;in,( , ) is the pixel coordinate, is the internal parameter matrix, ( , , ) is the coordinate in the camera coordinate system. Therefore, the coordinate in the camera coordinate system can be obtained according to the above formula.

[0057] Since the movement of the movable device will drive the camera to move, and the camera movement will cause the captured image to change, the feature points of the environment images of adjacent frames taken by the camera are matched, that is, the positions of the same obstacle in the environment images of adjacent frames are matched, so as to obtain the camera's posture result, that is, the rotation matrix R and the translation vector t. The rotation matrix R and the translation vector t describe the transformation between the camera coordinate systems.

[0058] Therefore, starting from the world coordinate system, the position and posture of any point in any other coordinate system in space can be transformed by a posture transformation matrix T, which is expressed as follows. The posture transformation matrix T consists of the rotation matrix R and the translation vector t, that is, T= ; The pose transformation matrix T is also called the camera's external parameter matrix.

[0059] ;in, is the pose transformation matrix, ( , , ) is the coordinate in the world coordinate system. Therefore, according to the above formula, the coordinate in the camera coordinate system can be converted to the coordinate in the world coordinate system.

[0060] Step 104: Plan a motion path for the movable device according to the depth information.

[0061] In some embodiments, before planning the motion path of the movable device based on the depth information, the depth information is also smoothed and denoised to improve the accuracy of depth estimation, thereby improving the reliability of the planned path obtained based on the smoothed and denoised depth information.

[0062] For ease of understanding, the automatic obstacle avoidance process based on the mobile device of this embodiment is specifically described below: The movable device includes a monocular camera, a CPU (Central Processing Unit) and data processing hardware, and the data processing hardware is used to process the environmental image. During the movement of the movable device, the monocular camera captures the environmental image, and the CPU inputs the environmental image into the data processing hardware. The data processing hardware calculates feature points based on the environmental image and sends the feature points to the CPU. The CPU obtains the depth information of the environmental image based on the feature points and the internal and external parameter matrices of the monocular camera, and the CPU plans the flight path of the movable device based on the depth information.

[0063] It is worth mentioning that, from the automatic obstacle avoidance process of the above-mentioned movable device, it can be seen that the extraction of feature points is performed in the data processing hardware, thereby greatly reducing the CPU resource usage.

[0064] Compared with the related art, the embodiments of the present application have at least the following advantages: the environmental image is input into the data processing hardware to obtain the feature points of the obstacles in the environmental image. In other words, the step of extracting the feature points in the environmental image is implemented in the data processing hardware, thereby reducing the computing resource consumption of the central processor of the mobile device when processing obstacle avoidance, and enabling accurate obstacle avoidance of the mobile device. In addition, during the movement of the mobile device, it is only necessary to capture the environmental image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the mobile device.

[0065] Please refer to Figure 2 , which is a flow chart of an automatic obstacle avoidance method based on a mobile device provided in an embodiment of the present application. This embodiment is a further improvement on the previous embodiment, and the main improvement is that: in this embodiment, the mobile device includes a drone, and the environment image is also processed before inputting the environment image into the data processing hardware to reduce the computing resource consumption of the data processing hardware.

[0066] The specific process of this embodiment is as follows Figure 2 As shown, the following steps are included: Step 201: Obtain the flight speed and flight attitude data of the UAV.

[0067] In some embodiments, the flight speed is not only the specific value of the speed, but also the determination of the direction of the speed. The flight attitude data is the state of the three axes of the drone in the air relative to a reference line or a reference plane, or a fixed coordinate system. The drone has various flight attitudes, and the flight attitude determines the direction of the aircraft, affecting both the flight altitude and the direction of flight. The maneuverability of the drone is related to the flight performance and flight direction of the drone. The drone can capture images with a larger field of view in the flight direction during flight, but the speed, flight performance and other factors of the drone restrict the range of the area that the drone can reach. Therefore, not all areas in the image will affect the safety performance of the drone.

[0068] Step 202: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0069] Specifically, the flight channel area includes boundary areas in vertical and horizontal directions.

[0070] In some embodiments, the flight speed includes at least a flight speed value and a speed direction, and the flight attitude data includes at least a lateral acceleration critical upper limit value and a pitch angle; the flight channel area under the target camera corresponding to the flight direction of the UAV is determined in the following manner: the flight channel boundary in the horizontal direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value and pixel parameters of the target camera; the flight channel boundary in the vertical direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value, pitch angle and pixel parameters of the target camera.

[0071] In some embodiments, the flight channel boundary in the horizontal direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value and pixel parameters of the target camera, including: determining a first lateral displacement of the UAV in the horizontal direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining a first forward displacement of the UAV in the horizontal direction of the world coordinate system according to the speed value and the preset time; determining a direction angle of the UAV in the horizontal direction according to the first lateral displacement and the first forward displacement; determining the flight channel boundary in the horizontal direction according to the direction angle in the horizontal direction, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0072] It is worth noting that there is no limitation on how to determine the above-mentioned flight speed value, speed direction, lateral acceleration critical upper limit value and preset time in this embodiment. The main purpose is to establish the angle and pixel relationship based on the pixel parameters of the drone camera and the area captured by the drone camera according to the coordinate system, where the angle can be determined based on the flight speed value and speed direction of the drone. At this time, according to the angle and pixel relationship, the left and right boundary ranges that the drone can actually reach can be determined.

[0073] In some embodiments, the flight channel boundary in the vertical direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value, pitch angle and pixel parameters of the target camera, including: determining the second lateral displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining the second forward displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value and the preset time; determining the direction angle of the UAV in the vertical direction according to the second lateral displacement and the second forward displacement; determining the flight channel boundary in the vertical direction according to the direction angle in the vertical direction, the pitch angle, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0074] It should be noted that the process of determining the flight channel boundary in the vertical direction is similar to the process of determining the flight channel boundary in the horizontal direction. Since the pitch angle parameter is introduced in the vertical direction in the world coordinate system, and the pitch angle is related to the speed, the algorithm for determining the flight channel boundary in the horizontal direction in the above embodiment is based on the relationship between the angle corresponding to the pitch angle and the pixel, that is, the upper and lower boundary ranges that the drone can actually reach are determined.

[0075] Step 203: Collect environmental images during the flight of the drone.

[0076] Step 204: performing object segmentation on the environment image to segment the environment image into a plurality of object segmentation regions.

[0077] It should be noted that how to perform target segmentation on the environment image is described in detail in subsequent embodiments, and will not be described again here to avoid repetition.

[0078] Step 205: Determine the obstacle avoidance area of ​​the UAV according to the multiple target segmentation areas and the flight channel area.

[0079] Step 206: Input the obstacle avoidance area into the data processing hardware, so that the data processing hardware calculates feature points for characterizing obstacles in the obstacle avoidance area according to the obstacle avoidance area.

[0080] Step 207: Obtain depth information of the obstacle avoidance area according to the feature points. The depth information is used to characterize the distance between the obstacle and the drone.

[0081] Step 208: Plan the motion path of the UAV according to the depth information.

[0082] Steps 206 to 208 of this embodiment are similar to steps 102 to 104 of the above embodiment, and will not be described again to avoid repetition.

[0083] Compared with the related art, the embodiment of the present application has at least the following advantages: the environmental image is input into the data processing hardware to obtain the feature points of the obstacles in the environmental image. In other words, the step of extracting the feature points in the environmental image is implemented in the data processing hardware, thereby reducing the computing resource consumption of the central processor of the drone when processing obstacle avoidance, and enabling the drone to achieve accurate obstacle avoidance. In addition, during the movement of the drone, it is only necessary to collect the environmental image through the camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0084] Please refer to Figure 3 , is a flow chart of an automatic obstacle avoidance method based on a mobile device provided in an embodiment of the present application. This embodiment is a specific description of the above embodiment, mainly describing how to perform target segmentation on an environment image to segment the environment image into multiple target segmentation areas.

[0085] The specific process of this embodiment is as follows Figure 3 As shown, the following steps are included: Step 301: Obtain the flight speed and flight attitude data of the UAV.

[0086] Step 302: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0087] Step 303: Collect environmental images during the flight of the drone.

[0088] Step 304: Input the environment image into a pre-trained image segmentation network model, and determine multiple target segmentation areas based on the output results of the image segmentation network model.

[0089] Specifically, the image segmentation network model is used to output the annotation information of the objects in each target segmentation area, and the annotation information includes at least the size information of the object and the position information in the environment image. It is worth noting that the objects in the environment image include obstacles and non-obstacles. For example, the objects in the environment image can be tree crowns and the sky, the tree crown is an obstacle, and the sky is a non-obstacle. By outputting the annotation information of the object through the image segmentation network model, it is possible to determine whether the object is an obstacle based on the annotation information.

[0090] In some embodiments, the image segmentation network model is trained according to the following steps: obtaining historical environment images of the drone during historical flights; annotating historical objects in the historical environment images to obtain training images with real labels, wherein the real labels include at least the size of the historical objects and the position in the historical environment images; and training the initial image segmentation network model using the training images to obtain an image segmentation network model.

[0091] In some embodiments, since the computing power of the embedded chip selected by the drone platform is relatively weak, a lightweight neural network model needs to be selected to ensure the real-time requirements of image segmentation. Therefore, the image segmentation network model can be MobileNetV2 combined with U-Net. MobileNetV2 can efficiently extract image features, and combined with the U-Net structure, image segmentation can be performed quickly.

[0092] In some embodiments, the size of the image segmentation network model is further reduced and the running speed is improved through model quantization, pruning and other technologies. Taking model quantization as an example, the model parameters of the image segmentation network model are converted into fixed-point representation according to a preset quantization calibration table, wherein the quantization calibration table includes at least the number of quantization bits, quantization method and quantization range. By quantizing the image segmentation network model, the storage requirements of the image segmentation network model can be reduced, while the inference running speed on the NPU embedded side can be improved: model quantization converts floating-point model parameters into fixed-point representation, thereby reducing the space occupied by the image segmentation network model on the storage medium, memory bandwidth and memory overhead; at the same time, the quantized model can take advantage of the higher computational efficiency of integer operations on the NPU compared to floating-point operations, thereby achieving more efficient calculations on embedded hardware.

[0093] In some embodiments, the trained image segmentation network model is deployed on the NPU chip of the drone, and the NPU is used to perform image segmentation processing.

[0094] For ease of understanding, the training process of the image segmentation network model of this embodiment is specifically described below: (1) Collect a large number of object images during drone flight, mark the precise locations of the objects in the object images, and construct a training set.

[0095] (2) Perform noise processing such as distortion, occlusion, motion blur, and defect on the above-mentioned images of labeled objects to supplement the training set.

[0096] (3) Use the above training set to train the image segmentation network model to obtain the first weight of the image segmentation network model.

[0097] (4) Repeat steps (1) to (3) until the image segmentation network model converges.

[0098] Step 305: Determine the obstacle avoidance area of ​​the UAV according to the multiple target segmentation areas and the flight channel area.

[0099] From the above description, it can be seen that the image segmentation network model is used to output the annotation information of objects in each target segmentation area. The annotation information at least includes the size information of the object and the position information in the environmental image. Therefore, obstacles in the flight channel area can be determined through the annotation information, and then the area with obstacles can be used as an obstacle avoidance area.

[0100] Step 306: Input the obstacle avoidance area into the data processing hardware, so that the data processing hardware calculates feature points for characterizing obstacles in the obstacle avoidance area according to the obstacle avoidance area.

[0101] Step 307: Obtain depth information of the obstacle avoidance area according to the feature points. The depth information is used to characterize the distance between the obstacle and the drone.

[0102] Step 308: Plan the motion path of the UAV according to the depth information.

[0103] Steps 301 to 303 and 305 to 308 of this embodiment are similar to steps 201 to 203 and 205 to 208 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0104] Compared with the related art, the embodiment of the present application has at least the following advantages: the environmental image is input into the data processing hardware to obtain the feature points of the obstacles in the environmental image. In other words, the step of extracting the feature points in the environmental image is implemented in the data processing hardware, thereby reducing the computing resource consumption of the central processor of the drone when processing obstacle avoidance, and enabling the drone to achieve accurate obstacle avoidance. In addition, during the movement of the drone, it is only necessary to collect the environmental image through the camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0105] Please refer to Figure 4 , which is a flow chart of an automatic obstacle avoidance method based on a mobile device provided in an embodiment of the present application. This embodiment is a further improvement on the aforementioned embodiment, and the main improvement is that: in this embodiment, the priority of each target segmentation area is also obtained, and the target segmentation area whose priority meets the preset priority screening rule is used as the final segmentation area, and then the obstacles in the flight channel area are determined based on the final segmentation area. In this way, while improving the accuracy of the obstacle avoidance area, the flight path of the drone in the obstacle avoidance area can be reduced, thereby improving the flight efficiency of the drone.

[0106] The specific process of this embodiment is as follows Figure 4 As shown, the following steps are included: Step 401: Obtain the flight speed and flight attitude data of the UAV.

[0107] Step 402: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0108] Step 403: Collect environmental images during the flight of the drone.

[0109] Step 404: Perform object segmentation on the environment image to segment the environment image into a plurality of object segmentation regions.

[0110] Step 405: Obtain the priority of each target segmentation region, wherein the greater the priority of the target segmentation region, the smaller the shortest straight-line distance between the drone and the target segmentation region.

[0111] In some embodiments, the correspondence between priority and distance can be stored in the drone in advance. For example, the priorities are set from high to low: priority 1 to priority 5, the distance corresponding to priority 1 is 0 to 2 meters, the distance corresponding to priority 2 is 2 meters to 5 meters, the distance corresponding to priority 3 is 5 meters to 10 meters, the distance corresponding to priority 4 is 10 meters to 20 meters, and the distance corresponding to priority 5 is more than 20 meters. After obtaining multiple target segmentation areas, the drone calculates the shortest straight-line distance between the drone and each target segmentation area, and determines the priority corresponding to each target segmentation area based on the calculated shortest straight-line distance.

[0112] Step 406: The target segmentation region whose priority satisfies the preset priority screening rule is taken as the final segmentation region.

[0113] In some embodiments, the priority of each target segmentation area is sorted from large to small to obtain a priority queue; the target segmentation areas corresponding to the first N priorities in the priority queue are used as the final segmentation areas, where N is an integer greater than or equal to 1.

[0114] For ease of understanding, the following takes the case where the target segmentation regions are 6 and the value of N is 3 as an example to specifically describe how to obtain the final segmentation regions in this embodiment: 1. Preset the correspondence between priority and distance, and store the correspondence in the drone. For example, the correspondence can be set as follows: the priority is from high to low, from priority 1 to priority 5, the distance corresponding to priority 1 is greater than 0 and less than 2 meters; the distance corresponding to priority 2 is greater than or equal to 2 meters and less than 5 meters; the distance corresponding to priority 3 is greater than or equal to 5 meters and less than 10 meters; the distance corresponding to priority 4 is greater than or equal to 10 meters and less than 20 meters; the distance corresponding to priority 5 is greater than or equal to 20 meters.

[0115] 2. After the drone divides the environment image into 6 target segmentation areas, the shortest straight-line distance between each target segmentation area and the drone is calculated. Assume that the calculated shortest straight-line distances of the 6 target segmentation areas are: the shortest straight-line distance between target segmentation area 1 and the drone is 5 meters, the shortest straight-line distance between target segmentation area 2 and the drone is 15 meters, the shortest straight-line distance between target segmentation area 3 and the drone is 2 meters, the shortest straight-line distance between target segmentation area 4 and the drone is 6 meters, the shortest straight-line distance between target segmentation area 5 and the drone is 12 meters, and the shortest straight-line distance between target segmentation area 6 and the drone is 20 meters.

[0116] 3. Based on the preset correspondence between priority and distance, the drone obtains that the priority of target segmentation area 1 is priority 2, the priority of target segmentation area 2 is priority 4, the priority of target segmentation area 3 is priority 1, the priority of target segmentation area 4 is priority 2, the priority of target segmentation area 5 is priority 4, and the priority of target segmentation area 6 is priority 5.

[0117] 4. The UAV determines that the target segmentation areas corresponding to the first three priorities are: target segmentation area 1, target segmentation area 3 and target segmentation area 4. The UAV uses target segmentation area 1, target segmentation area 3 and target segmentation area 4 as the final segmentation areas.

[0118] Step 407: According to the final segmented area, obstacles in the flight channel area are determined, and the area with obstacles in the flight channel area is used as an obstacle avoidance area.

[0119] Step 408: Input the obstacle avoidance area into the data processing hardware, so that the data processing hardware extracts feature points for characterizing obstacles in the obstacle avoidance area according to the obstacle avoidance area.

[0120] Step 409: Obtain the depth information of the obstacle avoidance area according to the feature points. The depth information is used to characterize the distance between the obstacle and the drone.

[0121] Step 410: Plan the motion path of the UAV according to the depth information.

[0122] Compared with the related art, the embodiment of the present application has at least the following advantages: the environmental image is input into the data processing hardware to obtain the feature points of the obstacles in the environmental image. In other words, the step of extracting the feature points in the environmental image is implemented in the data processing hardware, thereby reducing the computing resource consumption of the central processor of the drone when processing obstacle avoidance, and enabling the drone to achieve accurate obstacle avoidance. In addition, during the movement of the drone, it is only necessary to collect the environmental image through the camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0123] Please refer to Figure 5 , is a schematic diagram of the hardware structure of the electronic device 1000 provided in the embodiment of the present application. Figure 5 As shown, the electronic device 1000 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the above automatic obstacle avoidance method based on a movable device in the electronic device 1000.

[0124] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 1000. In other embodiments, the electronic device 1000 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0125] The processor 1001 may include one or more processing units, for example, the processor 1001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0126] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 1001 has just used or circulated. If the processor 1001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.

[0127] In some embodiments, the processor 1001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0128] In some embodiments, processor 1001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0129] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0130] This embodiment also provides a computer-readable storage medium, which stores computer instructions. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the automatic obstacle avoidance method based on a movable device in the above-mentioned embodiment.

[0131] Among them, the electronic device and computer storage medium provided in this embodiment are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0132] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0133] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0137] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. An automatic obstacle avoidance method based on a movable device, characterized in that: The mobile device includes a central processing unit and data processing hardware; The automatic obstacle avoidance method is applied to the central processor, comprising: Collecting environmental images of the movable device during its movement; Inputting the environment image into the data processing hardware so that the data processing hardware extracts feature points for characterizing obstacles in the environment image according to the environment image, wherein the central processing unit receives the feature points extracted by the data processing hardware; Obtaining depth information of the environment image according to the feature points, wherein the depth information is used to represent the distance between the obstacle and the movable device; A motion path is planned for the movable device according to the depth information.

2. The automatic obstacle avoidance method based on a movable device according to claim 1, characterized in that: The data processing hardware at least includes a feature detection module, a direction assignment module and a descriptor generation module; The feature points include key points and feature descriptors, wherein the key points are pixels with geometric features in the environment image, and the feature descriptors are used to describe the difference between the key points and the pixels around the key points; The step of inputting the environment image into the data processing hardware so that the data processing hardware calculates feature points for characterizing obstacles in the environment image according to the environment image, comprises: The environmental image is input into the data processing hardware, wherein the feature detection module is used to calculate the key points in the environmental image, the direction assignment module is used to calculate the main direction of the key points according to the calculation results of the feature detection module, and the descriptor generation module is used to calculate the feature descriptor according to the calculation results of the feature detection module.

3. The automatic obstacle avoidance method based on a movable device according to claim 2, characterized in that: The feature detection module includes a comparator and an integral image calculation circuit, wherein the comparator is used to compare the grayscale values ​​of each pixel in the environment image with the surrounding pixels and determine the corner points from each pixel; the integral image calculation circuit is used to calculate the response value of the corner point; The direction allocation module includes a multiplier, an adder, and a coordinate rotation digital calculation circuit, wherein the multiplier and the adder are used to calculate the grayscale centroid of each pixel in the environment image, and the coordinate rotation digital calculation circuit is used to calculate the angle of the line between the grayscale centroid and the corner point; The descriptor generation module includes a pixel pair comparison circuit, which is used to compare the grayscale values ​​of the pixel pairs around the corner point and generate the feature descriptor according to the comparison result.

4. The automatic obstacle avoidance method based on a movable device according to claim 1, characterized in that: The movable device includes a drone; Before inputting the environment image into the data processing hardware, the method further includes: Obtaining the flight speed and flight attitude data of the UAV; Determine, according to the flight speed and the flight attitude data, a flight channel area under the target camera corresponding to the flight direction of the UAV, wherein the flight channel area includes boundary areas in vertical and horizontal directions; Performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation regions, wherein each of the target segmentation regions includes at least one object in the environment image; Determine an obstacle avoidance area of ​​the UAV according to the plurality of target segmentation areas and the flight channel area; The step of inputting the environment image into the data processing hardware comprises: The obstacle avoidance area is input into the data processing hardware.

5. The automatic obstacle avoidance method based on a movable device according to claim 4, characterized in that: The performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation areas includes: Inputting the environment image into a pre-trained image segmentation network model, and determining a plurality of target segmentation regions based on an output result of the image segmentation network model; The image segmentation network model is used to output the annotation information of the object in each target segmentation area, and the annotation information at least includes the size information of the object and the position information in the environment image.

6. The automatic obstacle avoidance method based on a movable device according to claim 5, characterized in that: The image segmentation network model is trained according to the following steps: Obtain historical environmental images of drones during historical flights; Labeling the historical objects in the historical environment image to obtain a training image with a real label, wherein the real label includes at least the size of the historical object and the position in the historical environment image; The initial image segmentation network model is trained using the training images to obtain the image segmentation network model.

7. The automatic obstacle avoidance method based on a movable device according to claim 4, characterized in that: The determining the obstacle avoidance area of ​​the UAV according to the plurality of target segmentation areas and the flight channel area comprises: Obtaining the priority of each target segmentation area, wherein the greater the priority of the target segmentation area, the smaller the shortest straight-line distance between the UAV and the target segmentation area; The target segmentation area whose priority satisfies a preset priority screening rule is taken as a final segmentation area; Determining obstacles within the flight path area according to the final segmented area; The area with obstacles in the flight channel area is used as the obstacle avoidance area.

8. The automatic obstacle avoidance method based on a movable device according to claim 1, characterized in that: Before planning a motion path for the movable device according to the depth information, the method further includes: Performing smoothing and denoising on the depth information; The performing motion path planning for the movable device according to the depth information includes: The motion path of the movable device is planned according to the final depth information after smoothing and denoising.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the automatic obstacle avoidance method based on a movable device as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The invention comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the automatic obstacle avoidance method based on a movable device as claimed in any one of claims 1 to 8.

Images