Control method of mobile equipment and related equipment

Abstract

The invention provides a control method of a mobile device and a related device. The method comprises the following steps: acquiring an environment image in a movement process of the mobile device; initial depth information of the environment image is acquired, and the initial depth information is used for representing the distance between a part of objects in the environment image and the mobile device; inputting the environment image and the initial depth information into a first deep learning model to obtain final depth information of the environment image; wherein the final depth information is used for representing distances between all objects in the environment image and the mobile device, and the first deep learning model is used for fusing the environment image and the initial depth information; and performing obstacle detection on the environment image according to the final depth information, and performing motion path planning on the movable equipment based on an obstacle detection result. According to the method and the device, the computing resources for obstacle avoidance can be reduced, accurate obstacle avoidance of the mobile equipment can be realized, and the obstacle avoidance cost can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of visual obstacle avoidance, and in particular to a control method for a movable device and related devices. Background Art

[0002] Obstacle avoidance technology is the key to ensuring the movement safety of movable devices and achieving autonomous navigation. Taking the movable device as a drone as an example, the mainstream drone obstacle avoidance solutions can be roughly divided into sensor-based solutions and vision-based solutions. Sensor-based solutions: including laser radar (LiDAR), sonar, infrared sensors, etc. These solutions can provide direct distance measurement and have high obstacle avoidance accuracy and reliability. Vision-based solutions: mainly use cameras to capture images and achieve obstacle avoidance through image processing technology. Compared with sensor solutions, vision solutions are low-cost, light-weight, and more suitable for small drones.

[0003] However, high-precision sensors are expensive and increase the load of drones, affecting flight time and flexibility. Vision solutions are sensitive to lighting conditions and environmental complexity, require complex algorithms to ensure accuracy, and have high demands on computing resources. Summary of the invention

[0004] In view of this, the present application provides a control method for a movable device and related devices, which can achieve accurate obstacle avoidance of the movable device while reducing the computing resources used for obstacle avoidance, and can reduce the obstacle avoidance cost.

[0005] An embodiment of the present application provides a control method for a movable device, comprising: collecting an environmental image during the movement of the movable device; obtaining initial depth information of the environmental image, wherein the initial depth information is used to characterize the distance between some objects in the environmental image and the movable device; inputting the environmental image and the initial depth information into a first deep learning model to obtain final depth information of the environmental image; wherein the final depth information is used to characterize the distance between all objects in the environmental image and the movable device, and the first deep learning model is used to fuse the environmental image and the initial depth information; performing obstacle detection on the environmental image according to the final depth information, and then planning a motion path for the movable device based on the result of the obstacle detection.

[0006] Compared with the related art, the embodiment of the present application has at least the following advantages: the environment image and the initial depth information are input into the first deep learning model to obtain the final depth information of the environment image. On the one hand, the first deep learning model can fuse the environment image and the initial depth information so that the final depth information obtained can represent the distance between all objects in the environment image and the movable device, so that the motion path planned based on the final depth information is highly accurate and can achieve accurate obstacle avoidance of the movable device; on the other hand, the step of fusing the environment image and the initial depth information is implemented in the first deep learning model, which reduces the computing resources of the central processor of the movable device when processing obstacle avoidance. In addition, during the movement of the movable device, it is only necessary to collect the environment image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the movable device.

[0007] In some possible implementations, obtaining the initial depth information of the environmental image includes: inputting the environmental image into a second deep learning model to obtain feature points for characterizing the partial object, wherein the second deep learning model is trained by historical environmental images collected by the movable device during historical movement; and obtaining the initial depth information based on the feature points.

[0008] By adopting the above technical solution, the environment image is input into the second deep learning model to obtain the feature points of some obstacles in the environment image. In other words, the step of extracting the feature points in the environment image is implemented in the second deep learning model, thereby further reducing the computing resources of the central processor of the mobile device when processing obstacle avoidance.

[0009] In some possible implementations, the movable device includes a camera for photographing the environment image; obtaining the initial depth information based on the feature points includes: matching feature points of adjacent frames of the environment image and the environment image to obtain a pose transformation matrix; obtaining coordinates of the feature points in a world coordinate system based on the pose transformation matrix and an intrinsic parameter matrix of the camera; the feature points and the coordinates of the feature points in the world coordinate system constitute the initial depth information.

[0010] By adopting the above technical solution, it is ensured that the obtained initial depth information can characterize the distance between some objects and the movable device, thereby improving the accuracy of the final depth information output by the first deep learning model, and further realizing accurate obstacle avoidance of the movable device.

[0011] In some possible implementations, the second deep learning model is trained in the following manner: performing noise processing on the historical environment image to obtain a preset amount of noise data; performing target detection on the historical environment image to obtain a final historical environment image with a label; training the original second deep learning model based on the final historical environment image and the noise data; updating the first weight of the original second deep learning model based on the training result of the original second deep learning model until the original second deep learning model converges, and using the converged original second deep learning model as the second deep learning model.

[0012] By adopting the above technical solution, the model accuracy of the second deep learning model can be ensured, thereby ensuring the accuracy of the feature points output by the second deep learning model.

[0013] In some possible implementations, the first deep learning model is trained in the following manner: performing data enhancement on initial training data to obtain target training data; wherein the initial training data includes historical environmental images collected by the movable device during historical movement, and historical feature points corresponding to the historical environmental images; inputting the target training data into the original first deep learning model, and calculating the loss value of the loss function corresponding to the original first deep learning model; iteratively updating the second weight of the original first deep learning model based on the loss gradient of the loss function to reduce the loss value to meet the preset accuracy requirements; and using the second weight corresponding to the final loss value as the final weight of the first deep learning model, wherein the first deep learning model includes a jump connection structure and a fusion module, the jump connection structure is used to directly transmit the initial depth information to the output layer of the first deep learning model, and the fusion module is used to fuse the initial depth information with the environmental image to obtain the final depth information.

[0014] By adopting the above technical solution, the model accuracy of the first deep learning model can be ensured, thereby ensuring the accuracy of the final depth information output by the first deep learning model.

[0015] In some possible implementations, before obtaining the initial depth information of the environmental image, it also includes: obtaining a flight channel area corresponding to the movement direction of the movable device, wherein the flight channel area includes boundary areas in vertical and horizontal directions; performing target segmentation on the environmental image to segment the environmental image into multiple target segmentation areas, wherein each of the target segmentation areas includes at least one object in the environmental image; determining an obstacle avoidance area of ​​the movable device based on the multiple target segmentation areas and the flight channel area; obtaining the initial depth information of the environmental image includes: obtaining the initial depth information of the obstacle avoidance area.

[0016] By adopting the above technical solution, data processing can be performed only based on the obstacle avoidance area, which reduces the amount of data and thus reduces the consumption of computing data, while improving data processing accuracy and achieving accurate obstacle avoidance for movable equipment.

[0017] In some possible implementations, performing target segmentation on the environmental image to divide the environmental image into multiple target segmentation areas includes: inputting the environmental image into a pre-trained image segmentation network model, and determining the multiple target segmentation areas based on the output result of the image segmentation network model; wherein the image segmentation network model is used to output labeling information of objects in each of the target segmentation areas, and the labeling information includes at least size information of the object and position information in the environmental image.

[0018] By adopting the above technical solution, it is possible to ensure that there is at least one object in each target segmentation area, thereby improving the accuracy of the obstacle avoidance area determined subsequently.

[0019] The second aspect of the present application discloses a control device for a movable device, comprising: an acquisition module, a first processing module and a second processing module; wherein the second processing module comprises a first deep learning model; the acquisition module is used to acquire an environmental image during the movement of the movable device; the first processing module is used to obtain initial depth information of the environmental image, wherein the initial depth information is used to characterize the distance between some objects in the environmental image and the movable device; the first processing module is also used to input the environmental image and the initial depth information into the first deep learning model; the first deep learning model is used to fuse the environmental image and the initial depth information, and output the final depth information of the environmental image; wherein the final depth information is used to characterize the distance between all objects in the environmental image and the movable device; the first processing module is also used to perform obstacle detection on the environmental image according to the final depth information, and then plan the motion path of the movable device based on the result of the obstacle detection.

[0020] The third aspect of the present application discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the above-mentioned control method for a movable device.

[0021] The fourth aspect of the present application discloses a storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned control method for a movable device.

[0022] It can be understood that the device of the second aspect, the electronic device of the third aspect and the storage medium of the fourth aspect provided above all correspond to the method of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a control method of a movable device according to an embodiment of the present invention.

[0024] Figure 2 is a flow chart of a control method of a movable device according to an embodiment of the present invention.

[0025] Figure 3 is a flow chart of a control method of a movable device according to an embodiment of the present invention.

[0026] Figure 4 is a flow chart of a control method of a movable device according to an embodiment of the present invention.

[0027] Figure 5 This is an application scenario diagram of image segmentation according to an embodiment of the present invention.

[0028] Figure 6 1 is a schematic diagram of functional modules of a control device for a movable device according to an embodiment of the present invention.

[0029] Figure 7 2 is a schematic diagram of functional modules of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0033] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0034] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0035] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0036] Please refer to Figure 1 , is a flow chart of a control method for a mobile device provided in an embodiment of the present application. Figure 1 As shown, this embodiment is applied to a central processor of a mobile device, and includes the following steps: Step 101: Collect environmental images during the movement of the movable device.

[0037] In some embodiments, the movable device includes but is not limited to a drone, a vehicle, etc. This embodiment does not specifically limit the type of the movable device.

[0038] In some embodiments, a photographing device is provided on the movable device, and the environmental image during the movement of the movable device is collected by the photographing device.

[0039] Taking the movable device as a drone as an example, the shooting device is preferably a monocular camera. The monocular camera has the advantages of small size, light weight, low energy consumption, and wide field of view, and is more suitable for obstacle avoidance perception of small drones.

[0040] Step 102: Acquire initial depth information of the environment image, wherein the initial depth information is used to characterize the distance between some objects in the environment image and the movable device.

[0041] In some embodiments, the method of obtaining initial depth information includes: inputting the environmental image into a second deep learning model to obtain feature points for characterizing some objects, wherein the second deep learning model is trained by historical environmental images collected by the movable device during historical movement; and obtaining initial depth information based on the feature points.

[0042] In some embodiments, the second deep learning model is trained by the historical environment images of the mobile device during the historical movement. The specific training process of the second deep learning model is described in detail in the subsequent embodiments. In order to avoid repetition, it will not be repeated here. In addition, the process of how to obtain the initial depth information based on the feature points will also be described in detail in the subsequent embodiments. In order to avoid repetition, it will not be repeated here.

[0043] Specifically, feature points include key points and feature descriptors. Key points are points with significant geometric features in the point cloud corresponding to the environment image. These points are usually points with high curvature or large local curvature changes. They may be inflection points, edges, surface bumps, etc. in the point cloud. Feature descriptors are tools used to describe the geometric shape and structure of local areas in point clouds.

[0044] It should be noted that, from the above description, key points are points with higher curvature or larger local curvature changes. Therefore, the objects represented by feature points are objects with significant geometric features in the environment image, that is, some objects in the environment image.

[0045] In some embodiments, the movable device includes a camera for capturing an environmental image; initial depth information of the environmental image is obtained based on feature points, including: matching feature points of environmental images of adjacent frames captured by the camera to obtain a pose transformation matrix of the camera; obtaining coordinates of the feature points in a world coordinate system based on the pose transformation matrix and an intrinsic parameter matrix of the camera; wherein the initial depth information includes the feature points and their coordinates in the world coordinate system.

[0046] Step 103: Input the environment image and the initial depth information into the first deep learning model to obtain final depth information of the environment image; wherein the final depth information is used to characterize the distance between all objects in the environment image and the movable device.

[0047] Specifically, the first deep learning model is used to fuse the environment image and the initial depth information. The specific training process of the first deep learning model is described in detail in the subsequent embodiments, and is not repeated here for the sake of example to avoid repetition.

[0048] Step 104: perform obstacle detection on the movable device according to the final depth information, and then plan a motion path for the movable device based on the result of the obstacle detection.

[0049] In some embodiments, before planning the motion path of the movable device based on the final depth information, the final depth information is also smoothed and denoised, thereby improving the accuracy of depth estimation, and further improving the reliability of the planned path obtained based on the final depth information after smoothing and denoising.

[0050] For ease of understanding, the following takes the mobile device as a drone as an example to specifically describe the automatic obstacle avoidance process based on deep learning in this embodiment: The drone includes a monocular camera, a CPU (Central Processing Unit) and an NPU (Network Processor Unit), and the NPU is used to call a first deep learning model and a second deep learning model. During the flight of the drone, the monocular camera captures an environmental image, the CPU inputs the environmental image into the first deep learning model, and the NPU calls the second deep learning model so that the second deep learning model outputs feature points of the environmental image. The NPU sends the feature points to the CPU, and the CPU obtains initial depth information of the environmental image based on the feature points, that is, sparse depth information. The CPU inputs the environmental image and the initial depth information into the first deep learning model, and the NPU calls the first deep learning model so that the first deep learning model fuses the environmental image and the initial depth information and outputs the final depth information of the environmental image, that is, dense depth information. The NPU sends the final depth information to the CPU, and the CPU performs obstacle detection on the drone based on the final depth information, and then plans the motion path of the movable device based on the result of the obstacle detection.

[0051] It is worth mentioning that from the automatic obstacle avoidance process of the above-mentioned drone, it can be seen that the extraction of feature points and the fusion of the initial depth information of the environmental image are both performed in the NPU, which greatly reduces the CPU resource usage.

[0052] Compared with the related art, the embodiment of the present application has at least the following advantages: the environment image and the initial depth information are input into the first deep learning model to obtain the final depth information of the environment image. On the one hand, the first deep learning model can fuse the environment image and the initial depth information so that the final depth information obtained can represent the distance between all objects in the environment image and the movable device, so that the motion path planned based on the final depth information is highly accurate and can achieve accurate obstacle avoidance of the movable device; on the other hand, the step of fusing the environment image and the initial depth information is implemented in the first deep learning model, which reduces the computing resources of the central processor of the movable device when processing obstacle avoidance. In addition, during the movement of the movable device, it is only necessary to collect the environment image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the movable device.

[0053] Please refer to Figure 2 , which is a flow chart of a control method for a mobile device provided in an embodiment of the present application. This embodiment is a specific description of the above-mentioned embodiment, mainly describing: a specific training method of the second deep learning model, and how to obtain initial depth information based on feature points.

[0054] The specific process of this embodiment is as follows Figure 2 As shown, the following steps are included: Step 201: Perform noise processing on the historical environment image to obtain a preset amount of noise data.

[0055] In some embodiments, a preset amount of noise data is obtained by performing noise processing such as distortion, occlusion, motion blur, and defect on the historical environment image.

[0056] In some embodiments, the size of the preset number is not specifically limited and can be set according to the needs of actual applications.

[0057] It is worth noting that by using the noise data as the training data of the second deep learning model, the robustness of the subsequently trained second deep learning model can be improved.

[0058] Step 202: Perform target detection on the historical environment image to obtain a final historical environment image with a label.

[0059] Step 203: Train the original second deep learning model according to the final historical environment image and noise data, update the first weight of the original second deep learning model according to the training result of the original second deep learning model, until the original second deep learning model converges, and use the converged original second deep learning model as the second deep learning model.

[0060] For ease of understanding, the training process of the second deep learning model of this embodiment is specifically described below by taking the mobile device as a drone as an example: a. Collect a large amount of monocular image data from drone flights, use traditional image feature extraction algorithms such as ORB (Oriented Fast and Rotated BRIEF, computer vision features for image matching and recognition), SIFT (Scale-Invariant Feature Transform), and SURF (Speeded Up Robust Features) to extract and match feature points and build a training set.

[0061] b. Perform noise processing such as distortion, occlusion, motion blur, and defect on the above images containing feature points to supplement the training set.

[0062] c. Use the above training set to train a second deep learning model to obtain a first weight of the second deep learning model.

[0063] d. Repeat steps a to c until the second deep learning model converges.

[0064] Step 204: Collecting an environmental image of the movable device during its movement.

[0065] Step 205: Input the environment image into the second deep learning model to obtain feature points for characterizing some obstacles in the environment image.

[0066] Step 206: perform feature point matching on adjacent frames of the environment image and the environment image to obtain a posture transformation matrix.

[0067] Step 207: Obtain the coordinates of the feature points in the world coordinate system according to the posture transformation matrix and the intrinsic parameter matrix of the camera, wherein the feature points and their coordinates in the world coordinate system constitute initial depth information.

[0068] In some embodiments, after obtaining the feature point, the pixel coordinates of the feature point in the environment image can be obtained. It is understandable that due to the optical structure and imaging system of the camera, any point in its field of view can be mapped to the pixel coordinate system of the image. The expression is as follows, indicating that the point in other coordinate systems (such as the camera coordinate system) is converted to the pixel coordinate system through the imaging model of the camera. The imaging model here is called the intrinsic parameter matrix K, which is obtained by camera calibration.

[0069] ;in,( , ) is the pixel coordinate, is the internal parameter matrix, ( , , ) is the coordinate in the camera coordinate system. Therefore, the coordinate in the camera coordinate system can be obtained according to the above formula.

[0070] Since the movement of the movable device will drive the camera to move, and the camera movement will cause the captured image to change, the feature points of the environment images of adjacent frames taken by the camera are matched, that is, the positions of the same obstacle in the environment images of adjacent frames are matched, so as to obtain the camera's posture result, that is, the rotation matrix R and the translation vector t. The rotation matrix R and the translation vector t describe the transformation between the camera coordinate systems.

[0071] Therefore, starting from the world coordinate system, the position and posture of any point in any other coordinate system in space can be transformed by a posture transformation matrix T, which is expressed as follows. The posture transformation matrix T consists of the rotation matrix R and the translation vector t, that is, T= ; The pose transformation matrix T is also called the camera's external parameter matrix.

[0072] ;in, is the pose transformation matrix, ( , , ) is the coordinate in the world coordinate system. Therefore, according to the above formula, the coordinate in the camera coordinate system can be converted to the coordinate in the world coordinate system.

[0073] Step 208: Input the environment image and the initial depth information into the first deep learning model to obtain final depth information of the environment image; wherein the final depth information is used to characterize the distance between all objects in the environment image and the movable device.

[0074] Step 209: performing obstacle detection on the movable device according to the final depth information, and then performing motion path planning for the movable device based on the result of the obstacle detection.

[0075] Compared with the related art, the embodiment of the present application has at least the following advantages: the environment image and the initial depth information are input into the first deep learning model to obtain the final depth information of the environment image. On the one hand, the first deep learning model can fuse the environment image and the initial depth information so that the final depth information obtained can represent the distance between all objects in the environment image and the movable device, so that the motion path planned based on the final depth information is highly accurate and can achieve accurate obstacle avoidance of the movable device; on the other hand, the step of fusing the environment image and the initial depth information is implemented in the first deep learning model, which reduces the computing resources of the central processor of the movable device when processing obstacle avoidance. In addition, during the movement of the movable device, it is only necessary to collect the environment image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the movable device.

[0076] Please refer to Figure 3 , which is a flow chart of a control method for a mobile device provided in an embodiment of the present application. This embodiment is a specific description of the above embodiment, and further describes: a specific training process of the first deep learning model.

[0077] The specific process of this embodiment is as follows Figure 3 As shown, the following steps are included: Step 301: Perform data enhancement on the initial training data to obtain target training data; wherein the initial training data includes a historical environment image and historical feature points corresponding to the historical environment image.

[0078] It is worth noting that by adopting data enhancement technology, that is, rotating, scaling, and color adjusting the initial training data, the generalization ability of the first deep learning model trained subsequently can be improved.

[0079] In some embodiments, a 2D image and a corresponding 3D depth map can be simultaneously captured by an RGB-D camera, and points can be marked in the 2D image by using ORB and SURF feature point detection methods to obtain initial training data.

[0080] Step 302: Input the target training data into the original first deep learning model, and calculate the loss value of the loss function corresponding to the original first deep learning model.

[0081] In some embodiments, the loss function is a hybrid loss function; the loss value includes pixel error and structural similarity error. Specifically, the pixel error is the Euclidean distance between the real position and the predicted position of the feature point in the environment image of the same frame; the structural similarity error is the error between the structures corresponding to the feature points representing the same obstacle in the environment images of adjacent frames. In this way, the prediction ability of the first deep learning model for depth details can be optimized.

[0082] Step 303: Iteratively update the second weight of the original first deep learning model based on the loss gradient of the loss function to reduce the loss value to meet the preset accuracy requirements; and use the second weight corresponding to the final loss value as the final weight of the first deep learning model.

[0083] In some embodiments, the first deep learning model is a network model with a symmetrical structure and jump connections, and the jump connection structure is used to directly transmit the initial depth information to the output layer of the first deep learning model to recover / generate dense final depth information from the sparse initial depth information. The first deep learning model of this implementation also includes a fusion module, which can fuse the historical environment image and the initial depth information corresponding to the historical environment image, maintain the transmission of the initial depth information in the network, so as to reconstruct the dense final depth information in the final output layer.

[0084] In some embodiments, considering the uneven distribution of sparse points in space, an attention mechanism is introduced into the first deep learning model to guide the first deep learning model to pay more attention to areas with initial depth information, thereby improving the accuracy of depth recovery.

[0085] In some embodiments, the first deep learning model adopts dual input channels, that is, the model accepts two inputs, one is the original single-frame historical environment image, and the other is the initial depth information corresponding to the historical environment image.

[0086] For ease of understanding, a feasible structure of the first deep learning model of this embodiment is described below. The first deep learning model includes the following structure and functions: Encoder: Downsampling path.

[0087] Convolution layer: Use convolution kernels of different sizes to extract feature information of historical environment images at different scales to obtain feature maps.

[0088] Max Pooling Layer: Adding a pooling layer for downsampling reduces the size of the feature map and retains the most important features.

[0089] Number of channels: Each downsampling step of the encoder increases the number of channels to extract higher-level features.

[0090] Decoder: Upsampling path.

[0091] Transposed convolution layer: Use transposed convolution for upsampling and gradually restore the size of the feature map.

[0092] Convolutional layer: The upsampled feature map is further processed through the convolutional layer.

[0093] Skip connection: Feature maps of corresponding sizes are passed from the encoder to the decoder and combined through concatenation operations to restore detailed information of historical environment images.

[0094] Halving the number of channels: In each upsampling step of the decoder, the number of channels is gradually halved until it approaches the number of channels of the historical environment images.

[0095] Convolutional layer: Finally, a convolutional layer with a smaller convolution kernel is used to adjust the number of channels and output a dense depth map of the same size as the original historical environment image, which is the final depth information.

[0096] Step 304: Collect environmental images during the movement of the movable device.

[0097] Step 305: Input the environment image into the second deep learning model to obtain feature points for characterizing some obstacles in the environment image.

[0098] Step 306: Obtain initial depth information of the environment image according to the feature points. The initial depth information is used to characterize the distance between some obstacles and the movable device.

[0099] Step 307: Input the environment image and the initial depth information into the first deep learning model to obtain final depth information of the environment image; wherein the final depth information is used to characterize the distance between all objects in the environment image and the movable device.

[0100] Step 308: Perform obstacle detection on the movable device according to the final depth information, and then plan a motion path for the movable device based on the result of the obstacle detection.

[0101] Steps 304 to 308 of this embodiment are similar to steps 101 to 104 of the above-mentioned embodiment, and will not be described again here to avoid repetition.

[0102] Compared with the related art, the embodiment of the present application has at least the following advantages: the environment image and the initial depth information are input into the first deep learning model to obtain the final depth information of the environment image. On the one hand, the first deep learning model can fuse the environment image and the initial depth information so that the final depth information obtained can represent the distance between all objects in the environment image and the movable device, so that the motion path planned based on the final depth information is highly accurate and can achieve accurate obstacle avoidance of the movable device; on the other hand, the step of fusing the environment image and the initial depth information is implemented in the first deep learning model, which reduces the computing resources of the central processor of the movable device when processing obstacle avoidance. In addition, during the movement of the movable device, it is only necessary to collect the environment image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the movable device.

[0103] Please refer to Figure 4, which is a flow chart of a control method for a movable device provided in an embodiment of the present application. This embodiment is a further improvement on the aforementioned embodiment, and the main improvement is that: in this embodiment, the environmental image is segmented into multiple target segmentation areas, and the obstacle avoidance area of ​​the movable device is determined based on the multiple target segmentation areas and the flight channel area of ​​the movable device. In this way, data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, while improving data processing accuracy, thereby achieving accurate obstacle avoidance of the movable device.

[0104] The specific process of this embodiment is as follows Figure 4 As shown, the following steps are included: Step 401: Collecting an environmental image of the movable device during its movement.

[0105] Step 402: Acquire a flight channel area corresponding to the moving direction of the movable device, wherein the flight channel area includes boundary areas in the vertical direction and the horizontal direction.

[0106] Taking a drone as an example, the flight speed and flight attitude data of the drone are obtained, and then the flight channel area under the target camera corresponding to the flight direction of the drone is determined based on the flight speed and flight attitude data.

[0107] Specifically, flight speed is not only the specific value of speed, but also the direction of speed. Flight attitude data is the state of the three axes of the drone in the air relative to a reference line or a reference plane, or a fixed coordinate system. Drones have various flight attitudes, which determine the direction of the aircraft, affecting both the flight altitude and the flight direction. The maneuverability of a drone is related to its flight performance and flight direction. During flight, a drone can capture images with a larger field of view in the direction of flight, but factors such as the speed and flight performance of the drone restrict the range of areas that the drone can reach. Therefore, not all areas in the image will affect the safety performance of the drone.

[0108] In some embodiments, the flight speed includes at least a flight speed value and a speed direction, and the flight attitude data includes at least a lateral acceleration critical upper limit value and a pitch angle; the flight channel area under the target camera corresponding to the flight direction of the drone is determined in the following manner: the flight channel boundary in the horizontal direction is determined according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value and the pixel parameters of the target camera; the flight channel boundary in the vertical direction is determined according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value, the pitch angle and the pixel parameters of the target camera.

[0109] In some embodiments, the flight channel boundary in the horizontal direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value and pixel parameters of the target camera, including: determining a first lateral displacement of the UAV in the horizontal direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining a first forward displacement of the UAV in the horizontal direction of the world coordinate system according to the speed value and the preset time; determining a direction angle of the UAV in the horizontal direction according to the first lateral displacement and the first forward displacement; determining the flight channel boundary in the horizontal direction according to the direction angle in the horizontal direction, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0110] It is worth noting that there is no limitation on how to determine the above-mentioned flight speed value, speed direction, lateral acceleration critical upper limit value and preset time in this embodiment. The main purpose is to establish the angle and pixel relationship based on the pixel parameters of the drone camera and the area captured by the drone camera according to the coordinate system, where the angle can be determined based on the flight speed value and speed direction of the drone. At this time, according to the angle and pixel relationship, the left and right boundary ranges that the drone can actually reach can be determined.

[0111] In some embodiments, the flight channel boundary in the vertical direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value, pitch angle and pixel parameters of the target camera, including: determining the second lateral displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining the second forward displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value and the preset time; determining the direction angle of the UAV in the vertical direction according to the second lateral displacement and the second forward displacement; determining the flight channel boundary in the vertical direction according to the direction angle in the vertical direction, the pitch angle, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0112] It should be noted that the process of determining the flight channel boundary in the vertical direction is similar to the process of determining the flight channel boundary in the horizontal direction. Since the pitch angle parameter is introduced in the vertical direction in the world coordinate system, and the pitch angle is related to the speed, the algorithm for determining the flight channel boundary in the horizontal direction in the above embodiment is based on the relationship between the angle corresponding to the pitch angle and the pixel, that is, the upper and lower boundary ranges that the drone can actually reach are determined.

[0113] Step 403: performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation regions, wherein each target segmentation region includes at least one object in the environment image.

[0114] In some embodiments, the environment image is input into a pre-trained image segmentation network model, and multiple target segmentation areas are determined based on the output results of the image segmentation network model.

[0115] Specifically, the image segmentation network model is used to output the annotation information of the object in each target segmentation area, and the annotation information includes at least the size information of the target object and the position information in the environment image. It should be noted that the objects in the environment image include obstacles and non-obstacles. For example, the objects in the environment image can be tree crowns and the sky, the tree crown is an obstacle, and the sky is a non-obstacle. By outputting the annotation information of the object through the image segmentation network model, it is possible to determine whether the object is an obstacle based on the annotation information, and then use the segmentation area with obstacles as the target segmentation area.

[0116] For ease of understanding, the following Figure 5 How to obtain multiple target segmentation regions in this embodiment is specifically described: Please refer to Figure 5 , is a diagram of the application scenario of image segmentation provided in the embodiment of the present application. Figure 5 As shown in (a), the environment image is captured by the target camera. The CPU transmits the environment image to the image segmentation network model of the NPU. The image segmentation network model segments the target object in the environment image and obtains Figure 5 The environment image is segmented into multiple target segmentation regions as shown in (b).

[0117] In some embodiments, the image segmentation network model is trained according to the following steps: obtaining historical environment images of the drone during historical flights; annotating historical target objects in the historical environment images to obtain training images with real labels, wherein the real labels include at least the size of the historical target objects and the position in the historical environment images; and training the initial image segmentation network model using the training images to obtain an image segmentation network model.

[0118] In some embodiments, since the computing power of the embedded chip selected by the drone platform is relatively weak, a lightweight neural network model needs to be selected to ensure the real-time requirements of image segmentation. Therefore, the image segmentation network model can be MobileNetV2 combined with U-Net. MobileNetV2 can efficiently extract image features, and combined with the U-Net structure, image segmentation can be performed quickly.

[0119] In some embodiments, the size of the image segmentation network model is further reduced and the running speed is improved through model quantization, pruning and other technologies. Taking model quantization as an example, the model parameters of the image segmentation network model are converted into fixed-point representation according to a preset quantization calibration table, wherein the quantization calibration table includes at least the number of quantization bits, quantization method and quantization range. By quantizing the image segmentation network model, the storage requirements of the image segmentation network model can be reduced, while the inference running speed on the NPU embedded side can be improved: model quantization converts floating-point model parameters into fixed-point representation, thereby reducing the space occupied by the image segmentation network model on the storage medium, memory bandwidth and memory overhead; at the same time, the quantized model can take advantage of the higher computational efficiency of integer operations on the NPU compared to floating-point operations, thereby achieving more efficient calculations on embedded hardware.

[0120] In some embodiments, the trained image segmentation network model is deployed on the NPU chip of the drone, and the NPU is used to perform image segmentation processing.

[0121] For ease of understanding, the training process of the image segmentation network model of this embodiment is specifically described below: (1) Collect a large number of target object images during the flight of the UAV, mark the precise location of the target object in the target object image, and construct a training set.

[0122] (2) Perform noise processing such as distortion, occlusion, motion blur, and defect on the above-mentioned images of the labeled target objects to supplement the training set.

[0123] (3) Use the above training set to train the image segmentation network model to obtain the first weight of the image segmentation network model.

[0124] (4) Repeat steps (1) to (3) until the image segmentation network model converges.

[0125] Step 404: Determine an obstacle avoidance area for the movable device according to the multiple target segmentation areas and the flight channel area.

[0126] Specifically, from the description of the aforementioned step 403, it can be seen that the image segmentation network model is used to output the labeling information of the objects in each target segmentation area. The labeling information includes at least the size information of the object and the position information in the environmental image. Therefore, the obstacles in the flight channel area can be determined through the labeling information, and then the area with obstacles can be used as the obstacle avoidance area.

[0127] Step 405: Acquire initial depth information of the obstacle avoidance area, wherein the initial depth information is used to characterize the distance between some objects in the obstacle avoidance area and the movable device.

[0128] Step 406: Input the obstacle avoidance area and the initial depth information into the first deep learning model to obtain final depth information of the obstacle avoidance area; wherein the final depth information is used to characterize the distance between all objects in the obstacle avoidance area and the movable device.

[0129] Step 407: perform obstacle detection on the movable device according to the final depth information, and then plan a motion path for the movable device based on the result of the obstacle detection.

[0130] Step 401 and step 405 to step 407 of this embodiment are similar to step 101 to step 104 of the above-mentioned embodiment, and will not be described again here to avoid repetition.

[0131] Compared with the related art, the embodiment of the present application has at least the following advantages: the environment image and the initial depth information are input into the first deep learning model to obtain the final depth information of the environment image. On the one hand, the first deep learning model can fuse the environment image and the initial depth information so that the final depth information obtained can represent the distance between all objects in the environment image and the movable device, so that the motion path planned based on the final depth information is highly accurate and can achieve accurate obstacle avoidance of the movable device; on the other hand, the step of fusing the environment image and the initial depth information is implemented in the first deep learning model, which reduces the computing resources of the central processor of the movable device when processing obstacle avoidance. In addition, during the movement of the movable device, it is only necessary to collect the environment image through the camera, without the need to use a high-precision sensor, which reduces the obstacle avoidance cost of the movable device.

[0132] Please refer to Figure 6 , is a functional module diagram of a control device 100 of a mobile device provided in an embodiment of the present application. Figure 6 As shown, the control device 100 of the movable device includes: an acquisition module 1, a first processing module 2 and a second processing module 3; wherein the second processing module 3 includes a first deep learning model 31.

[0133] The acquisition module 1 is used to acquire the environmental image during the movement of the movable device; the first processing module 2 is used to obtain the initial depth information of the environmental image, wherein the initial depth information is used to characterize the distance between some objects in the environmental image and the movable device; the first processing module 2 is also used to input the environmental image and the initial depth information into the first deep learning model 31; the first deep learning model 31 is used to fuse the environmental image and the initial depth information, and output the final depth information of the environmental image; wherein the final depth information is used to characterize the distance between all objects in the environmental image and the movable device; the first processing module 2 is also used to perform obstacle detection on the environmental image according to the final depth information, and then plan the motion path of the movable device based on the result of the obstacle detection.

[0134] Please refer to Figure 7 , is a schematic diagram of the hardware structure of the electronic device 1000 provided in the embodiment of the present application. Figure 7 As shown, the electronic device 1000 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the control method of the above movable device in the electronic device 1000.

[0135] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 1000. In other embodiments, the electronic device 1000 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0136] The processor 1001 may include one or more processing units, for example, the processor 1001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0137] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 1001 has just used or circulated. If the processor 1001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.

[0138] In some embodiments, the processor 1001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0139] In some embodiments, processor 1001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0140] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0141] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the control method of the movable device in the above-mentioned embodiment.

[0142] Among them, the electronic device and computer storage medium provided in this embodiment are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0143] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0144] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0148] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A control method for a movable device, characterized in that: include: Collect environmental images during the movement of mobile equipment; Acquire initial depth information of the environment image, wherein the initial depth information is used to characterize the distance between some objects in the environment image and the movable device; Inputting the environment image and the initial depth information into a first deep learning model to obtain final depth information of the environment image; wherein the final depth information is used to characterize the distance between all objects in the environment image and the movable device, and the first deep learning model is used to fuse the environment image and the initial depth information; Obstacle detection is performed on the environment image according to the final depth information, and a motion path planning is performed on the movable device based on the result of the obstacle detection.

2. The control method of a movable device according to claim 1, characterized in that: The obtaining of initial depth information of the environment image includes: Inputting the environmental image into a second deep learning model to obtain feature points for characterizing the partial object, wherein the second deep learning model is trained by historical environmental images collected by the movable device during historical movement; The initial depth information is obtained according to the feature points.

3. The control method of the movable device according to claim 2, characterized in that: The movable device comprises a camera for capturing the image of the environment; The obtaining the initial depth information according to the feature points includes: Matching feature points of adjacent frames of the environment image with the environment image to obtain a posture transformation matrix; Obtaining the coordinates of the feature points in the world coordinate system according to the posture transformation matrix and the intrinsic parameter matrix of the camera; The feature points and the coordinates of the feature points in the world coordinate system constitute the initial depth information.

4. The control method of a movable device according to claim 2, characterized in that: The second deep learning model is trained in the following way: Performing noise processing on the historical environment image to obtain a preset amount of noise data; Performing target detection on the historical environment image to obtain a final historical environment image with a label; Training the original second deep learning model according to the final historical environment image and the noise data; Update the first weight of the original second deep learning model according to the training result of the original second deep learning model until the original second deep learning model converges, and use the converged original second deep learning model as the second deep learning model.

5. The control method of a movable device according to claim 1, characterized in that: The first deep learning model is trained in the following way: Performing data enhancement on the initial training data to obtain target training data; wherein the initial training data includes historical environment images collected by the movable device during historical movement, and historical feature points corresponding to the historical environment images; Input the target training data into the original first deep learning model, and calculate the loss value of the loss function corresponding to the original first deep learning model; The second weight of the original first deep learning model is iteratively updated based on the loss gradient of the loss function to reduce the loss value to meet the preset accuracy requirement; and the second weight corresponding to the final loss value is used as the final weight of the first deep learning model, wherein the first deep learning model includes a jump connection structure and a fusion module, the jump connection structure is used to transmit the initial depth information directly to the output layer of the first deep learning model, and the fusion module is used to fuse the initial depth information with the environmental image to obtain the final depth information.

6. The control method of a movable device according to claim 1, characterized in that: Before obtaining the initial depth information of the environment image, the method further includes: Acquire a flight channel area corresponding to the moving direction of the movable device, wherein the flight channel area includes boundary areas in vertical and horizontal directions; Performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation regions, wherein each of the target segmentation regions includes at least one object in the environment image; Determine an obstacle avoidance area of ​​the movable device according to the plurality of target segmentation areas and the flight channel area; The obtaining of initial depth information of the environment image includes: Acquire initial depth information of the obstacle avoidance area.

7. The control method of the movable device according to claim 6, characterized in that: The performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation areas includes: Inputting the environment image into a pre-trained image segmentation network model, and determining a plurality of target segmentation regions based on an output result of the image segmentation network model; The image segmentation network model is used to output the annotation information of the object in each target segmentation area, and the annotation information at least includes the size information of the object and the position information in the environment image.

8. A control device for a movable device, characterized in that: include: An acquisition module, a first processing module and a second processing module; wherein the second processing module includes a first deep learning model; The acquisition module is used to acquire environmental images during the movement of the movable device; The first processing module is used to obtain initial depth information of the environment image, wherein the initial depth information is used to represent the distance between some objects in the environment image and the movable device; The first processing module is further used to input the environment image and the initial depth information into the first deep learning model; The first deep learning model is used to fuse the environment image and the initial depth information, and output final depth information of the environment image; wherein the final depth information is used to characterize the distance between all objects in the environment image and the movable device; The first processing module is further configured to perform obstacle detection on the environment image according to the final depth information, and then perform motion path planning on the movable device based on the result of the obstacle detection.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the control method of the movable device according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The invention comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the control method of the movable device according to any one of claims 1 to 7.

Images