Navigation method and device of unmanned device, unmanned device and storage medium

Abstract

The application discloses a navigation method and device of an unmanned device, the unmanned device and a storage medium, and relates to the technical field of unmanned devices. The method comprises the following steps: acquiring a first image collected by a binocular vision sensor, the binocular vision sensor being installed on both sides of the unmanned device, and the content of the first image containing a first region in front of the unmanned device; constructing an edge point cloud of the first region based on the first image, determining a first navigation line of the unmanned device according to the edge point cloud, and the edge point cloud being used for representing edge information of crops on both sides in the first region; and controlling the unmanned device to move according to the first navigation line. Through the technical means, the problem of low visual navigation precision and limited use scene in the prior art is solved, the applicability of the unmanned device is improved, and the popularization and use of the unmanned device are facilitated.

Description

Technical Field

[0001] This application relates to the field of unmanned equipment technology, and in particular to a navigation method, device, unmanned equipment and storage medium for unmanned equipment. Background Technology

[0002] Automatic navigation is a key supporting technology for expanding the application fields of unmanned equipment. Navigation technology, based on the classification of environmental perception sensors, can be broadly divided into visual navigation, satellite system navigation, lidar navigation, and multi-sensor fusion navigation. Because satellite system navigation is easily affected by obstructions and has low positioning accuracy, lidar navigation is prone to incomplete feature recognition and poor positioning accuracy due to obstructions during scanning, and multi-sensor fusion navigation is costly, unmanned equipment generally adopts visual navigation technology.

[0003] In existing technologies, an image is captured in front of an unmanned device using a monocular vision sensor. Movable and immovable regions are detected within this image, and a navigation line is determined within the movable region to guide the device's movement. However, accurate identification of the movable and immovable regions is only possible when there is a significant difference in texture between them. Therefore, current visual navigation technologies can only guarantee high navigation accuracy in specific scenarios, hindering the widespread adoption of unmanned devices. Summary of the Invention

[0004] This application provides a navigation method, device, unmanned device, and storage medium for unmanned equipment, which solves the problems of low accuracy and limited application scenarios of visual navigation in the prior art, improves the applicability of unmanned equipment, and is conducive to the promotion and use of unmanned equipment.

[0005] Firstly, this application provides a navigation method for unmanned equipment, including:

[0006] A first image is acquired by a binocular vision sensor, which is installed on both sides of the unmanned device. The content of the first image includes a first area in front of the unmanned device.

[0007] An edge point cloud of the first region is constructed based on the first image, and a first navigation line of the unmanned device is determined based on the edge point cloud. The edge point cloud is used to characterize the edge information of crops on both sides of the first region.

[0008] The unmanned equipment is controlled to move according to the first navigation line.

[0009] Secondly, this application provides a navigation device for unmanned equipment, comprising:

[0010] The first image acquisition module is configured to acquire a first image collected by a binocular vision sensor, the binocular vision sensor being installed on both sides of the unmanned device, and the content of the first image including a first area in front of the unmanned device.

[0011] The first navigation line generation module is configured to construct an edge point cloud of the first region based on the first image, and determine the first navigation line of the unmanned device according to the edge point cloud. The edge point cloud is used to characterize the edge information of crops on both sides of the first region.

[0012] The motion control module is configured to control the unmanned device to move according to the first navigation line.

[0013] Thirdly, this application provides an unmanned device, comprising:

[0014] One or more processors; a memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the navigation method for unmanned equipment as described in the first aspect.

[0015] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the navigation method for an unmanned device as described in the first aspect.

[0016] In this application, a first image is generated by capturing images of a first area in front of an unmanned device using binocular vision sensors mounted on both sides. An edge point cloud of the first area is then constructed based on the first image, and a first navigation line is determined based on the edge point cloud. The unmanned device is then controlled to move according to the first navigation line. Through this technique, the edge point cloud of the first area constructed from the first image acquired by the binocular vision sensors can represent the edge information of crops on both sides of the first area. Therefore, a first navigation line that effectively avoids crops can be planned based on the edge point cloud of the first area. The unmanned device is then navigated according to the first navigation line to avoid touching crops during movement, thus protecting both the unmanned device and the crops. Furthermore, the accuracy of the edge point cloud constructed based on the first image is not affected by the texture of the area, and the accuracy of the first navigation line generated based on the edge point cloud is high and not limited by the scene. This solves the problems of low accuracy and limited application scenarios in existing visual navigation technologies, improves the applicability of unmanned devices, and facilitates their widespread use. Attached Figure Description

[0017] Figure 1 This is a flowchart of a navigation method for an unmanned device provided in an embodiment of this application;

[0018] Figure 2This is a first schematic diagram of the first area and the unmanned equipment provided in the embodiments of this application;

[0019] Figure 3 This is a second schematic diagram of the first area and the unmanned equipment provided in the embodiments of this application;

[0020] Figure 4 This is a flowchart of planning navigation lines based on edge point clouds provided in an embodiment of this application;

[0021] Figure 5 This is a first schematic diagram of the edge line of the first region provided in the embodiments of this application;

[0022] Figure 6 This is a second schematic diagram of the edge line of the first region provided in the embodiments of this application;

[0023] Figure 7 This is a flowchart of another navigation method for unmanned equipment provided in an embodiment of this application;

[0024] Figure 8 This is a flowchart of determining the second navigation line provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of the crop area in the bird's-eye view provided in the embodiments of this application;

[0026] Figure 10 This is a schematic diagram of the structure of a navigation device for an unmanned device provided in an embodiment of this application;

[0027] Figure 11 This is a schematic diagram of the structure of an unmanned device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The navigation method for unmanned equipment provided in this embodiment can be executed by the unmanned equipment itself, which can be implemented through software and / or hardware. The unmanned equipment can consist of two or more physical entities, or it can consist of a single physical entity. Here, "unmanned equipment" refers to a flying device or ground platform that operates according to remote control commands or preset commands, such as drones and unmanned vehicles.

[0031] The unmanned device is equipped with at least one type of operating system. Based on this operating system, the unmanned device can install at least one application. This application can be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the unmanned device has at least one application capable of executing its navigation method.

[0032] For ease of understanding, this embodiment uses unmanned equipment as the main body for executing the navigation method of unmanned equipment, and takes unmanned vehicle for operating crops as an example for description.

[0033] In one embodiment, automated navigation of unmanned equipment is a key supporting technology for the development of smart agriculture and other industries such as logistics and surveying. Agriculture includes crop farming, forestry, animal husbandry, and fisheries. Navigation technologies, based on environmental perception sensors, can be broadly categorized into visual navigation, satellite system navigation, lidar navigation, and multi-sensor fusion navigation. This embodiment describes the application of unmanned equipment in smart agriculture as an example. In smart agriculture, satellite system navigation is easily interfered with by obstructions and is typically used in open operating scenarios, unsuitable for special operating environments such as forests or under canopies. LiDAR navigation is also generally unsuitable due to incomplete feature recognition and poor positioning accuracy caused by foliage obstruction during scanning. Multi-sensor fusion navigation uses satellite system navigation for coarse heading adjustment and visual navigation for fine heading adjustment. Multi-sensor fusion navigation is primarily loosely coupled, with the satellite system playing a certain auxiliary role, but it does not significantly improve navigation accuracy. Therefore, in smart agriculture, visual navigation technology is generally used to achieve automated navigation of unmanned equipment.

[0034] Existing visual navigation technologies use a monocular vision sensor to capture images of the area in front of an unmanned aerial vehicle (UAV), detect crop regions within those images, and determine a navigation line based on these crop regions to enable the UAV to move and perform tasks. Specifically, a pre-trained deep learning model identifies the crop regions in the work area image. The accuracy of the crop region identification by the deep learning model is only guaranteed when the texture of the crop region differs significantly from other areas. Therefore, existing visual navigation technologies can only guarantee high navigation accuracy in specific scenarios, hindering the widespread use of UAVs.

[0035] To address the issues of low accuracy and limited application scenarios in existing visual navigation technologies, this embodiment provides a navigation method for unmanned equipment.

[0036] Figure 1 A flowchart of a navigation method for unmanned equipment provided in an embodiment of this application is given. (Reference) Figure 1 The navigation method of this unmanned device specifically includes:

[0037] S110. Acquire the first image captured by the binocular vision sensor. The binocular vision sensor is installed on both sides of the unmanned device. The content of the first image includes a first area in front of the unmanned device.

[0038] For example, the binocular vision sensor includes a first camera and a second camera. The first camera is mounted on the left side of the unmanned device, and the second camera is mounted on the right side. The first and second cameras simultaneously capture images of the work area in front of the unmanned device to obtain two first images from different shooting angles. The first and second cameras can capture the first images at intervals or at certain distances, or they can record video and extract the first images from the video. The specific method of acquiring the first images can be set according to actual needs. During movement, the unmanned device acquires the first images in real time through the first and second cameras and plans a real-time navigation line based on the first images to move according to the navigation line.

[0039] In this embodiment, the first area can be understood as the working area in front of the unmanned equipment falling within the shooting area of ​​the binocular vision sensor. Crops, which can be food crops or cash crops, are located on one or both sides of the working area, and the center of the working area is the flat ground where the unmanned equipment can move. The first and second cameras capture images of the working area in front of the unmanned equipment, obtaining two first images. The content of both first images can include the crops on one or both sides of the working area and the flat ground in the center of the working area. This embodiment aims to plan a navigation line for the unmanned equipment within the working area based on the feature information in the two first images, so that the unmanned equipment can avoid the crops on one or both sides of the working area when moving based on the navigation line, protecting both the crops and the unmanned equipment.

[0040] S120. Construct an edge point cloud of a first region based on the first image, and determine the first navigation line of the unmanned equipment based on the edge point cloud. The edge point cloud is used to represent the edge information of crops on both sides of the first region.

[0041] The first navigation line is generated using a binocular vision navigation method. For example, Figure 2 and Figure 3 A schematic diagram of the first area and unmanned equipment provided in an embodiment of this application is shown. Figure 2 and Figure 3 As shown, crops 15 are located on both sides of the first area 17, and the middle of the first area 17 is a flat area where the unmanned equipment 11 can move. A first camera 12 is mounted on the left side of the unmanned equipment 11, and a second camera 13 is mounted on the right side. It should be noted that, limited by the shooting angles of the first camera 12 and the second camera 13, the shooting area (…) Figure 2 The area shown by the dashed line only contains part of the work area. The first image generated by the first camera 12 and the second camera 13 taking pictures of the work area in front of the unmanned equipment shows crops 15 on one side of the flat ground where the unmanned equipment 11 is traveling.

[0042] Furthermore, pixel matching is performed on the two first images simultaneously captured by the first and second cameras. A disparity matrix is ​​generated based on the matched feature points, the intrinsic parameters of the first and second cameras, and the extrinsic parameters between the two cameras. This disparity matrix is ​​then solved to obtain the depth of each pixel in the first image captured by the first camera. Based on the pixel depth, the intrinsic parameters of the first camera, and the extrinsic parameters of the first camera relative to the world coordinate system, the 3D coordinates corresponding to each pixel are determined. The 3D coordinates corresponding to each pixel in the first image captured by the first camera are combined to obtain the edge point cloud of the first region. (Reference) Figure 2 and Figure 3When there are no obstacles on the driving flat ground, the edge point cloud 16 of the first region 17 is composed of the edge point cloud of the crops 15 on both sides of the first region, which are closer to the driving flat ground of the unmanned equipment 11. Therefore, the edge point cloud 16 of the first region 17 can represent the edge information of the crops 15 on both sides of the driving flat ground. Based on the edge point cloud 16 of the first region 17, a navigation line that can effectively avoid the crops 15 on both sides can be planned, so as to control the unmanned equipment to move on the driving flat ground according to the navigation line to avoid touching the crops 15 on both sides.

[0043] In one embodiment, the edge point cloud of the first region can be clustered using a density clustering algorithm to divide the edge point cloud of the first region into the edge point clouds of the crops on the left and right sides. A first surface is obtained by fitting the edge point cloud of the crops on the left, and a second surface is obtained by fitting the edge point cloud of the crops on the right. A center line between the first and second surfaces is determined in the space formed between the first and second surfaces, and this center line is designated as the first navigation line. In this embodiment, besides using a clustering algorithm to divide the edge point cloud of the first region into the edge point clouds of the crops on the left and right sides, the edge point cloud of the first region can also be divided into the edge point clouds of the crops on the left and right sides based on the central line of sight of the binocular vision sensor. Here, the central line of sight of the binocular vision sensor is the center line of the first camera and the second camera.

[0044] In another embodiment, since fitting multiple point clouds to a surface is complex and the planning efficiency of navigation lines is low, the three-dimensional edge point cloud can be converted into two-dimensional edge points to accurately and quickly plan navigation lines based on the two-dimensional edge points. For example, Figure 4 This is a flowchart illustrating navigation line planning based on edge point clouds, provided in an embodiment of this application. For example... Figure 4 As shown, the steps for planning navigation lines based on edge point clouds specifically include S1201-S1203:

[0045] S1201. Project the edge point cloud onto the first plane to obtain two-dimensional edge points. The first plane is the plane where the unmanned equipment is located.

[0046] For example, assuming the first plane is the XY plane of the world coordinate system, the corresponding two-dimensional edge point can be obtained by deleting the Z-axis coordinate in the three-dimensional coordinate of the edge point cloud. For example, if the three-dimensional coordinate of the edge point cloud is (x1, y1, z1), then the two-dimensional coordinate of the corresponding two-dimensional edge point is (x1, y1).

[0047] S1202. Obtain the edge line of the first region by fitting the two-dimensional edge points.

[0048] The edge line of the first area can be understood as the edge line of the crops in the first area closer to the flat ground. Figure 5This is a first schematic diagram of the edge line of the first region provided in an embodiment of this application. For example... Figure 5 As shown, the two-dimensional edge points 18 can be divided into two-dimensional edge points 18 of the left and right crops by density clustering algorithm or the central line of sight of binocular vision sensor. The two-dimensional edge points 18 of the left crop are fitted into the first edge line 19, and the two-dimensional edge points 18 of the right crop are fitted into the second edge line 20.

[0049] Because the number of 2D edge points obtained by projecting the edge point cloud is large and their distribution is uneven, the edge lines generated by directly fitting the 2D edge points on the left and right sides have low accuracy. To improve the accuracy of the edge lines, 2D edge points that are evenly distributed and can characterize the edge information of crops can be selected from the 2D edge points, and high-precision edge lines can be fitted based on the selected 2D edge points. The specific implementation process is as follows: According to the central line of sight of the binocular vision sensor, the first region is divided into a second region and a third region; the second region and the third region are divided into multiple grids according to a preset interval, and the 2D edge points falling into the grids are determined; the first edge line is fitted based on the 2D edge point closest to the central line of sight in each grid of the second region, and the second edge line is fitted based on the 2D edge point closest to the central line of sight in each grid of the third region. Figure 6 This is a second schematic diagram of the edge line of the first region provided in an embodiment of this application. For example... Figure 6As shown, the first region 17 can be divided into a second region 21 and a third region 22 through the central line of sight of the binocular vision sensor. The first region 17 is then divided into multiple rows at preset intervals, so that the second region 21 and the third region 22 are each divided into multiple grids 23. Assuming the preset interval is 20cm, the width of each grid is 20cm, and the length of the grid is equal to the length of the second or third region. The position range of each grid is determined based on the position range of the first region and the preset interval. The position range of each grid is compared with the two-dimensional coordinates of each two-dimensional edge point to determine the grid into which the two-dimensional edge point falls. Based on the position coordinates of the central line of sight, the two-dimensional edge point closest to the central line of sight is selected from the two-dimensional edge points of each grid in the second region. If there is no two-dimensional edge point in the grid, it is not selected. It can be understood that if the second region can be divided into N grids, then at most N two-dimensional edge points can be selected. The first edge line 19 is obtained by fitting the two-dimensional edge points selected from the second region. Based on the position coordinates of the central line of sight, the two-dimensional edge point closest to the central line of sight is selected from the two-dimensional edge points of each grid in the third region. A second edge line 20 is then fitted based on this selected two-dimensional edge point. In this embodiment, the second and third regions are uniformly divided into multiple grids. Obtaining a two-dimensional edge point from a grid ensures that the two-dimensional edge points used for fitting the edge line are uniformly distributed. Furthermore, the edge point closest to the central line of sight in the grid best represents the edge information of the crop. Therefore, the edge line generation method provided in this embodiment can greatly improve the accuracy of the edge line, thereby improving the accuracy of the navigation line.

[0050] S1203. Determine the first navigation line of the unmanned equipment based on the edge line.

[0051] For example, the center line between the first edge line and the second edge line can be determined as the first navigation line of the unmanned equipment. When the first edge line and the second edge line are generated through fitting, corresponding straight line expressions can be obtained. Based on the straight line expressions of the first edge line and the second edge line, a center line equidistant from the first edge line and the second edge line can be determined, and this center line can be determined as the first navigation line of the unmanned equipment.

[0052] In another embodiment, since the height of the unmanned vehicle is limited, when some branches and leaves of the crop are taller than the unmanned vehicle, these branches and leaves will not collide with the unmanned vehicle during its movement. Therefore, based on the height of the unmanned vehicle and the height information of the edge point cloud, edge point clouds with height information higher than the unmanned vehicle can be filtered out, and the first navigation line of the unmanned vehicle can be determined using the remaining edge point cloud. Specifically, determining the first navigation line of the unmanned vehicle using the remaining edge point cloud can be implemented using steps S1201-S1203.

[0053] It should be noted that when there are obstacles on flat ground, the edge point cloud of the first region includes the edge point cloud of crops and the edge point cloud of obstacles. The edge point cloud of crops can be selected from the edge point cloud of the first region, and the first navigation line can be planned based on the edge point cloud of crops. When selecting the edge point cloud of crops, a density clustering algorithm can be used to cluster the edge point cloud of the first region to divide the edge point cloud of the first region into the edge point cloud of crops on the left, crops on the right, and the edge point cloud of the obstacle in the middle. The edge point cloud of the obstacle in the middle is removed from the edge point cloud of the first region to obtain the edge point cloud of crops. Alternatively, the distance between the edge point cloud and the center line of sight can be determined, and the edge point cloud with a distance less than a preset distance threshold is identified as the edge point cloud of the obstacle. The edge point cloud of the obstacle is removed from the edge point cloud of the first region to obtain the edge point cloud of crops. Here, the preset distance threshold can be understood as the minimum distance between the crops in the first region and the center line of sight.

[0054] S130: Control the unmanned equipment to move according to the first navigation line.

[0055] For example, based on the position coordinates of each trajectory point in the first navigation line, the unmanned equipment is controlled to move along the first navigation line on a flat surface between crops on both sides of a first area. In this embodiment, the unmanned equipment can be positioned using RTK (Real-Time Kinematic) carrier phase differential positioning technology to collect its RTK position information. Based on the RTK position information and the first navigation line, the unmanned equipment is controlled to move, so that it avoids the crops on both sides during movement and performs plant protection operations such as spraying and fertilizing the crops.

[0056] In one embodiment, the unmanned equipment can pre-plan a global operating path based on a distribution map of crops and driving flat ground within the work area. The unmanned equipment can then move within the work area based on RTK location information and the global operating path. It should be noted that because the distribution map has low accuracy, the global operating path also has low accuracy. If the unmanned equipment is controlled to move on driving flat ground solely based on the global operating path, it is highly likely to collide with the branches and leaves of crops on both sides. Therefore, the navigation method provided in this embodiment can plan a high-precision navigation line so that the unmanned equipment avoids the branches and leaves of crops on both sides when moving based on the navigation line.

[0057] If the edge point cloud of obstacles is removed from the edge point cloud of the first region when planning the first navigation line, it can be confirmed that there are obstacles on the flat driving ground in the first region. To ensure the safety of the unmanned equipment, it can be determined whether there are obstacles in front of the unmanned equipment based on the edge point cloud and the first navigation line. For example, the width of the unmanned equipment and the first navigation line determine the movement area of ​​the unmanned equipment when moving based on the first navigation line. It is determined whether the obstacle is within the movement area based on the edge point cloud of the obstacle. If it is, it is determined that there is an obstacle in front of the unmanned equipment; if it is not, it is determined that there is no obstacle in front of the unmanned equipment. Further, if there is an obstacle in front of the unmanned equipment, the unmanned equipment is controlled to move around the obstacle or stop moving. For example, the distance between the obstacle and the crops can be determined based on the edge point cloud of the obstacle and the edge point cloud of the crops on both sides. If the distance is greater than the width of the unmanned equipment, an obstacle avoidance path passing through the middle of the two can be planned based on the edge point cloud of the obstacle and the edge point cloud of the crops. The unmanned equipment is controlled to move around the obstacle based on the obstacle avoidance path. If the interval is less than or equal to the width of the unmanned equipment, the movement stops, and a prompt message can be sent to the staff so that the staff can move the obstacle away.

[0058] Based on the above embodiments, this embodiment also provides another navigation method for unmanned equipment. This navigation method aims to integrate the binocular vision navigation method and the monocular vision navigation method provided in the above embodiments to further improve navigation accuracy. Figure 7 This is a flowchart of another navigation method for unmanned equipment provided in an embodiment of this application. For example... Figure 7 As shown, the navigation method of this unmanned device specifically includes:

[0059] S210, Acquire the first image captured by the binocular vision sensor.

[0060] S220. Construct an edge point cloud of the first region based on the first image, and determine the first navigation line of the unmanned equipment based on the edge point cloud.

[0061] Steps S210-S220 can be referred to steps S110-S120.

[0062] S230. Acquire a second image captured by a monocular vision sensor. The monocular vision sensor is installed in the middle of the unmanned device. The content of the second image includes a fourth region in front of the unmanned device.

[0063] For example, the monocular vision sensor is a third camera, reference Figure 2 A third camera 14 is mounted in the middle of the unmanned device. The third camera captures a first area in front of the unmanned device to obtain a second image. The third camera can capture the second image at certain intervals or from a certain distance, or it can record video and extract the second image from the video.

[0064] In this embodiment, the fourth region can be understood as the working area in front of the unmanned equipment that falls within the shooting area of ​​the monocular vision sensor. The monocular vision sensor and the binocular vision sensor will simultaneously capture the second image and the first image to ensure that the content of the first image and the second image contains approximately the same working area, thereby ensuring the accuracy of the navigation line generated later by fusion.

[0065] S240. Based on the second image, determine the second navigation line of the unmanned equipment.

[0066] The second navigation line is generated using a monocular vision navigation method. For example, the second image is segmented into crop and non-crop regions using an adaptive threshold segmentation algorithm, and the second navigation line is determined based on the centerline of the non-crop region.

[0067] Since the third camera captures a second image of the work area at an angle, the crop and non-crop areas in the second image are distorted due to the shooting angle, affecting the accuracy of the generated second navigation line. To address this, the second image can be converted into a bird's-eye view of the first area to determine the second navigation line. For example, Figure 8 This is a flowchart illustrating the determination of the second navigation line provided in an embodiment of this application. For example... Figure 8 As shown, the steps for determining the second navigation line specifically include S2401-S2402:

[0068] S2401. Convert the second image to obtain a bird's-eye view of the first region.

[0069] For example, by using a pre-calibrated transformation matrix or viewpoint transformation algorithm, the second image is transformed from an oblique shooting view to a top-down shooting view to obtain a bird's-eye view of the fourth region. Here, the transformation matrix can be understood as the relative transformation matrix between the world coordinate system when the third camera takes the second image and the world coordinate system when the second image is taken from above.

[0070] S2402. Determine the fifth region in the bird's-eye view, and determine the second navigation line of the unmanned equipment based on the fifth region.

[0071] In one embodiment, a pre-defined semantic segmentation model is used to segment the bird's-eye view to extract a fifth region. This fifth region is the crop area in the bird's-eye view. The dataset used for training the semantic segmentation model includes multiple sample images and the pixel coordinates of both crop and non-crop areas within the sample images. In another embodiment, an adaptive thresholding segmentation algorithm can be used to segment the bird's-eye view to extract the crop area. Figure 9 This is a schematic diagram of a crop area in a bird's-eye view provided in an embodiment of this application. For example... Figure 9As shown, after extracting the crop region 25 from the bird's-eye view 24, the extracted crop region 25 is divided into a left crop region 25 and a right crop region 25 according to the center line of the bird's-eye view. The pixel coordinates of the center point 26 of the crop region are determined based on the pixel coordinates of the crop region 25. A first center line 27 is fitted based on the pixel coordinates of the center point 26 of the left crop region 25, and a second center line 28 is fitted based on the pixel coordinates of the center point 26 of the right crop region 25. The center line between the first center line 26 and the second center line 27 is determined as the pixel coordinates of the second navigation line. The position information of the second navigation line in the world coordinate system is determined based on the pixel coordinates of the second navigation line and the scale factor of the monocular vision sensor. The scale factor is used to represent the relationship between the pixel distance in the bird's-eye view and the actual distance.

[0072] It should be noted that the binocular vision sensor used in this embodiment has the same scale factor as the monocular vision sensor. That is, the first navigation line generated by the binocular vision navigation method and the second navigation line generated by the monocular vision navigation method are at the same scale and can be fused. If the binocular vision sensor and the monocular vision sensor have different scale factors, their scales need to be unified before fusion to ensure fusion accuracy.

[0073] S250. Merge the first and second navigation lines to obtain the third navigation line.

[0074] The third navigation line is generated by fusing binocular and monocular visual navigation methods. For example, the center line between the first and second navigation lines can be used as the third navigation line. However, the first navigation line has higher accuracy than the second. If the center line between the two is used as the third navigation line, the accuracy of the third navigation line may be lower than that of the first navigation line, failing to improve the accuracy of the navigation line. Therefore, the third navigation line can be determined based on the first navigation line and its corresponding first confidence level, and the second navigation line and its corresponding second confidence level. The first confidence level characterizes the accuracy of the first navigation line, and the second confidence level characterizes the accuracy of the second navigation line. Assume the first navigation line is l... s (x s ,y s The second navigation line is l. a (x a ,y a Then the third navigation line l r (x r ,y r The expression for ) can be: l r (x r ,y r ) = c s *l s (x s ,ys )+c a *l a (x a ,y a ), where c s For the first confidence level, c a This represents the second confidence level.

[0075] In this embodiment, a first confidence level and a second confidence level can be pre-set based on the texture features of crop areas and non-crop areas within the work area. It is understood that when the texture features of crop areas and non-crop areas differ significantly, it indicates that the accuracy of the second navigation line generated by the monocular vision navigation method is high; in this case, both the first and second confidence levels can be set to 0.5. When the differences between crop areas and non-crop areas are small, it indicates that the accuracy of the second navigation line generated by the monocular vision navigation method is low; in this case, the first and second confidence levels can be set to 0.7 and 0.3 or 0.8 and 0.2, respectively.

[0076] In addition, during the planning of the first navigation line using a binocular vision navigation method, a first confidence level can be determined based on the data used to plan the first navigation line. The specific implementation process is as follows: Based on the central line of sight of the binocular vision sensor, two-dimensional edge points are divided into grids in the second and third regions. The two-dimensional edge points are generated by projecting the edge point cloud onto the first plane; this step can be referred to steps S1201-S1203. The first confidence level is determined based on the total number of grids in the second and third regions, and the total number of two-dimensional edge points in each grid that are closest to the central line of sight. For example, if a grid contains a two-dimensional edge point, one two-dimensional edge point is taken from that grid to fit the edge line. Therefore, the total number of two-dimensional edge points in each grid that are closest to the central line of sight is obtained by adding the number of grids containing two-dimensional edge points in the second region to the number of grids containing two-dimensional edge points in the third region. This total number is then divided by twice the total number of grids to obtain the first confidence level. The expression for the first confidence level is: c s =(n l +n r ) / (2*n s ), where n l and n r n represents the number of graticules containing two-dimensional edge points in the second region and the number of graticules containing two-dimensional edge points in the third region, respectively. s This represents the total number of grid cells in the second and third regions. It's understandable that when n... l and n rA higher value indicates a greater number of two-dimensional edge points used to fit and generate the first and second edge lines. Consequently, the accuracy of the first navigation line planned based on the first and second edge lines is higher, and the first confidence level is also higher.

[0077] Similarly, during the planning of the second navigation line using a monocular vision navigation method, a second confidence level can be determined based on the relevant data used for planning the second navigation line. For example, the second confidence level is determined based on the confidence level output when extracting the fourth region using a preset semantic segmentation model. The second confidence level is obtained by multiplying the confidence level by a coefficient, which can be set according to the actual situation. When extracting crop regions from the bird's-eye view using the semantic segmentation model, the confidence level of each crop region is output. This confidence level characterizes the accuracy of the semantic segmentation model's prediction. Accordingly, the higher the confidence level, the higher the accuracy of the crop region, and the higher the accuracy of the second navigation line planned based on the crop region, resulting in a higher second confidence level. When multiple crop region confidence levels are output, the average of each confidence level is taken, and the second confidence level is determined based on this average.

[0078] S260: Control the unmanned equipment to move according to the third navigation line.

[0079] For example, the unmanned equipment is located using RTK (Real Time Kinematic) carrier phase differential positioning technology to collect the RTK position information of the unmanned equipment. Based on the RTK position information of the unmanned equipment and the third navigation line, the unmanned equipment is controlled to move so that it can avoid the crops on both sides during the movement and carry out plant protection operations such as spraying and fertilizing the crops on both sides.

[0080] In summary, the navigation method for unmanned equipment provided in this application involves using binocular vision sensors mounted on both sides of the unmanned equipment to capture images of a first area in front of the unmanned equipment, generating a first image. An edge point cloud of the first area is then constructed based on the first image. A first navigation line for the unmanned equipment is determined based on the edge point cloud, and the unmanned equipment is controlled to move according to the first navigation line. Through these technical means, the edge point cloud of the first area constructed from the first image acquired by the binocular vision sensors can represent the edge information of crops on both sides of the first area. Therefore, a first navigation line that effectively avoids crops on both sides can be planned based on the edge point cloud of the first area. The unmanned equipment is then navigated according to the first navigation line to avoid touching crops during movement, thus protecting the safety of both the unmanned equipment and the crops. Furthermore, the accuracy of the edge point cloud constructed based on the first image is not affected by the texture of the area, and the accuracy of the first navigation line generated based on the edge point cloud is high and not limited by the scene. This solves the problems of low accuracy and limited application scenarios in existing visual navigation technologies, improves the applicability of unmanned equipment, and facilitates its widespread use.

[0081] Based on the above embodiments, Figure 10 This is a schematic diagram of the structure of a navigation device for an unmanned device provided in an embodiment of this application. (Reference) Figure 8 The navigation device for unmanned equipment provided in this embodiment specifically includes: a first image acquisition module 31, a first navigation line generation module 32, and a movement control module 33.

[0082] The first image acquisition module is configured to acquire a first image collected by a binocular vision sensor. The binocular vision sensor is installed on both sides of the unmanned device, and the content of the first image includes a first area in front of the unmanned device.

[0083] The first navigation line generation module is configured to construct an edge point cloud of a first region based on a first image, and determine the first navigation line of the unmanned device based on the edge point cloud. The edge point cloud is used to represent the edge information of crops on both sides of the first region.

[0084] The motion control module is configured to control the movement of the unmanned equipment according to the first navigation line.

[0085] Based on the above embodiments, the first navigation line generation module 22 includes: an edge point generation submodule, configured to project an edge point cloud onto a first plane to obtain two-dimensional edge points, wherein the first plane is the plane where the unmanned device is located; an edge line generation submodule, configured to fit the two-dimensional edge points to obtain the edge line of the first region; and a first navigation line generation submodule, configured to determine the first navigation line of the unmanned device based on the edge line.

[0086] Based on the above embodiments, the edge line generation submodule includes: a region division unit configured to divide a first region into a second region and a third region according to the central line of sight of the binocular vision sensor; a grid division unit configured to divide the second region and the third region into multiple grids based on a preset interval, and determine the two-dimensional edge points falling into the grids; and an edge line generation unit configured to fit a first edge line based on the two-dimensional edge point closest to the central line of sight in each grid of the second region, and to fit a second edge line based on the two-dimensional edge point closest to the central line of sight in each grid of the third region.

[0087] Based on the above embodiments, the first navigation line generation submodule includes: a first navigation line generation unit, configured to determine the center line between the first edge line and the second edge line as the first navigation line of the unmanned equipment.

[0088] Based on the above embodiments, the navigation device further includes: a second image acquisition module configured to acquire a second image collected by a monocular vision sensor, the monocular vision sensor being installed in the middle of the unmanned device, the content of the second image including a fourth region in front of the unmanned device; a second navigation line generation module configured to determine a second navigation line for the unmanned device based on the second image; and a third navigation line generation module configured to fuse the first navigation line and the second navigation line to obtain a third navigation line, the third navigation line being used to guide the unmanned device to move.

[0089] Based on the above embodiments, the second navigation line generation module includes: an image conversion submodule configured to convert a second image to obtain a bird's-eye view of the first region; and a second navigation line generation submodule configured to determine a fifth region in the bird's-eye view and determine the second navigation line of the unmanned equipment based on the fifth region.

[0090] Based on the above embodiments, the second navigation line generation submodule includes: a fifth region extraction unit, configured to segment the bird's-eye view using a preset semantic segmentation model to extract the fifth region in the bird's-eye view.

[0091] Based on the above embodiments, the third navigation line generation module includes: a third navigation line generation submodule, configured to determine a third navigation line based on a first navigation line and a corresponding first confidence level, and a second navigation line and a corresponding second confidence level.

[0092] Based on the above embodiments, the edge line generation submodule includes: an edge point division unit, configured to divide two-dimensional edge points into grids in the second and third regions according to the central line of sight of the binocular vision sensor, and the two-dimensional edge points are generated by projecting the edge point cloud onto the first plane; correspondingly, the third navigation line generation submodule includes: a first confidence level determination unit, configured to determine a first confidence level based on the total number of grids in the second and third regions and the total number of two-dimensional edge points in each grid that are closest to the central line of sight.

[0093] Based on the above embodiments, the third navigation line generation submodule includes: a second confidence determination unit, configured to determine the second confidence based on the confidence output when extracting the fourth region using a preset semantic segmentation model.

[0094] Based on the above embodiments, the motion control module 23 includes: an obstacle detection submodule, configured to determine whether there is an obstacle in front of the unmanned device based on the edge point cloud and the first navigation line; and a first motion control submodule, configured to control the unmanned device to move around the obstacle or stop moving when there is an obstacle in front of the unmanned device.

[0095] Based on the above embodiments, the motion control module includes: a second motion control submodule, configured to control the unmanned device to move according to the RTK location information of the unmanned device and the first navigation line.

[0096] The navigation device for unmanned equipment provided in this application embodiment, as described above, uses binocular vision sensors installed on both sides of the unmanned equipment to capture images of a first area in front of the unmanned equipment to generate a first image. Based on the first image, an edge point cloud of the first area is constructed. A first navigation line for the unmanned equipment is determined based on the edge point cloud, and the unmanned equipment is controlled to move according to the first navigation line. Through the above technical means, the edge point cloud of the first area constructed from the first image acquired by the binocular vision sensors can represent the edge information of crops on both sides of the first area. Therefore, a first navigation line that can effectively avoid crops on both sides can be planned based on the edge point cloud of the first area. The unmanned equipment is navigated according to the first navigation line to avoid touching crops during movement, thus protecting the safety of both the unmanned equipment and the crops. Moreover, the accuracy of the edge point cloud constructed based on the first image is not affected by the regional texture, and the accuracy of the first navigation line generated based on the edge point cloud is high and not limited by the scene. This solves the problems of low accuracy and limited application scenarios in existing visual navigation technologies, improves the applicability of unmanned equipment, and is conducive to the widespread use of unmanned equipment.

[0097] The navigation device for unmanned equipment provided in this application embodiment can be used to execute the navigation method for unmanned equipment provided in the above embodiment, and has corresponding functions and beneficial effects.

[0098] Figure 11 This is a schematic diagram of the structure of an unmanned device provided in an embodiment of this application, with reference to... Figure 11 The unmanned device includes a processor 41, a memory 42, a communication device 43, an input device 44, and an output device 45. The number of processors 41 and the number of memories 42 in the unmanned device can be one or more. The processor 41, memory 42, communication device 43, input device 44, and output device 45 of the unmanned device can be connected via a bus or other means.

[0099] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the navigation method of the unmanned device in any embodiment of this application (e.g., the first image acquisition module 31, the first navigation line generation module 32, and the movement control module 33 in the navigation device of the unmanned device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 42 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0100] The communication device 43 is used for data transmission.

[0101] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the navigation method of the unmanned device described above.

[0102] Input device 44 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 45 may include display devices such as a display screen.

[0103] The unmanned equipment provided above can be used to execute the navigation method of the unmanned equipment provided in the above embodiments, and has corresponding functions and beneficial effects.

[0104] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute a navigation method for an unmanned device. The navigation method for the unmanned device includes: acquiring a first image collected by a binocular vision sensor, the binocular vision sensor being installed on both sides of the unmanned device, the content of the first image including a first region in front of the unmanned device; constructing an edge point cloud of the first region based on the first image; determining a first navigation line for the unmanned device based on the edge point cloud, the edge point cloud being used to represent the edge information of crops on both sides of the first region; and controlling the unmanned device to move according to the first navigation line.

[0105] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0106] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the navigation method of the unmanned device described above, but can also execute related operations in the navigation method of the unmanned device provided in any embodiment of this application.

[0107] The navigation device, storage medium, and unmanned equipment provided in the above embodiments can execute the navigation method of the unmanned equipment provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the navigation method of the unmanned equipment provided in any embodiment of this application.

[0108] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A navigation method for unmanned equipment, characterized in that, include: A first image is acquired by a binocular vision sensor, which is installed on both sides of the unmanned device. The content of the first image includes a first area in front of the unmanned device. Based on the first image, an edge point cloud of the first region is constructed, and the edge point cloud is projected onto a first plane to obtain two-dimensional edge points. The first plane is the plane where the unmanned device is located. Based on the central line of sight of the binocular vision sensor, the first region is divided into a second region and a third region; the second region and the third region are divided into multiple grids based on a preset interval, and two-dimensional edge points falling into the grids are determined; A first edge line is fitted based on the two-dimensional edge point closest to the central line of sight in each of the grids in the second region, and a second edge line is fitted based on the two-dimensional edge point closest to the central line of sight in each of the grids in the third region; a first navigation line of the unmanned device is determined based on the first edge line and the second edge line, and the edge point cloud is used to characterize the edge information of crops on both sides in the first region; The unmanned equipment is controlled to move according to the first navigation line.

2. The navigation method for unmanned equipment according to claim 1, characterized in that, Determining the first navigation line of the unmanned device based on the edge line includes: The center line between the first edge line and the second edge line is defined as the first navigation line of the unmanned equipment.

3. The navigation method for unmanned equipment according to any one of claims 1-2, characterized in that, The method further includes: A second image is acquired by a monocular vision sensor, which is installed in the middle of the unmanned device. The content of the second image includes a fourth region in front of the unmanned device. Based on the second image, determine the second navigation line of the unmanned device; The first navigation line and the second navigation line are merged to obtain a third navigation line, which is used to guide the unmanned equipment to move.

4. The navigation method for unmanned equipment according to claim 3, characterized in that, Determining the second navigation line of the unmanned device based on the second image includes: The second image is converted to obtain a bird's-eye view of the first region; The fifth region in the bird's-eye view is determined, and the second navigation line of the unmanned equipment is determined based on the fifth region.

5. The navigation method for unmanned equipment according to claim 4, characterized in that, Determining the fifth region in the bird's-eye view includes: The bird's-eye view is segmented using a preset semantic segmentation model to extract the fifth region from the bird's-eye view.

6. The navigation method for unmanned equipment according to claim 3, characterized in that, The step of fusing the first navigation line and the second navigation line to obtain the third navigation line includes: The third navigation line is determined based on the first navigation line and its corresponding first confidence level, and the second navigation line and its corresponding second confidence level.

7. The navigation method for unmanned equipment according to claim 6, characterized in that, Before determining the third navigation line based on the first navigation line and its corresponding first confidence level, and the second navigation line and its corresponding second confidence level, the method further includes: Based on the central line of sight of the binocular vision sensor, the two-dimensional edge points are divided into grids in the second and third regions. The second and third regions are obtained by dividing the first region based on the central line of sight of the binocular vision sensor. The two-dimensional edge points are generated by projecting the edge point cloud onto the first plane. The first confidence level is determined based on the total number of grids in the second region and the third region, and the total number of two-dimensional edge points in each grid that are closest to the central line of sight.

8. The navigation method for unmanned equipment according to claim 6, characterized in that, Before determining the third navigation line based on the first navigation line and its corresponding first confidence level, and the second navigation line and its corresponding second confidence level, the method further includes: The second confidence level is determined by the confidence level output when extracting the fourth region based on the preset semantic segmentation model.

9. The navigation method for unmanned equipment according to claim 1, characterized in that, The step of controlling the unmanned device to move according to the first navigation line includes: Based on the edge point cloud and the first navigation line, determine whether there is an obstacle in front of the unmanned device; When there is an obstacle in front of the unmanned equipment, control the unmanned equipment to move around the obstacle or stop moving.

10. The navigation method for unmanned equipment according to claim 1, characterized in that, The step of controlling the unmanned device to move according to the first navigation line includes: The unmanned device is controlled to move based on its RTK location information and the first navigation line.

11. A navigation device for unmanned equipment, characterized in that, include: The first image acquisition module is configured to acquire a first image collected by a binocular vision sensor, the binocular vision sensor being installed on both sides of the unmanned device, and the content of the first image including a first area in front of the unmanned device. The first navigation line generation module is configured to construct an edge point cloud of the first region based on the first image, and project the edge point cloud onto a first plane to obtain two-dimensional edge points. The first plane is the plane where the unmanned device is located. Based on the central line of sight of the binocular vision sensor, the first region is divided into a second region and a third region; the second region and the third region are divided into multiple grids based on a preset interval, and two-dimensional edge points falling into the grids are determined; A first edge line is fitted based on the two-dimensional edge point closest to the central line of sight in each of the grids in the second region, and a second edge line is fitted based on the two-dimensional edge point closest to the central line of sight in each of the grids in the third region; a first navigation line of the unmanned device is determined based on the first edge line and the second edge line, and the edge point cloud is used to characterize the edge information of crops on both sides in the first region; The motion control module is configured to control the unmanned device to move according to the first navigation line.

12. An unmanned device, characterized in that, include: One or more processors; A memory that stores one or more programs that, when executed by one or more processors, cause the one or more processors to implement the navigation method for an unmanned device as described in any one of claims 1-10.

13. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the navigation method for an unmanned device as described in any one of claims 1-10.

Images