Unmanned aerial vehicle binocular vision obstacle avoidance system and ranging and obstacle avoidance method thereof

Abstract

This invention relates to the field of UAV visual obstacle avoidance technology, specifically to a UAV binocular visual obstacle avoidance system and its ranging and obstacle avoidance methods. By adding a laser module to a dual-UAV binocular visual obstacle avoidance system to emit customized patterns combined with filters, it solves the obstacle avoidance problem in low light or when binocular vision is abnormal. The laser module can supplement environmental texture and light, allowing binocular vision to function normally again. Alternatively, a monocular ranging and coordination scheme can be used, projecting the laser pattern onto the reflective surface of the obstacle. Since the projected pattern is smaller the closer the distance, and larger the distance the projected pattern, this invention utilizes this characteristic to perform monocular ranging, assisting binocular ranging, providing more accurate distance data to obstacles ahead, and reducing the probability of obstacle avoidance failure. It achieves flexible adaptation between normal binocular operation and abnormal switching, significantly improving the stability of UAV obstacle avoidance and its adaptability to complex environments.

Description

Technical Field

[0001] This invention relates to the field of visual obstacle avoidance technology for unmanned aerial vehicles (UAVs), and in particular to a binocular visual obstacle avoidance system for UAVs and its ranging and obstacle avoidance methods. Background Technology

[0002] Current binocular vision obstacle avoidance technology solutions for drones mostly adopt pure vision solutions, or solutions that combine vision and ranging sensors.

[0003] A purely vision-based approach relies entirely on video images captured by binocular cameras to calculate the distance to obstacles ahead, thus achieving obstacle avoidance. While this approach can provide complex depth information and enable advanced functions like path planning when textures are rich and non-repetitive, its drawbacks are significant: it is heavily dependent on lighting and texture. In low light conditions, visual obstacle avoidance fails, and in situations with weak or highly repetitive textures, the probability of ranging errors is very high, potentially rendering it unusable. Therefore, visual obstacle avoidance is ineffective in certain scenarios, leading to the development of an alternative approach: vision combined with a ranging sensor.

[0004] The cost of ranging sensors varies considerably. Generally, single-point ranging sensors, such as ToF, ultrasonic, and millimeter-wave radar, have relatively low costs, while area array ranging sensors, such as lidar, have higher costs, making them unaffordable for low- to mid-range drones.

[0005] The pure vision plus range sensor solution effectively solves the drawbacks of the pure vision solution, but it also has its limitations. Single-point range sensors can only measure the distance to one point, offering limited assistance for visual obstacle avoidance in complex environments. Area array range sensors, on the other hand, are relatively expensive. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs) and its ranging and obstacle avoidance methods, so as to solve the problem of high cost of existing pure vision plus ranging sensor solutions.

[0007] To achieve the above objectives, the present invention provides a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), including a binocular camera, and further comprising:

[0008] Laser modules are used to emit laser patterns with custom shapes or custom texture patterns;

[0009] A filter, corresponding to the laser wavelength emitted by the laser module, is set in the lens of the binocular camera so that the binocular camera can capture the laser pattern emitted by the laser module.

[0010] The monocular ranging module is used to extract the laser pattern from the image captured by the binocular camera and calculate the distance from the obstacle reflector to the laser module based on the size of the laser pattern.

[0011] Preferably, the laser emitted by the laser module is invisible light.

[0012] This invention also provides an obstacle ranging method for a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), comprising the following steps:

[0013] The drone emits customized laser patterns using a laser module installed in its binocular vision obstacle avoidance system.

[0014] The system acquires images and extracts laser patterns from them. Based on the distances of preset feature points in the laser patterns, it performs monocular ranging.

[0015] Preferably, the monocular ranging adopts the approximate triangle principle, and calculates the distance from the obstacle reflector to the laser module based on the distance between feature points in the pre-calibrated distance and the real-time captured image with laser pattern.

[0016] The distance calibration steps include:

[0017] A laser pattern is emitted from a laser module toward a reflective surface at a specified distance.

[0018] The laser pattern emitted by the laser module was captured by a binocular vision system of a drone;

[0019] Identify feature points of laser patterns from images captured by any camera in a drone's binocular vision system, record the distances between these feature points, and perform distance calibration.

[0020] Preferably, the feature points include at least one reference point and multiple endpoints that are equidistant from the reference point, or include multiple sets of one-to-one corresponding reference points, with the distance between each set of corresponding feature points being equal.

[0021] When the feature points include at least one reference point and multiple endpoints equidistant from the reference point, the process of identifying the feature points of the laser pattern, recording the distances between the feature points, and performing distance calibration includes:

[0022] Multiple endpoints and reference points are extracted to obtain the distance values ​​between each endpoint and the reference point. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method.

[0023] When the feature points include multiple sets of one-to-one corresponding reference points, and the distance between each set of corresponding feature points is the same, the process of identifying the feature points of the laser pattern, recording the distance between the feature points, and performing distance calibration includes:

[0024] Multiple sets of corresponding reference points are extracted to obtain the distance values ​​between the corresponding reference points in each set. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method.

[0025] Preferably, the obstacle ranging method further includes:

[0026] The distance calculated by monocular ranging is compared with the obstacle distance calculated by binocular vision. If they are the same, there is no need to recalibrate the distance. If the difference between the calculated distances is greater than the set value, the information that the distance needs to be recalibrated is fed back. Before recalibrating the distance, obstacle avoidance is performed using the distance data calculated by monocular ranging.

[0027] This invention also provides an obstacle avoidance method for a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), comprising the following steps:

[0028] When the binocular vision obstacle avoidance system of the UAV is working normally, it uses binocular vision to calculate the depth information of the environment and then performs obstacle avoidance actions or path planning tasks.

[0029] When the ambient light is lower than the set value or the binocular vision obstacle avoidance is abnormal, the laser module is activated to emit a customized texture pattern to supplement the ambient light or texture, so that the binocular vision can resume normal operation. Alternatively, the obstacle distance data can be measured using the obstacle ranging method mentioned above to avoid obstacles.

[0030] The beneficial effects of this invention are as follows: This invention solves the obstacle avoidance problem in low light or when binocular vision is abnormal by adding a laser module to a dual-UAV binocular vision obstacle avoidance system and emitting a customized pattern combined with a filter. The laser module can supplement environmental texture and light, allowing the binocular vision to function normally again. Alternatively, a monocular ranging collaborative scheme can be adopted, projecting the laser pattern onto the reflective surface of the obstacle. Since the projected pattern is smaller the closer the distance, and larger the distance the projected pattern, this invention utilizes this characteristic to perform monocular ranging and assist binocular ranging, providing more accurate distance data to obstacles ahead and reducing the probability of obstacle avoidance failure. It achieves flexible adaptation between normal binocular operation and abnormal switching, significantly improving the stability of UAV obstacle avoidance and its adaptability to complex environments. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the ranging method of the UAV binocular vision obstacle avoidance system according to an embodiment of the present invention;

[0033] Figure 2 The crosshair pattern captured by the drone at close range in low light in an embodiment of the present invention;

[0034] Figure 3 This is a crosshair pattern captured by a drone at a distance under normal lighting conditions in an embodiment of the present invention;

[0035] Figure 4 This is an M x N array light spot captured by a drone at close range in low light in an embodiment of the present invention;

[0036] Figure 5 This is an example of an M x N array light spot captured by a drone at a distance under normal light conditions in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0038] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0039] Example 1:

[0040] This specification provides an embodiment of a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), including a binocular camera, and further comprising:

[0041] The laser module is used to emit laser patterns with customized shapes or customized texture patterns. In order to reduce the interference of the laser patterns emitted by the laser module to the user, this embodiment selects a laser module that can emit invisible light, such as common 850nm or 940nm wavelength light. The customized texture pattern here refers to a pattern with obvious regularity, such as cross lines, star lines, M x N array light spots, etc.

[0042] In order for the binocular camera to capture the pattern emitted by the laser module, this system has a filter on the lens of the binocular camera, and the filter corresponds to the wavelength of the laser emitted by the laser module.

[0043] The monocular ranging module is used to extract the laser pattern from the image captured by the binocular camera and calculate the distance from the obstacle reflector to the laser module based on the size of the laser pattern.

[0044] When the ambient light is sufficient and the textures are rich and non-repetitive, pure binocular vision can be used to calculate the depth information of the environment in order to perform obstacle avoidance or path planning tasks.

[0045] When the ambient light is so dark that the pure binocular vision solution cannot work properly, or when the environment has a simple texture or no texture at all, the laser module is activated to emit a customized texture pattern to supplement the ambient light or texture, so that the binocular vision can work normally again.

[0046] This system can also employ a monocular ranging cooperative scheme, where the laser module emits a customized pattern. Because the laser module emits a customized pattern, such as a crosshair or a speckle array, the size of this pattern varies depending on the reflective surface it is projected onto. The closer the distance, the smaller the projected pattern; the farther the distance, the larger the projected pattern. This embodiment utilizes this characteristic to perform monocular ranging, assisting binocular ranging, providing more accurate distance data to obstacles ahead and reducing the probability of obstacle avoidance failure.

[0047] Example 2:

[0048] like Figure 1 As shown, this embodiment provides an obstacle ranging method for a UAV binocular vision obstacle avoidance system. Based on the UAV binocular vision obstacle avoidance system provided in Embodiment 1, the method includes:

[0049] The drone emits customized laser patterns using a laser module installed in its binocular vision obstacle avoidance system.

[0050] Acquire images and extract laser patterns from them. Perform monocular ranging based on the distances of preset feature points within the laser pattern. This includes:

[0051] Before the drone operates, distance calibration should be performed in advance. For example, place a white cardboard piece 1 meter away from the laser module, turn on the binocular camera and laser module, and use the binocular camera to photograph the crosshair pattern emitted by the laser module in low light conditions. Figure 2 and Figure 3 The images shown are crosshair patterns captured at close range in low light and at a distance in normal light, respectively.

[0052] Extract the center point of the crosshair and the feature points of its four endpoints from the image captured by the binocular camera (either the left or right image). Calculate the distance from each endpoint to the center point in pixels. Normally, the distances from the four endpoints to the center point should be the same or very similar. To prevent unexpected situations (e.g., one endpoint's projection distance is 2 meters, while the other points' projection distances are 1 meter, a significant difference), use a histogram statistical filtering method to select the distance value with the highest probability, denoted as d0. This process is for the distance calibration of the laser module.

[0053] During normal use, if the environmental texture meets the requirements of binocular vision, the distance information provided by binocular vision is trusted. If a clear crosshair emitted by a laser emitter can be extracted from the image, the feature points of the four endpoints and the center point of the crosshair are extracted. Then, the pixel distance from the endpoints to the center point is calculated by histogram statistical filtering and denoted as d.

[0054] Based on the principle of similar triangles, the distance from the reflecting surface to the laser emitting module... , L=d / d0.

[0055] Compare the distance calculated by monocular ranging with the obstacle distance calculated by binocular vision. If they match, it indicates that the current binocular module has high accuracy and does not require recalibration. If there is a persistent and significant difference, it indicates that the accuracy of the binocular module has decreased and recalibration is required. Before recalibration, choose to trust the distance data calculated by monocular ranging for obstacle avoidance.

[0056] In another implementation, if the pattern emitted by the laser module is not a crosshair but a speckle array, multiple groups of equally spaced points in the speckle array can be used as reference points for each group. When extracting feature points, the distance values ​​between these reference points will be obtained. To prevent errors, a histogram statistical filtering method is used, selecting the distance value with the highest probability as d0. Subsequent steps are the same as in the crosshair scheme. Figure 4 and Figure 5 The images shown are M x N array light spots captured at close range in low light and at a distance in normal light, respectively.

[0057] Example 3:

[0058] This embodiment provides an obstacle avoidance method for the above-mentioned UAV binocular vision obstacle avoidance system, including the following steps:

[0059] When the binocular vision obstacle avoidance system of the UAV is working normally, it uses binocular vision to calculate the depth information of the environment and then performs obstacle avoidance actions or path planning tasks.

[0060] When the ambient light brightness is lower than the set value or the binocular vision obstacle avoidance is abnormal, the laser module is activated to emit a customized texture pattern to supplement the ambient light or texture, so that the binocular vision can resume normal operation. Alternatively, the obstacle distance data can be measured using the obstacle ranging method in Embodiment 2 above to avoid obstacles.

[0061] This invention uses a pure binocular vision system with a laser module to supplement patterns or textures, thus solving the problem of obstacle avoidance failure in low-light, no-light, textureless, or repetitive texture environments.

[0062] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in the details for the sake of brevity.

[0063] This invention is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A binocular visual obstacle avoidance system for unmanned aerial vehicles (UAVs), comprising binocular cameras, characterized in that, Also includes: Laser modules are used to emit crosshair, star, or dot array patterns. A filter, corresponding to the laser wavelength emitted by the laser module, is set in the lens of the binocular camera so that the binocular camera can capture the laser pattern emitted by the laser module. The monocular ranging module is used to extract the laser pattern in the image captured by the binocular camera and calculate the distance from the obstacle reflector to the laser module based on the size of the laser pattern. The monocular ranging uses the approximate triangle principle, based on the distance calibration in advance and the distance between feature points in the real-time captured image with laser pattern, to calculate the distance from the obstacle reflector to the laser module; The distance calibration steps include: A laser pattern is emitted from a laser module toward a reflective surface at a specified distance. The laser pattern emitted by the laser module was captured by a binocular vision system of a drone; Identify feature points of laser patterns from images captured by any camera in a drone's binocular vision system, record the distances between feature points, and perform distance calibration. The feature points include at least one reference point and multiple endpoints that are equidistant from the reference point, or include multiple sets of one-to-one corresponding reference points, with the distance between each set of corresponding feature points being equal. When the feature points include at least one reference point and multiple endpoints equidistant from the reference point, the process of identifying the feature points of the laser pattern, recording the distances between the feature points, and performing distance calibration includes: Multiple endpoints and reference points are extracted to obtain the distance values ​​between each endpoint and the reference point. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method. When the feature points include multiple sets of one-to-one corresponding reference points, and the distance between each set of corresponding feature points is the same, the process of identifying the feature points of the laser pattern, recording the distance between the feature points, and performing distance calibration includes: Multiple sets of corresponding reference points are extracted to obtain the distance values ​​between the corresponding reference points in each set. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method.

2. The UAV binocular vision obstacle avoidance system according to claim 1, characterized in that, The laser emitted by the laser module is invisible light.

3. An obstacle ranging method for a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: The drone emits customized laser patterns using a laser module installed in its binocular vision obstacle avoidance system. Acquire images and extract laser patterns from them; perform monocular ranging based on the distances of preset feature points in the laser patterns. The monocular ranging adopts the approximate triangle principle, and calculates the distance from the obstacle reflector to the laser module based on the distance between feature points in the pre-calibrated distance and the real-time captured image with laser pattern. The distance calibration steps include: A laser pattern is emitted from a laser module toward a reflective surface at a specified distance. The laser pattern emitted by the laser module was captured by a binocular vision system of a drone; Identify feature points of laser patterns from images captured by any camera in a drone's binocular vision system, record the distances between feature points, and perform distance calibration. The feature points include at least one reference point and multiple endpoints that are equidistant from the reference point, or include multiple sets of one-to-one corresponding reference points, with the distance between each set of corresponding feature points being equal. When the feature points include at least one reference point and multiple endpoints equidistant from the reference point, the process of identifying the feature points of the laser pattern, recording the distances between the feature points, and performing distance calibration includes: Multiple endpoints and reference points are extracted to obtain the distance values ​​between each endpoint and the reference point. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method. When the feature points include multiple sets of one-to-one corresponding reference points, and the distance between each set of corresponding feature points is the same, the process of identifying the feature points of the laser pattern, recording the distance between the feature points, and performing distance calibration includes: Multiple sets of corresponding reference points are extracted to obtain the distance values ​​between the corresponding reference points in each set. The distance value with the highest probability is selected for calibration using the histogram statistical filtering method.

4. The obstacle ranging method of the UAV binocular vision obstacle avoidance system according to claim 3, characterized in that, The obstacle ranging method also includes: The distance calculated by monocular ranging is compared with the obstacle distance calculated by binocular vision. If they are the same, there is no need to recalibrate the distance. If the difference between the calculated distances is greater than the set value, the information that the distance needs to be recalibrated is fed back. Before recalibrating the distance, obstacle avoidance is performed using the distance data calculated by monocular ranging.

5. An obstacle avoidance method for a binocular vision obstacle avoidance system for unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: When the binocular vision obstacle avoidance system of the UAV is working normally, it uses binocular vision to calculate the depth information of the environment and then performs obstacle avoidance actions or path planning tasks. When the ambient light is lower than the set value or the binocular vision obstacle avoidance is abnormal, obstacle distance data is measured by the obstacle distance measurement method as described in any one of claims 3-4 to avoid obstacles.

Images