Unmanned aerial vehicle adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception

Abstract

The application relates to the technical field of unmanned aerial vehicle control, and discloses an unmanned aerial vehicle adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception, which comprises the following steps: synchronously collecting left images and right images; performing frequency domain analysis on a row gray mean sequence of the left images to obtain a pulse width modulation period and a corresponding stripe row distance; performing cross-correlation operation on the row gray mean sequences of the left images and the right images to determine an acquisition deviation; performing phase positioning according to the stripe row distance to determine a stripe reference row, and taking the stripe reference row as a benchmark to perform cyclic calibration according to the stripe row distance and construct a row-level mask; performing block calculation original parallax on the left images and the right images, and correcting the original parallax based on the row-level mask to obtain corrected parallax; configuring a confidence for each pixel according to the row-level mask, projecting the corrected parallax into point cloud, and generating an occupancy probability by voxel division; constructing an obstacle potential field based on the occupancy probability, and outputting a control instruction according to the obstacle potential field to complete unmanned aerial vehicle obstacle avoidance.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and more specifically, to an adaptive intelligent navigation and obstacle avoidance system for UAVs based on multimodal perception. Background Technology

[0002] Modern urban building facades often feature high-brightness LED displays and lighting equipment. These light sources typically use pulse width modulation (PWM) technology for brightness adjustment, with operating frequencies ranging from hundreds of Hz to tens of kHz. Meanwhile, to reduce sensor power consumption and improve imaging speed, most lightweight unmanned aerial vehicle (UAV) platforms utilize rolling shutter CMOS cameras as their visual input devices.

[0003] When a drone's rolling shutter binocular camera images a high-frequency PWM LED light source, the following situation may occur:

[0004] 1. Rolling shutter cameras do not expose the entire frame simultaneously during image acquisition. Instead, they capture images line by line, either from top to bottom or left to right, with a time difference of several to tens of microseconds between the start of each line's exposure. If a PWM light source is present in the scene, its brightness state may differ at each exposure moment. Consequently, the brightness states recorded in different scan lines will exhibit alternating bright and dark distributions, forming bright or dark bands with fixed intervals in the image. This banded artifact is produced by the mixing effect formed during the spatial scanning process of the temporal dimming signal.

[0005] 2. In a stereo vision system, the two cameras should ideally be exposed synchronously to acquire parallax information. However, due to cost and hardware design limitations, binocular cameras installed on commercial drones often cannot achieve perfect synchronization, resulting in a time deviation of several microseconds between their triggering moments. Because the LED light source is in a high-frequency flickering state, even a tiny time difference can cause a phase shift in the bright and dark bands in the left and right images.

[0006] 3. Most drone obstacle avoidance systems use disparity matching algorithms based on pixel grayscale or local texture, such as block matching, SAD (Single Absolute Difference), and Census transform. These algorithms assume that grayscale changes within the matching area are due to the shape or structure of objects and derive the disparity values ​​of corresponding pixels in the left and right images accordingly. However, the bright and dark bands caused by rolling shutter imaging do not originate from the actual geometric differences of the scene, but are artificial brightness modulations. The disparity matching process interprets these brightness bands as object edges or three-dimensional structures, thus producing unrealistic disparity solutions, misjudging bright surfaces that should be in the distance as nearby obstacles.

[0007] The direct consequence of the above problems is the appearance of large areas of well-distributed false near obstacles in the depth map, whose depth values ​​are usually significantly smaller than the true values. In the obstacle avoidance logic, the system treats these as obstacles about to be collided with, triggering avoidance actions, leading to the following problems:

[0008] 1. The aircraft mistakenly avoids obstacles or makes an abrupt stop, affecting mission stability;

[0009] 2. Path planning is interrupted or enters a backtracking loop, reducing track efficiency;

[0010] 3. Repeated obstacle avoidance errors caused the aircraft's attitude to oscillate, inducing control instability;

[0011] 4. It poses a real risk of collision in confined spaces. Summary of the Invention

[0012] This invention provides an adaptive intelligent navigation and obstacle avoidance system for unmanned aerial vehicles (UAVs) based on multimodal perception, which solves the technical problems mentioned in the background art.

[0013] This invention provides an adaptive intelligent navigation and obstacle avoidance system for unmanned aerial vehicles (UAVs) based on multimodal perception, comprising:

[0014] The data acquisition module is used to simultaneously acquire the left and right images;

[0015] The data processing module is used to perform frequency domain analysis based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing.

[0016] The time difference analysis module is used to perform cross-correlation calculations on the mean grayscale values ​​of the left and right images to determine the acquisition bias.

[0017] The interference marking module is used to perform phase positioning according to the stripe row spacing, determine the stripe reference row, and construct a row-level mask by cyclically calibrating according to the stripe row spacing based on the stripe reference row.

[0018] The disparity correction module is used to perform block calculations on the left and right images to obtain the original disparity, and then perform correction processing on the original disparity based on the row-level mask to obtain the corrected disparity.

[0019] The confidence calculation module is used to assign confidence to each pixel based on the row-level mask, and to generate occupancy probability by dividing the corrected parallax into point cloud and then dividing it into voxels.

[0020] The drone obstacle avoidance module is used to construct an obstacle potential field based on the occupancy probability and output control commands based on the obstacle potential field to complete the drone obstacle avoidance.

[0021] Furthermore, frequency domain analysis is performed based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing, including:

[0022] Calculate the average gray level for each row of pixels in the left image to form a row gray level average sequence;

[0023] The amplitude spectrum is obtained by performing a fast Fourier transform on the gray-level mean sequence.

[0024] Determine the dominant frequency index ks of the amplitude spectrum, and calculate the stripe row spacing Δs = 1 / ks based on the dominant frequency index;

[0025] The pulse width modulation period TP = Δs × Trow is calculated using the prior readout period Trow.

[0026] Furthermore, cross-correlation is performed on the mean grayscale sequences of the left and right images to determine the acquisition bias, including:

[0027] Calculate the cross-correlation value between the row gray-level mean sequence element y of the left image and the row gray-level mean sequence element y of the right image under the row offset δ;

[0028] Determine the range of values ​​for the row offset δ, iterate through the values ​​of the row offset δ, and find the row offset δ that results in the largest cross-correlation value as the target row offset δ*.

[0029] The acquisition bias is calculated based on the target row offset δ*, as follows:

[0030]

[0031] in, This indicates a data acquisition bias.

[0032] Furthermore, phase positioning is performed according to the stripe row spacing to determine the stripe reference row. Using the stripe reference row as a reference, cyclic calibration is performed according to the stripe row spacing to construct a row-level mask, including:

[0033] Within the row index interval [1, Δs], calculate the local brightness contrast of candidate row yc;

[0034] The candidate row corresponding to the maximum local brightness contrast is used as the stripe reference row yref;

[0035] Calculate the modulus value for each row of pixels in the left image based on the stripe reference row;

[0036] If the modulus of the y-th row pixel is less than the preset effective bandwidth of the stripe, then mark the y-th row pixel in the left image as 1; otherwise, mark the y-th row pixel in the left image as 0 to obtain the row-level mask.

[0037] Furthermore, the original disparity is calculated block by block for the left and right images, and the original disparity is corrected based on the row-level mask to obtain the corrected disparity, including:

[0038] For pixel coordinates (x, y), perform block matching with a fixed window size of w×w between the left and right images to obtain the original disparity;

[0039] The original disparity draw(x,y) is sign-flipped using a row-level mask to obtain the corrected disparity.

[0040] Furthermore, confidence is assigned to each pixel based on the row-level mask, including:

[0041] Calculate the row-level baseline confidence level R0(y) based on the row-level mask;

[0042] Generate a disparity validity label V(x,y) based on corrected disparity;

[0043] For pixel coordinates (x, y), a weighted fusion is performed based on row-level baseline confidence and disparity validity label to obtain the fused pixel confidence.

[0044] The maximum value of the fused pixel confidence is determined, and the fused pixel confidence of each pixel coordinate is normalized based on the maximum value to obtain the confidence Rref(x,y) of each pixel coordinate.

[0045] Furthermore, after correcting the parallax projection into a point cloud, occupancy probabilities are generated by voxel partitioning, including:

[0046] Pixel depth is calculated from the corrected parallax.

[0047] Generate a 3D point cloud P(x,y) based on pixel depth;

[0048] Set the voxel side length svox and calculate the voxel index (a,b,c);

[0049] Determine the confidence level and value S1 of the point cloud in each voxel, and the number of point clouds S2;

[0050] The ratio of S1 to S2 is used as the occupancy probability of the corresponding voxel.

[0051] Furthermore, a barrier potential field is constructed based on the occupancy probability, including:

[0052] Determine the coordinates of the voxel center (Xc, Yc, Zc);

[0053] The distance attenuation factor is determined based on the voxel center coordinates;

[0054] The original potential energy is obtained by weighting the occupancy probability of each voxel according to the distance decay factor.

[0055] The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain a smoothed potential energy.

[0056] The smooth potential energy of the voxels is binarized to obtain the barrier potential field Uobs.

[0057] Furthermore, obstacle avoidance for the UAV is accomplished by outputting control commands based on the obstacle potential field, including:

[0058] Potential energy Up is obtained by interpolation based on the spatial position of the drone;

[0059] By combining the potential energy Up and the obstacle potential field Uobs, the potential energy gradient ∇Uobs is obtained using the three-dimensional finite difference method;

[0060] Calculate the avoidance vector based on the potential energy gradient. ;

[0061] The desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector;

[0062] The saturation velocity is obtained by saturating the amplitude of the desired velocity.

[0063] The saturation velocity is resolved into pitch angle commands, roll angle commands, and vertical velocity commands, which are then processed by the UAV's flight control execution unit.

[0064] The beneficial effects of this invention are as follows: By constructing an image processing and obstacle avoidance control process, the system can accurately identify and eliminate periodic false parallax signals even in the presence of high-frequency PWM LED light interference and asynchronous acquisition errors of the rolling shutter, effectively preventing false obstacles from triggering obstacle avoidance commands. Based on frequency domain analysis to extract fringe parameters, cross-correlation calculation of acquisition deviation, row-level masking to mark interference areas, parallax sign correction, and confidence weight allocation, this invention ultimately constructs a continuous obstacle potential field and outputs dynamic control commands, achieving closed-loop control from interference detection to flight control avoidance. This significantly improves the obstacle avoidance reliability and trajectory stability of UAVs in complex urban lighting environments. Attached Figure Description

[0065] Figure 1 This is a flowchart of the present invention;

[0066] Figure 2 This is a block diagram of the present invention;

[0067] Figure 3 This is a flowchart of the drone obstacle avoidance process of the present invention. Detailed Implementation

[0068] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0069] like Figures 1 to 3 As shown, the UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception includes:

[0070] The data acquisition module is used to simultaneously acquire the left and right images;

[0071] The data processing module is used to perform frequency domain analysis based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing.

[0072] The time difference analysis module is used to perform cross-correlation calculations on the mean grayscale values ​​of the left and right images to determine the acquisition bias.

[0073] The interference marking module is used to perform phase positioning according to the stripe row spacing, determine the stripe reference row, and construct a row-level mask by cyclically calibrating according to the stripe row spacing based on the stripe reference row.

[0074] The disparity correction module is used to perform block calculations on the left and right images to obtain the original disparity, and then perform correction processing on the original disparity based on the row-level mask to obtain the corrected disparity.

[0075] The confidence calculation module is used to assign confidence to each pixel based on the row-level mask, and to generate occupancy probability by dividing the corrected parallax into point cloud and then dividing it into voxels.

[0076] The drone obstacle avoidance module is used to construct an obstacle potential field based on the occupancy probability and output control commands based on the obstacle potential field to complete the drone obstacle avoidance.

[0077] In one embodiment of the present invention, frequency domain analysis is performed based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing, including:

[0078] Calculate the average gray level for each row of pixels in the left image to form a row gray level average sequence;

[0079] The amplitude spectrum is obtained by performing a fast Fourier transform on the gray-level mean sequence.

[0080] Determine the dominant frequency index ks of the amplitude spectrum, and calculate the stripe row spacing Δs = 1 / ks based on the dominant frequency index;

[0081] The pulse width modulation period TP = Δs × Trow is calculated using the prior readout period Trow.

[0082] In detail, due to the row-by-row exposure characteristics of the rolling shutter, the exposure time of each row of pixels in the left image is temporally correlated with the pulse width modulation period of the high-frequency LED: when the LED is on, the gray level of the corresponding row of pixels is generally higher; when it is off, the gray level of the corresponding row of pixels is generally lower. Calculating the average gray level of each row of pixels in the left image can eliminate column-direction noise interference and condense the gray level fluctuations between rows caused by the alternation of LED on and off into a row gray level mean sequence (denoted as gL(y), where y is the row index). The periodicity of this sequence directly reflects the spatiotemporal coupling relationship between the LED PWM signal and the rolling shutter exposure.

[0083] In detail, the periodicity of the gray-scale mean sequence is represented in the time domain as a repeating pattern of bright lines and dark lines. This time-domain periodicity can be converted into a frequency-domain amplitude spectrum using FFT. In the frequency domain, periodic signals exhibit significant peaks (dominant frequencies), whose frequencies correspond to the periodic characteristics of the signal.

[0084] If the PWM period of the LED is TP and the rolling shutter line reading period is Trow, then when the LED completes one on / off cycle, the rolling shutter will sweep across Δs lines (i.e., the stripe line spacing). Therefore, the spatial period of the line grayscale mean sequence is Δs lines, which corresponds to the dominant frequency component in the frequency domain.

[0085] In detail, in the amplitude spectrum obtained by FFT, the peak with the largest amplitude is searched, and its corresponding dominant frequency index ks reflects the spatial frequency of the row gray-scale mean sequence (unit: row negative one, i.e., the number of cycles per unit row). Since the spatial period (stripe row spacing Δs) is inversely related to the spatial frequency, therefore: Δs = 1 / ks; for example: if the dominant frequency index ks = 0.02, then the bright and dark stripe pattern repeats once every 50 rows (1 / 0.02), that is, the stripe row spacing Δs = 50 rows.

[0086] In detail, the line reading period Trow is an inherent hardware parameter (representing the exposure time difference between two adjacent lines). Since the stripe line spacing Δs corresponds to the number of lines swept by the rolling shutter when the LED completes one PWM cycle (on and off once), the relationship between the PWM period TP and Δs and Trow is: TP = Δs × Trow. Through this formula, the stripe line spacing in the spatial domain can be converted into the PWM period in the time domain, establishing a spatiotemporal correspondence between the LED signal period and the image line structure.

[0087] In one embodiment of the present invention, cross-correlation is performed on the mean grayscale sequences of the left and right images to determine the acquisition bias, including:

[0088] The cross-correlation value of the row gray-level mean sequence element y of the left image and the row gray-level mean sequence element y of the right image under the row offset δ is calculated as follows:

[0089]

[0090] Where C(δ) represents the cross-correlation value corresponding to the row offset δ, gL(y) represents the gray mean sequence element y of the left image row, gR(y) represents the gray mean sequence element y of the right image row, and gR(y+δ) represents the gray mean sequence element y+δ of the right image row.

[0091] Determine the range of values ​​for the row offset δ, iterate through the values ​​of the row offset δ, and find the row offset δ that results in the largest cross-correlation value as the target row offset δ*.

[0092] The acquisition bias is calculated based on the target row offset δ*, as follows:

[0093]

[0094] in, This indicates a data acquisition bias.

[0095] In detail, the mean gray level of the left image is calculated row by row to obtain the sequence gL(y) (where y is the row index); similarly, gR(y) is generated for the right image. This sequence has filtered out column noise and only retains the row brightness pattern (the brightness period of the LED stripes).

[0096] Detailed cross-correlation calculation: Define row offset δ, and calculate the sum of the products of the grayscale mean values ​​corresponding to all valid rows y: The logic is that when δ equals the actual misalignment of the left and right image row patterns, the brightness patterns of gLy) and gR(y+δ) are highly similar, and the product and C(δ) reach the global maximum value.

[0097] In detail, based on the consistency of row counts in the left and right images, the search range of δ is set as follows: (H is the number of rows in the image) to ensure coverage of all possible row misalignments. Iterate through each δ within the above range, calculate the corresponding cross-correlation value, and select the δ that maximizes the cross-correlation value as the target row offset δ*. This utilizes the periodicity and correlation of row patterns, i.e., the cross-correlation value at the actual misalignment will be significantly higher than at other positions due to pattern matching, forming a recognizable peak.

[0098] In detail, the hardware has a known row read cycle Trow (i.e., the time interval between each row of pixels output from the sensor, in seconds). Since δ* represents row-level misalignment, its corresponding time difference (i.e., the deviation τ between the trigger times of the left and right cameras) is: The logic is as follows: the exposure or reading time interval of each row is Trow, so the misalignment of δ* rows corresponds to the time difference of δ*×Trow.

[0099] In one embodiment of the present invention, phase positioning is performed according to the stripe row spacing to determine the stripe reference row, and a row-level mask is constructed by cyclic calibration according to the stripe row spacing based on the stripe reference row, including:

[0100] Within the row index interval [1, Δs], calculate the local brightness contrast of candidate row yc as follows:

[0101]

[0102] Where yc∈[1,Δs], D(yc) represents the local brightness contrast of candidate row yc, and gL(yc) represents the gray-level mean sequence element of the left image row. gL(yc+1) represents the gray-level mean sequence element of the left image row. +1, maxgL represents the maximum value in the gray-scale mean sequence of the left image row, and mingL represents the minimum value in the gray-scale mean sequence of the left image row;

[0103] The candidate row corresponding to the maximum local brightness contrast is used as the stripe reference row yref;

[0104] The magnitude value is calculated for each row of pixels in the left image based on the stripe reference row, as follows:

[0105]

[0106] in, This represents the magnitude of the pixel in the y-th row of the left image. This indicates the modulo operation;

[0107] If the modulus of the y-th row pixel is less than the preset effective bandwidth of the stripe, then mark the y-th row pixel in the left image as 1; otherwise, mark the y-th row pixel in the left image as 0 to obtain the row-level mask.

[0108] In detail, the stripe row spacing Δs represents the period of the LED brightness stripes in the image row direction (i.e., the brightness pattern of the stripes repeats every Δs rows). To locate the initial phase of the stripes, candidate rows need to be searched within the row index interval of a complete stripe period.

[0109] In detail, for each candidate row yc, calculate its local brightness contrast with the next row yc+1:

[0110]

[0111] The grayscale difference between candidate row yc and its next row yc+1 is quantified. If yc is located at the boundary between light and dark stripes, the grayscale difference between adjacent rows will increase significantly.

[0112] It is the global grayscale range of the grayscale mean sequence of the left image row, used for normalization processing, thereby eliminating the absolute numerical influence of different lighting conditions on contrast, so that D(yc) is uniformly mapped to the [0,1] interval, ensuring comparability under different scenes.

[0113] In detail, the fringe reference row is located by searching for the maximum value of local brightness contrast. The boundary row between light and dark areas of the fringe (or the center row) has the largest gray-level difference between adjacent rows. Therefore, the candidate row yc with the largest D(yc) corresponds to the phase starting point of the fringe pattern (such as the edge row of the first bright band). Within a fringe cycle, the fringe pattern changes continuously and periodically, and there exists only one globally largest D(yc), ensuring that yref is unique and providing a fixed phase reference for subsequent cyclic calibration.

[0114] In detail, for each row y in the left image, calculate its period offset relative to the stripe reference row yref: By utilizing the periodicity of the stripes, any row y can be mapped to the interval [1, Δs]. For example, the r value of the yref+kΔs-th row (k is an integer) is the same as that of the yref-th row, corresponding to the same phase of the stripes (such as bright bands, dark bands, or boundary rows), thus achieving single-period phase → full image periodic expansion.

[0115] In detail, the preset effective stripe bandwidth ws (ws=0.3Δs) is used to mark the row-level mask based on the modulus r. The preset effective stripe bandwidth defines the range of rows in the stripe that are affected by LED brightness interference (e.g., a bright band lasting 0.3Δs). When r < ws, the row is in the interference region of the stripe (bright or dark band, depending on the stripe pattern); when r ≥ ws, the row is in the non-interference region of the stripe (stable grayscale, unaffected by LED flicker).

[0116] In one embodiment of the present invention, the original disparity is calculated by performing block calculations on the left and right images, and the original disparity is corrected based on a row-level mask to obtain the corrected disparity, including:

[0117] For pixel coordinates (x, y), perform block matching with a fixed window size of w×w between the left and right images to obtain the original disparity, as follows:

[0118]

[0119] Where draw(x,y) represents the original disparity at pixel coordinates (x,y), IL(x+i,y+i) represents the pixel grayscale value at the left image with pixel coordinates (x,y) as the center, horizontally offset by i and vertically offset by j, IR(x+id,y+i) represents the pixel grayscale value in the right image corresponding to the position (x+i,y+i) in the left image, horizontally offset by disparity d, i and j represent the horizontal and vertical offsets within the block matching window, respectively, d represents the disparity candidate value, d∈D, and D represents the disparity search range;

[0120] The original disparity draw(x,y) is sign-flipped using a row-level mask, as follows:

[0121]

[0122] Where draw(x,y) represents the parallax correction, and M(y) represents element y in the row-level mask.

[0123] In detail, for any pixel coordinate (x, y) in an image, its correspondence in the left and right images needs to be determined to obtain the disparity value. Specifically, a fixed-size window of size w×w is selected in both the left and right images, centered at the pixel coordinate (x, y). The purpose of the window is to improve matching stability through the grayscale information of local regions, avoiding matching errors caused by noise or isolated grayscale changes in individual pixels.

[0124] In detail, i and j represent the horizontal and vertical offsets within the window, respectively, ranging from -w / 2 to w / 2, covering the entire window area. For each position (x+i, y+j) within the left image window, its grayscale value is represented by IL(x+i, y+j); the corresponding position in the right image needs to consider the horizontal disparity candidate value d, i.e., the grayscale value of the corresponding position within the right image window is represented by IR(x+id, y+j). To find the best matching disparity, the absolute difference of the grayscale values ​​of all corresponding pixels within the left and right image windows needs to be calculated, and these absolute differences are summed. All candidate values ​​d within the disparity search range D are traversed. When the sum of the absolute grayscale differences reaches its minimum value, the corresponding candidate value d is the original disparity draw(x, y) at the pixel coordinates (x, y). This is because the smaller the sum of the absolute grayscale differences, the more similar the window areas of the left and right images are, and the more reliable the corresponding disparity value.

[0125] In detail, the parallax correction process is implemented based on row-level masks. The row-level mask M(y) is used to mark rows affected by LED stripe interference. When M(y) is 1, it indicates that the pixels in that row are affected; when M(y) is 0, it indicates that the pixels in that row are normal. The parallax correction dcorr(x,y) is calculated as follows: When M(y) is 1, -1 raised to the power of 1 equals -1, and the corrected disparity is the opposite of the original disparity, thus reversing the sign of the disparity. When M(y) is 0, -1 raised to the power of 0 equals 1, and the corrected disparity is the same as the original disparity. The necessity of sign reversal stems from the disparity errors caused by LED stripe interference. The original disparity of the interfered line will have a sign error due to mismatches between bright and dark stripes, such as misjudging a distant area as a near area (negative disparity). Sign reversal can correct this error, allowing the disparity to accurately reflect the actual depth relationship.

[0126] In one embodiment of the present invention, the confidence level is assigned to each pixel based on the row-level mask, including:

[0127] The row-level baseline confidence score R0(y) is calculated as follows:

[0128]

[0129] in, Represents the row-level suppression coefficient. ;

[0130] Generate the disparity validity label V(x,y) as follows:

[0131]

[0132] Where V(x,y) represents the disparity validity label of pixel coordinates (x,y), and dmin represents the minimum credible disparity threshold;

[0133] For pixel coordinates (x, y), a weighted fusion is performed based on row-level baseline confidence and disparity validity label, as follows:

[0134]

[0135] Where R(x,y) represents the confidence score of the fused pixel, and Rrep(x,y) represents the row-level baseline confidence score of the row to which the pixel coordinate (x,y) belongs;

[0136] The maximum value of the fused pixel confidence is determined, and the fused pixel confidence of each pixel coordinate is normalized based on the maximum value to obtain the confidence Rref(x,y) of each pixel coordinate.

[0137] In detail, the formula for calculating the row-level baseline confidence level R0(y) is as follows: The row-level suppression coefficient α ranges from 0 to 1. The row-level mask M(y) is used to mark rows in the image that are affected by LED stripe interference. When M(y) is 1, the corresponding row is an affected row; when M(y) is 0, the corresponding row is a normal row. The core logic of this calculation is: normal rows are not affected, so their row-level baseline confidence remains at 1 (the highest value); affected rows have their baseline confidence reduced by adjusting α. The larger α is, the more significant the decrease in confidence of affected rows, thus reflecting the unreliability of the data in affected rows.

[0138] In detail, a disparity validity label V(x,y) is generated. The determination rule is as follows: if the absolute value of the corrected disparity dcorr(x,y) is greater than or equal to the minimum reliable disparity threshold dmin, then V(x,y) is set to 1; otherwise, it is set to 0. The minimum reliable disparity threshold dmin is a preset value used to filter out excessively small disparities caused by noise or incorrect matching. That is, such disparities usually do not reflect the depth relationship of the real scene, so they are marked as invalid (V(x,y)=0); while disparities that reach or exceed dmin are considered to have basic validity (V(x,y)=1).

[0139] In detail, the weighted fusion yields the fused pixel confidence score R(x,y). Its calculation formula is the product of the disparity validity label V(x,y) and the row-level baseline confidence score Rrep(x,y) of the row to which the pixel belongs, where Rrep(x,y) is the row-level baseline confidence score R0(y) of the row to which the pixel belongs. This combines the local validity of disparity with the overall reliability at the row level: when the disparity is valid (V(x,y)=1), the pixel confidence score inherits the baseline confidence score of that row; when the disparity is invalid (V(x,y)=0), regardless of the row-level baseline confidence score, the pixel confidence score is 0, thus filtering out unreliable information corresponding to invalid disparities.

[0140] In detail, normalization processing yields the final pixel confidence score Rref(x,y). The specific operation is as follows: first, determine the maximum value among all fused pixel confidence scores R(x,y); then, divide the R(x,y) of each pixel by this maximum value, mapping the result to the range of 0 to 1. The purpose of normalization is to unify the quantization scale of the confidence scores, ensuring that the confidence scores of different pixels are comparable.

[0141] In one embodiment of the present invention, generating occupancy probabilities by voxel division after correcting the parallax projection into a point cloud includes:

[0142] Pixel depth is calculated from the corrected parallax, including:

[0143]

[0144] Where Z(x,y) represents the pixel depth at pixel coordinates (x,y), f represents the camera focal length, and B represents the baseline length of the left and right cameras;

[0145] A 3D point cloud P(x,y) is generated based on the pixel depth, as follows:

[0146]

[0147] The confidence score Rref(x,y) is used as a weight to weight the 3D point cloud P(x,y) to obtain the weighted 3D point cloud PR(x,y);

[0148] Define the voxel side length svox and calculate the voxel index (a, b, c) as follows:

[0149]

[0150] Where (Xmin, Ymin, Zmin) represents the spatial location of the UAV;

[0151] Determine the confidence level and value S1 of the point cloud in each voxel, and the number of point clouds S2;

[0152] The ratio of S1 to S2 is used as the occupancy probability of the corresponding voxel.

[0153] In detail, pixel depth is calculated by correcting for parallax. The calculation formula is: The camera focal length *f* and baseline length *B* are inherent camera parameters, and the corrected disparity *dcorr(x,y)* reflects the horizontal offset of corresponding pixels in the left and right images. The core logic of this formula is based on the principle of stereo vision: disparity is inversely proportional to depth; that is, the greater the offset of the same spatial point in the left and right images (the greater the disparity), the closer it is to the camera (the smaller the depth); conversely, the greater the offset, the farther away it is. Through this calculation, the disparity information of pixels in a two-dimensional image can be converted into the depth information Z(x,y) of the corresponding spatial point.

[0154] In detail, a 3D point cloud P(x,y) is generated based on pixel depth. The coordinates (X,Y,Z) of the 3D point cloud are calculated from the image pixel coordinates (x,y) and depth Z(x,y): X is equal to the product of x and Z(x,y) divided by f, Y is equal to the product of y and Z(x,y) divided by f, and Z is the pixel depth Z(x,y) calculated above. This transformation is based on the perspective projection principle of camera imaging: there is a scaling relationship between the pixel coordinates (x,y) on the image plane and the coordinates (X,Y) in 3D space, with a scaling factor of Z / f. Through this relationship, 2D image pixels can be mapped to 3D space to form a point cloud composed of discrete points, which intuitively reflects the 3D structure of the scene.

[0155] In detail, the 3D point cloud is weighted. The confidence score Rref(x,y) of each 3D point is used as a weight to weight the 3D point cloud P(x,y), resulting in a weighted 3D point cloud PR(x,y). The confidence score Rref(x,y) quantifies the reliability of pixel information. The purpose of the weighting process is to give higher weight to points with high confidence in subsequent analysis, and reduce the weight of points with low confidence (such as points affected by LED stripe interference), thereby reducing the impact of unreliable data on 3D environment modeling.

[0156] In detail, the voxel side length is set and the voxel index is calculated. A voxel is a cubic unit with a side length of svox in 3D space, used to discretize a continuous 3D space into a regular mesh. The voxel index (a, b, c) is calculated as follows: subtract the UAV spatial position (Xmin, Ymin, Zmin) from the 3D point coordinates (X, Y, Z), then divide by the voxel side length svox, and take the integer part of the result. The UAV spatial position (Xmin, Ymin, Zmin) is used as the coordinate reference. This calculation determines the voxel number to which each 3D point belongs, realizing the spatial grouping of the 3D point cloud.

[0157] In detail, the confidence score and value of each voxel are statistically analyzed in relation to the number of points in the point cloud. For each voxel, the sum of the confidence scores (S1) of all its 3D points and the total number of points (S2) are calculated. S1 reflects the sum of the reliability of all points within the voxel, and S2 reflects the density of the data within the voxel. Together, they provide a basis for assessing whether the voxel is occupied by obstacles.

[0158] In detail, the occupancy probability of a voxel is calculated. The occupancy probability is the ratio of the voxel's confidence score S1 to the number of points in the point cloud S2. This ratio is essentially the average confidence score of all points within the voxel. The higher the ratio, the greater the likelihood that the voxel is occupied by a real obstacle, because more high-confidence points support this conclusion. The lower the ratio, the more likely the voxel is noise or unreliable data, thus achieving a quantitative description of obstacle distribution in the 3D environment.

[0159] It should be noted that the UAV is equipped with a Global Navigation Satellite System module, which can receive satellite signals in real time and calculate the UAV's current longitude, latitude, and altitude. After coordinate transformation, the three-dimensional spatial coordinates (Xmin, Ymin, Zmin) can be obtained. In environments with weak GNSS signals, such as indoors, the UAV can use a visual SLAM system to combine image features acquired by the camera with motion estimation to calculate its own position in three-dimensional space in real time, i.e., (Xmin, Ymin, Zmin), as a benchmark for environmental analysis.

[0160] In one embodiment of the present invention, constructing a barrier potential field based on the occupancy probability includes:

[0161] Determine the voxel center coordinates (Xc, Yc, Zc) as follows: ;

[0162] Calculate the distance attenuation factor: ;

[0163] The original potential energy is obtained by weighting the occupancy probability of each voxel according to the distance decay factor.

[0164] The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain a smoothed potential energy.

[0165] The smooth potential energy of the voxels is binarized to obtain the barrier potential field Uobs.

[0166] In detail, the voxel center coordinates (Xc, Yc, Zc) are determined. A voxel is a cubic unit with a side length of svox in three-dimensional space, and its center coordinates are calculated based on the voxel index and the UAV's spatial position. Specifically, Xc equals Yc and Zc are calculated in the same way, based on the UAV's Y-direction position Ymin and voxel Y-direction index b, and the UAV's Z-direction position Zmin and voxel Z-direction index c, respectively. This clarifies the precise center position of each voxel in three-dimensional space, providing a spatial reference for subsequent distance calculations and potential energy assignment.

[0167] In detail, the distance attenuation factor is calculated using the following formula: Zc reflects the distance between the voxel and the UAV in the depth direction. The closer the distance (the smaller Zc), the larger the attenuation factor; the farther the distance (the larger Zc), the smaller the attenuation factor. This makes nearby obstacles pose a more urgent threat to the UAV and require a higher weight in the potential field; distant obstacles pose a lower threat and have a correspondingly lower weight, thus allowing the potential field to reflect the difference in the urgency of obstacles.

[0168] In detail, the weighted processing yields the original potential energy, which is obtained by multiplying the voxel's occupancy probability by a distance decay factor. The occupancy probability reflects the likelihood that the voxel is occupied by a real obstacle, while the distance decay factor reflects the impact of the obstacle's proximity on the threat. Combined, the original potential energy reflects both the credibility of the obstacle's existence and the urgency of its spatial location; that is, voxels with high occupancy probabilities and close proximity have higher original potential energy, indicating a higher obstacle risk in that area.

[0169] In detail, the original potential energy is smoothed using 3D Gaussian convolution. 3D Gaussian convolution performs a weighted average of the potential energy of each voxel with that of its surrounding voxels (the weights follow a Gaussian distribution, with closer voxels having larger weights), making the originally discrete and potentially abrupt potential energy distribution continuous and smooth. This eliminates drastic fluctuations in the potential energy of individual voxels, preventing high-frequency oscillations in subsequent obstacle avoidance commands calculated based on the potential field gradient, and ensuring the stability of the UAV's movements.

[0170] In detail, the smoothed potential energy is binarized to obtain the obstacle potential field Uobs. Binarization is achieved by setting a threshold: if the smoothed potential energy exceeds the threshold, the obstacle potential field Uobs of that voxel is set to 1 (marked as an obstacle region); otherwise, it is set to 0 (marked as a safe region). The purpose of binarization is to clearly distinguish between obstacle regions and safe regions in the environment, transforming the continuous potential energy distribution into a clear obstacle identifier, providing clear spatial guidance for subsequent calculations of avoidance directions.

[0171] In one embodiment of the present invention, obstacle avoidance by the UAV is completed based on the output control command of the obstacle potential field, including:

[0172] The potential energy Up is obtained by interpolation based on the spatial location of the drone, as follows:

[0173]

[0174] in, is (Xmin, Ymin, Zmin), express function;

[0175] By combining the potential energy Up and the obstacle potential field Uobs, the potential energy gradient ∇Uobs is obtained using the three-dimensional finite difference method;

[0176] Calculate the avoidance vector for the potential energy gradient. ,as follows:

[0177]

[0178] The desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector, as follows:

[0179]

[0180] in, Indicates the track weight. This indicates an attempt to avoid weighting. Indicates the potential energy threshold. Represents the initial trajectory velocity vector;

[0181] The saturation velocity is obtained by saturating the amplitude of the desired velocity.

[0182] The saturation velocity is resolved into pitch angle commands, roll angle commands, and vertical velocity commands, which are then processed by the UAV's flight control execution unit.

[0183] In detail, the potential energy Up is obtained by interpolation at the UAV's spatial location. The UAV's spatial location is represented by (Xmin, Ymin, Zmin). The Interp function is used to interpolate the obstacle potential field Uobs to obtain the potential energy Up corresponding to the UAV's current location. The obstacle potential field Uobs is a discrete distribution based on voxel partitioning, while the actual location of the UAV may not be exactly at the voxel center. The purpose of the interpolation is to accurately obtain the potential energy value of the UAV's spatial location, which directly reflects the degree of obstacle risk faced by the UAV at its current location.

[0184] In detail, the potential energy gradient ∇Uobs is calculated by combining the potential energy Up and the obstacle potential field Uobs. The gradient is calculated using the three-dimensional finite difference method, which determines the direction and rate of potential energy change by comparing the potential energy difference between the target voxel and its neighboring voxels. The potential energy gradient ∇Uobs represents the direction of the fastest increase in potential energy within the obstacle potential field; this direction points towards the obstacle region, providing a basis for determining the subsequent avoidance direction.

[0185] In detail, the avoidance vector Vrep is calculated based on the potential energy gradient. When the magnitude of the potential energy gradient is greater than 0, it indicates a significant change in obstacle potential energy. In this case, the avoidance vector Vrep is the negative unit vector of the potential energy gradient, meaning its direction is opposite to the gradient and points away from the obstacle. When the magnitude of the potential energy gradient is equal to 0, it indicates no significant change in obstacle potential energy in the current area, and the avoidance vector Vrep is 0, meaning no active avoidance is needed. This ensures that the UAV can accurately adjust its movement direction towards a safe area.

[0186] In detail, the desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector. In the formula for calculating the desired velocity, Ktrk is the trajectory weight, used to maintain the influence of the initial trajectory velocity vector vtrk, ensuring that the UAV does not deviate from the core mission path while avoiding obstacles; Krep is the avoidance weight, used to adjust the strength of the avoidance vector's influence; the min function dynamically adjusts the actual effect of the avoidance weight through the ratio of Up to the potential energy threshold τU. That is, the larger Up is (i.e., the higher the current obstacle risk), the closer this ratio is to 1, and the stronger the influence of the avoidance vector; conversely, the avoidance influence weakens. This fusion mechanism achieves a balance between trajectory maintenance and obstacle avoidance.

[0187] In detail, the initial trajectory velocity vector is the velocity vector of the UAV when it is flying along a pre-planned trajectory without detecting any obstacles. It includes the magnitude and direction of the velocity and reflects the original flight path intention of the UAV.

[0188] In detail, the avoidance vector is a vector calculated based on the potential energy gradient of the obstacle potential field. When the magnitude of the potential energy gradient is greater than 0, its direction is opposite to that of the obstacle potential field gradient. It is used to indicate the direction in which the UAV deviates from the obstacle and help the UAV avoid the obstacle area.

[0189] In detail, the desired speed is saturated to obtain the saturated speed. Amplitude saturation limits the speed to the physical performance limit of the drone (maximum flight speed). This ensures that the generated speed command is within the drone's executable range, avoiding loss of control or hardware damage due to commands exceeding its capabilities.

[0190] In detail, the saturation velocity is parsed into specific control commands and executed. Saturation velocity is a velocity vector in three-dimensional space, which needs to be converted into attitude and power commands for the UAV, namely pitch angle commands, roll angle commands, and vertical speed commands. The pitch and roll angles determine the UAV's direction of motion on the horizontal plane, while the vertical speed command determines its climb and fall. These commands are then translated by the UAV's flight control unit into specific actions such as motor speed adjustments, ultimately enabling the UAV to perform obstacle avoidance maneuvers.

[0191] For example, if the saturation velocity is a three-dimensional vector (vx = 2 m / s, vy = -1 m / s, vz = 0.3 m / s):

[0192] vx represents the desired speed of the drone in the forward direction. Based on this, the pitch angle command is derived: a positive vx corresponds to a forward pitch angle (e.g., 2 degrees), and adjusting the pitch angle causes the drone to move forward; vy represents the desired speed of the drone in the right direction, and a negative vy corresponds to a left roll angle (e.g., -1 degree), and adjusting the roll angle causes the drone to move to the left; vz represents the desired speed of the drone in the vertical direction, and a positive vz corresponds to a vertical speed command (e.g., 0.3 m / s), which directly controls the motor output to make the drone ascend.

[0193] By analyzing the data, the saturation velocity is converted into attitude and velocity commands that the UAV can execute, thus enabling the motion adjustments required for obstacle avoidance.

[0194] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception, characterized in that, include: The data acquisition module is used to simultaneously acquire the left and right images; The data processing module is used to perform frequency domain analysis based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing. The time difference analysis module is used to perform cross-correlation calculations on the mean grayscale values ​​of the left and right images to determine the acquisition bias. The interference marking module is used to perform phase positioning according to the stripe row spacing, determine the stripe reference row, and construct a row-level mask by cyclically calibrating according to the stripe row spacing based on the stripe reference row. The disparity correction module is used to perform block calculations on the left and right images to obtain the original disparity, and then perform correction processing on the original disparity based on the row-level mask to obtain the corrected disparity. The confidence calculation module is used to assign confidence to each pixel based on the row-level mask, and to generate occupancy probability by dividing the corrected parallax into point cloud and then dividing it into voxels. The drone obstacle avoidance module is used to construct an obstacle potential field based on the occupancy probability and output control commands based on the obstacle potential field to complete the drone obstacle avoidance.

2. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 1, characterized in that, Frequency domain analysis was performed on the row gray-level mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing, including: Calculate the average gray level for each row of pixels in the left image to form a row gray level average sequence; The amplitude spectrum is obtained by performing a fast Fourier transform on the gray-level mean sequence. Determine the dominant frequency index ks of the amplitude spectrum, and calculate the stripe row spacing Δs = 1 / ks based on the dominant frequency index; The pulse width modulation period TP = Δs × Trow is calculated using the prior readout period Trow.

3. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 2, characterized in that, Perform cross-correlation on the mean grayscale values ​​of the left and right images to determine the acquisition bias, including: Calculate the cross-correlation value between the row gray-level mean sequence element y of the left image and the row gray-level mean sequence element y of the right image under the row offset δ; Determine the range of values ​​for the row offset δ, iterate through the values ​​of the row offset δ, and find the row offset δ that results in the largest cross-correlation value as the target row offset δ*. The acquisition bias is calculated based on the target row offset δ*, as follows: in, This indicates a data acquisition bias.

4. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 3, characterized in that, Phase positioning is performed according to the stripe row spacing to determine the stripe reference row. Using the stripe reference row as a reference, cyclic calibration is performed according to the stripe row spacing to construct a row-level mask, including: Within the row index interval [1, Δs], calculate the local brightness contrast of candidate row yc; The candidate row corresponding to the maximum local brightness contrast is used as the stripe reference row yref; Calculate the modulus value for each row of pixels in the left image based on the stripe reference row; If the modulus of the y-th row pixel is less than the preset effective bandwidth of the stripe, then mark the y-th row pixel in the left image as 1; otherwise, mark the y-th row pixel in the left image as 0 to obtain the row-level mask.

5. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 4, characterized in that, The original disparity is calculated by performing block-based calculations on the left and right images, and then corrected based on row-level masks to obtain the corrected disparity, including: For pixel coordinates (x, y), perform block matching with a fixed window size of w×w between the left and right images to obtain the original disparity; The original disparity draw(x,y) is sign-flipped using a row-level mask to obtain the corrected disparity.

6. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 5, characterized in that, Confidence is assigned to each pixel based on the row-level mask, including: Calculate the row-level baseline confidence level R0(y) based on the row-level mask; Generate a disparity validity label V(x,y) based on corrected disparity; For pixel coordinates (x, y), a weighted fusion is performed based on row-level baseline confidence and disparity validity label to obtain the fused pixel confidence. Determine the maximum value of the fused pixel confidence score, and normalize the fused pixel confidence score of each pixel coordinate based on the maximum value to obtain the confidence score Rref(x,y) of each pixel coordinate.

7. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 6, characterized in that, After correcting the parallax projection into a point cloud, occupancy probabilities are generated by voxel partitioning, including: Pixel depth is calculated from the corrected parallax. Generate a 3D point cloud P(x,y) based on pixel depth; Set the voxel side length svox and calculate the voxel index (a,b,c); Determine the confidence level and value S1 of the point cloud in each voxel, and the number of point clouds S2; The ratio of S1 to S2 is used as the occupancy probability of the corresponding voxel.

8. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 7, characterized in that, Constructing a barrier potential field based on occupancy probability, including: Determine the coordinates of the voxel center (Xc, Yc, Zc); The distance attenuation factor is determined based on the voxel center coordinates; The original potential energy is obtained by weighting the occupancy probability of each voxel according to the distance decay factor. The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain a smoothed potential energy. The smooth potential energy of the voxels is binarized to obtain the barrier potential field Uobs.

9. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 8, characterized in that, The UAV completes obstacle avoidance based on the output control commands of the obstacle potential field, including: Potential energy Up is obtained by interpolation based on the spatial position of the drone; By combining the potential energy Up and the obstacle potential field Uobs, the potential energy gradient ∇Uobs is obtained using the three-dimensional finite difference method; Calculate the avoidance vector based on the potential energy gradient. ; The desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector; The saturation velocity is obtained by saturating the amplitude of the desired velocity. The saturation velocity is resolved into pitch angle commands, roll angle commands, and vertical velocity commands, which are then processed by the UAV's flight control execution unit.

Images